Async Techniques and Examples in Python Transcripts
Chapter: Leveraging CPU cores with multiprocessing
Lecture: Concept: Scaling CPU-bound operations
Login or
purchase this course
to watch this video and the rest of the course contents.
0:00
Let's review the core concepts around running our code in parallel on multiple processes using Python's multiprocessing.
0:08
So, we're going to start by importing and creating a pool instance, and here we decided we're only
0:14
going to let it create four subprocesses 'cause we're only going to ask it to do four things in parallel.
0:19
So if you don't pass that processes equal four it's just going to pick the number of CPU cores you have a use that as the underlying process count.
0:27
Okay, so then we go to pool and we say apply_sync() and we give it a function in arguments and then we can just tell it no more work is coming.
0:34
And then do join, lets us just block kind of like when we join on a thread we join on this pool and it waits 'til all the work is done
0:41
and then it continues. So we start all the work by saying apply_sync() and then we have to call close() and then join().
0:49
I didn't make that super explicit but the docs say you should call close() and then join()
0:54
don't just join() or you're probably going to wait a long time something to that effect. So here's the overall concept for doing parallel work
1:02
actually skipping around the GIL by entirely sidestepping this GIL concept altogether and just going, well fine, if you're going to have a GIL
1:09
we're going to have a single process for each effective line of execution as if they were threads but we'll let the OS handle the details.
1:17
The trick is that the arguments get passed and we'll also see that you can get data back from these.