Async Techniques and Examples in Python Transcripts
Chapter: Course conclusion and review
Lecture: Review: multiprocessing
Login or
purchase this course
to watch this video and the rest of the course contents.
0:00
If you're trying to do computational work you would like to take maximum advantage of all the CPU cores. If you open up your system information
0:09
and look at your process usage you would like it to look like this if you're actually trying to do hardcore computation.
0:16
However, remember the Python GIL. If we create multiple threads the GIL blocks all the threads except for one while it's running.
0:25
And this is primarily to make memory management faster and easier for the single threaded, most common use case.
0:31
But it does mean threads don't really solve the problem. So, what we're going to do to get around the GIL say you know what, we're not going to run
0:37
more than one thread per process but let's just start a bunch of processes. Yes, they all still have the GIL but the Gil's not stopping them
0:44
because they're all single threaded. So, multiprocessing is to take the concurrency and spin up a bunch of different Python processes
0:51
feed little parts of work to them and then get the result back as if we're using threads or some other form of concurrency and this entirely sidesteps
1:01
the whole GIL problem. So, this is the most fundamental, straightforward way to take advantage of multiple cores and modern hardware in Python.
1:10
How do we do it, well, here's the code. Probably the best way is to use the multiprocessing pool. So, we're going to start by creating a pool
1:16
and notice we say processes equal four. If you leave that empty, if you just say pool with taking the defaults, it's going to create
1:24
as many processes or set that process count to as many processes as you have CPU cores. So, on mine that would be 12
1:31
which would probably be too many for what we're doing here. Next, we're going to start a bunch of operations with async work.
1:38
Here we're processing a bunch of operations from zero to 400 and breaking them into four 100 piece segments
1:44
zero to 100, 101 to two, 200 to 300, and so on. I guess maybe that should be 201. Anyway, you get the idea.
1:50
So, we're going to break our work up into little bits feed it off of the different processes and then you have to call close and then join.
1:58
Well, you should call close for sure. If you want join to work properly you have to call close first. So, this basically starts all that work
2:06
and then says, we're not going to submit any more work now let's wait for what we have submitted to be finished
2:12
and this is how you do multiprocessing in Python.