Async Techniques and Examples in Python Transcripts
Chapter: Course conclusion and review
Lecture: Review: multiprocessing

Login or purchase this course to watch this video and the rest of the course contents.
0:00 If you're trying to do computational work you would like to take maximum advantage of all the CPU cores. If you open up your system information
0:09 and look at your process usage you would like it to look like this if you're actually trying to do hardcore computation.
0:16 However, remember the Python GIL. If we create multiple threads the GIL blocks all the threads except for one while it's running.
0:25 And this is primarily to make memory management faster and easier for the single threaded, most common use case.
0:31 But it does mean threads don't really solve the problem. So, what we're going to do to get around the GIL say you know what, we're not going to run
0:37 more than one thread per process but let's just start a bunch of processes. Yes, they all still have the GIL but the Gil's not stopping them
0:44 because they're all single threaded. So, multiprocessing is to take the concurrency and spin up a bunch of different Python processes
0:51 feed little parts of work to them and then get the result back as if we're using threads or some other form of concurrency and this entirely sidesteps
1:01 the whole GIL problem. So, this is the most fundamental, straightforward way to take advantage of multiple cores and modern hardware in Python.
1:10 How do we do it, well, here's the code. Probably the best way is to use the multiprocessing pool. So, we're going to start by creating a pool
1:16 and notice we say processes equal four. If you leave that empty, if you just say pool with taking the defaults, it's going to create
1:24 as many processes or set that process count to as many processes as you have CPU cores. So, on mine that would be 12
1:31 which would probably be too many for what we're doing here. Next, we're going to start a bunch of operations with async work.
1:38 Here we're processing a bunch of operations from zero to 400 and breaking them into four 100 piece segments
1:44 zero to 100, 101 to two, 200 to 300, and so on. I guess maybe that should be 201. Anyway, you get the idea.
1:50 So, we're going to break our work up into little bits feed it off of the different processes and then you have to call close and then join.
1:58 Well, you should call close for sure. If you want join to work properly you have to call close first. So, this basically starts all that work
2:06 and then says, we're not going to submit any more work now let's wait for what we have submitted to be finished
2:12 and this is how you do multiprocessing in Python.

Talk Python's Mastodon Michael Kennedy's Mastodon