Async Techniques and Examples in Python Transcripts
Chapter: Leveraging CPU cores with multiprocessing
Lecture: Concept: Scaling CPU-bound operations

Login or purchase this course to watch this video and the rest of the course contents.
0:00 Let's review the core concepts around running our code in parallel on multiple processes using Python's multiprocessing.
0:08 So, we're going to start by importing and creating a pool instance, and here we decided we're only
0:14 going to let it create four subprocesses 'cause we're only going to ask it to do four things in parallel.
0:19 So if you don't pass that processes equal four it's just going to pick the number of CPU cores you have a use that as the underlying process count.
0:27 Okay, so then we go to pool and we say apply_sync() and we give it a function in arguments and then we can just tell it no more work is coming.
0:34 And then do join, lets us just block kind of like when we join on a thread we join on this pool and it waits 'til all the work is done
0:41 and then it continues. So we start all the work by saying apply_sync() and then we have to call close() and then join().
0:49 I didn't make that super explicit but the docs say you should call close() and then join()
0:54 don't just join() or you're probably going to wait a long time something to that effect. So here's the overall concept for doing parallel work
1:02 actually skipping around the GIL by entirely sidestepping this GIL concept altogether and just going, well fine, if you're going to have a GIL
1:09 we're going to have a single process for each effective line of execution as if they were threads but we'll let the OS handle the details.
1:17 The trick is that the arguments get passed and we'll also see that you can get data back from these.

Talk Python's Mastodon Michael Kennedy's Mastodon