Async Techniques and Examples in Python Transcripts
Chapter: Leveraging CPU cores with multiprocessing
Lecture: Concept: Scaling CPU-bound operations
0:00 Let's review the core concepts around
0:02 running our code in parallel on multiple processes
0:04 using Python's multiprocessing.
0:07 So, we're going to start by importing and creating
0:10 a pool instance, and here we decided we're only
0:13 going to let it create four subprocesses 'cause we're
0:15 only going to ask it to do four things in parallel.
0:18 So if you don't pass that processes equal four
0:20 it's just going to pick the number of CPU cores
0:22 you have a use that as the underlying process count.
0:26 Okay, so then we go to pool and we say apply_sync()
0:29 and we give it a function in arguments
0:31 and then we can just tell it no more work is coming.
0:33 And then do join, lets us just block
0:36 kind of like when we join on a thread
0:38 we join on this pool
0:39 and it waits 'til all the work is done
0:40 and then it continues.
0:42 So we start all the work by saying apply_sync()
0:45 and then we have to call close() and then join().
0:48 I didn't make that super explicit but the docs say
0:51 you should call close() and then join()
0:53 don't just join() or you're probably going to wait a long time
0:56 something to that effect.
0:57 So here's the overall concept for doing parallel work
1:01 actually skipping around the GIL by entirely
1:03 sidestepping this GIL concept altogether
1:06 and just going, well fine, if you're going to have a GIL
1:08 we're going to have a single process for each
1:10 effective line of execution
1:12 as if they were threads
1:13 but we'll let the OS handle the details.
1:16 The trick is that the arguments get passed
1:18 and we'll also see that you can get data back from these.