Async Techniques and Examples in Python Transcripts
Chapter: Course conclusion and review
Lecture: Review: multiprocessing
0:00 If you're trying to do computational work
0:02 you would like to take maximum advantage
0:04 of all the CPU cores.
0:06 If you open up your system information
0:08 and look at your process usage
0:10 you would like it to look like this
0:12 if you're actually trying to do hardcore computation.
0:15 However, remember the Python GIL.
0:18 If we create multiple threads
0:20 the GIL blocks all the threads
0:22 except for one while it's running.
0:24 And this is primarily to make memory management faster
0:27 and easier for the single threaded, most common use case.
0:30 But it does mean threads don't really solve the problem.
0:33 So, what we're going to do to get around the GIL
0:35 say you know what, we're not going to run
0:36 more than one thread per process
0:38 but let's just start a bunch of processes.
0:40 Yes, they all still have the GIL
0:41 but the Gil's not stopping them
0:43 because they're all single threaded.
0:45 So, multiprocessing is to take the concurrency
0:47 and spin up a bunch of different Python processes
0:50 feed little parts of work to them
0:52 and then get the result back
0:54 as if we're using threads or some other form
0:57 of concurrency and this entirely sidesteps
1:00 the whole GIL problem.
1:01 So, this is the most fundamental, straightforward way
1:04 to take advantage of multiple cores
1:06 and modern hardware in Python.
1:09 How do we do it, well, here's the code.
1:11 Probably the best way is to use the multiprocessing pool.
1:14 So, we're going to start by creating a pool
1:15 and notice we say processes equal four.
1:18 If you leave that empty, if you just say pool
1:20 with taking the defaults, it's going to create
1:23 as many processes or set that process count
1:26 to as many processes as you have CPU cores.
1:29 So, on mine that would be 12
1:30 which would probably be too many for what we're doing here.
1:33 Next, we're going to start
1:35 a bunch of operations with async work.
1:37 Here we're processing a bunch of operations from zero to 400
1:40 and breaking them into four 100 piece segments
1:43 zero to 100, 101 to two, 200 to 300, and so on.
1:46 I guess maybe that should be 201. Anyway, you get the idea.
1:49 So, we're going to break our work up into little bits
1:51 feed it off of the different processes
1:53 and then you have to call close and then join.
1:57 Well, you should call close for sure.
1:58 If you want join to work properly
2:01 you have to call close first.
2:02 So, this basically starts all that work
2:05 and then says, we're not going to submit any more work
2:07 now let's wait for what we have submitted to be finished
2:11 and this is how you do multiprocessing in Python.