Async Techniques and Examples in Python Transcripts
Lecture: Demo: Something productive with threads
0:00 Let's return to an old friend
0:01 the producer consumer that we learned about
0:03 in the asyncio chapter.
0:06 So here we have the single threaded
0:10 non-concurrent producer consumer thing
0:13 that we've been playing with in the previous chapter.
0:15 So we're going to generate a bunch of data
0:16 and we're going to generate some more data
0:18 then we're going to process it.
0:19 Now if we run it, remember it generates
0:21 some data the first time through
0:23 and then it does it again, and so on.
0:25 However, we saw that using asyncio let us really
0:29 speed this up.
0:30 Let's assume for some reason that we don't have asyncio
0:33 available to us. Now what we're actually doing when we do
0:36 and we've already implemented it and you've seen that
0:38 but assuming that we don't, how would we model
0:40 this with threads?
0:42 A lot of the reasons I gave you before
0:45 that might be the case here.
0:46 Maybe we're interacting with some system here
0:48 that we're waiting on, but it doesn't actually
0:50 have an asyncio option for us.
0:53 So we're going to do what we just talked about
0:54 and we're going to have the threads
0:56 that we're going to work with.
0:57 I'm going to go in here and I'm going to create some threads
0:59 need to import threading, set the target
1:06 to generate_data, and set the args to be
1:10 what they're going to be, so 20 and data
1:13 and I guess if you want to use data
1:14 you should probably define it above.
1:17 That all looks good, and are doing that again
1:19 and then the other one, we're using process_data
1:24 and we're using 40.
1:26 So that's going to get rid of those.
1:28 Now again, this doesn't start them.
1:30 So if we ran this now, it would be super fast
1:34 zero milliseconds, zero seconds at least.
1:36 However, it didn't do any work.
1:39 So what we want to do is use our little trick
1:40 t.start() for t in threads, and then we want
1:47 to print started, somethin' like that
1:51 and then we'll do join.
1:54 Alright, so this, that should really be
1:56 all there was to it, and let's try.
1:59 Now this assumes of course that process_data
2:04 will go through and it has this sort of
2:06 continue step to know how much it's supposed to process.
2:10 There's a little bit of work to make sure this works
2:12 but it was already in place, and so we should
2:13 be able to just run this now.
2:16 There they go, see them producing
2:18 and consuming in parallel. Perfect, right?
2:21 Looks just like the asyncio version
2:23 that we were working with, except for
2:26 apparently we can't pop from an empty list
2:30 let's see what we did there.
2:31 Ah yes, alright, let's fix this.
2:39 Alright, so that should work
2:40 sorry about that, try again.
2:42 See, threading will pull out interesting errors
2:44 you did not think were there.
2:48 Here you can see it's done, the latency's
2:50 a little bit higher than we were looking at previously.
2:54 The overall execution time is about the same
2:56 as our asyncio version. Pretty straightforward right?
2:59 We just use our technique, create a whole bunch of threads
3:03 start them, maybe do other work
3:05 maybe just chill, and then wait for them
3:07 to finish by joining on all of them.