Async Techniques and Examples in Python Transcripts
Chapter: async and await with asyncio
Lecture: Demo: The producer-consumer app

Login or purchase this course to watch this video and the rest of the course contents.
0:00 Let's begin our exploration of asyncio by looking at this common pattern called a producer, consumer pattern.
0:07 The idea is, there's some part of the system that typically runs independently asynchronously, that generates items.
0:15 Maybe it receives a request to fulfill a job charge to this credit card, send these emails import this data file, things like that.
0:23 There's another part of the system that is also running asynchronously that looks for work to be done, and as the consumer
0:31 it's going to pick up these jobs that are created by the producer, and start working on them. I have an extremely simple version of that here.
0:39 It doesn't really do anything useful. So, we have generate data, and we have process data. And we're just using this list to share
0:45 to exchange data between these two. So, let's look at generate data. It's going to go, and just create the square of whatever number we give it
0:55 and it's going to store in our list, a tuple. So that's one thing going in the list that is a tuple the item that was generated, and the time at which
1:03 that generation happened. And then it prints out in yellow. We're using this cool library, called Colorama to do different colored output.
1:09 And color, in threading concurrences is super helpful to understand what part is happening where and what order it's all happening in.
1:17 We're also flushing everything to make sure there's no delay in the buffer when we print stuff. As soon as we say print, you see it.
1:23 So, to simulate it taking some time to generate some data or do the producer side, we call time.sleep() for between 0.5 and 1.5 seconds.
1:33 On the process data side we're going to look for some data if it's not there, this structure doesn't really have a concept of get it, and wait
1:41 we'll talk more about that later. But once an item comes in, we just grab it. In cyan we say we found it, and we also print out
1:48 and we determine how long, from when it was created till we were able to get to it, and process it. Because the lower that number, the more concurrency
1:57 the better we have. A soon as something is deposited for us to use, and pick up we should immediately get to it. But this code is synchronous.
2:05 We do all the generation, then all the processing. So the first item is not going to be processed until all the generation is done, in this 20, and 20.
2:13 Let's just run it and see what this means. The app is running, you can see we're generating in yellow.
2:19 It's kind of a subdued color in the pycharm terminal but yellowish it's going to come out and cyan for the consumer. We're waiting for all 20.
2:33 There we are. Now we're starting to process them. Notice, we were not able to get to the first one until 20 seconds after it was created.
2:40 Now, the consumption of these is faster than the production so we were able to catch up somewhat and we got it down to 10 seconds, 10.48 seconds
2:49 for the best case scenario. You can see these numbers that we picked up out of there were the ones generated previously.
2:56 In the overall time it took to run that was 30, 31-ish seconds. None of this is great, and we can actually dramatically improve this with asyncio.
3:06 Not a little bit, a whole lot.


Talk Python's Mastodon Michael Kennedy's Mastodon