Async Techniques and Examples in Python Transcripts
Chapter: async and await with asyncio
Lecture: Demo: The producer-consumer app
Login or
purchase this course
to watch this video and the rest of the course contents.
0:00
Let's begin our exploration of asyncio by looking at this common pattern called a producer, consumer pattern.
0:07
The idea is, there's some part of the system that typically runs independently asynchronously, that generates items.
0:15
Maybe it receives a request to fulfill a job charge to this credit card, send these emails import this data file, things like that.
0:23
There's another part of the system that is also running asynchronously that looks for work to be done, and as the consumer
0:31
it's going to pick up these jobs that are created by the producer, and start working on them. I have an extremely simple version of that here.
0:39
It doesn't really do anything useful. So, we have generate data, and we have process data. And we're just using this list to share
0:45
to exchange data between these two. So, let's look at generate data. It's going to go, and just create the square of whatever number we give it
0:55
and it's going to store in our list, a tuple. So that's one thing going in the list that is a tuple the item that was generated, and the time at which
1:03
that generation happened. And then it prints out in yellow. We're using this cool library, called Colorama to do different colored output.
1:09
And color, in threading concurrences is super helpful to understand what part is happening where and what order it's all happening in.
1:17
We're also flushing everything to make sure there's no delay in the buffer when we print stuff. As soon as we say print, you see it.
1:23
So, to simulate it taking some time to generate some data or do the producer side, we call time.sleep() for between 0.5 and 1.5 seconds.
1:33
On the process data side we're going to look for some data if it's not there, this structure doesn't really have a concept of get it, and wait
1:41
we'll talk more about that later. But once an item comes in, we just grab it. In cyan we say we found it, and we also print out
1:48
and we determine how long, from when it was created till we were able to get to it, and process it. Because the lower that number, the more concurrency
1:57
the better we have. A soon as something is deposited for us to use, and pick up we should immediately get to it. But this code is synchronous.
2:05
We do all the generation, then all the processing. So the first item is not going to be processed until all the generation is done, in this 20, and 20.
2:13
Let's just run it and see what this means. The app is running, you can see we're generating in yellow.
2:19
It's kind of a subdued color in the pycharm terminal but yellowish it's going to come out and cyan for the consumer. We're waiting for all 20.
2:33
There we are. Now we're starting to process them. Notice, we were not able to get to the first one until 20 seconds after it was created.
2:40
Now, the consumption of these is faster than the production so we were able to catch up somewhat and we got it down to 10 seconds, 10.48 seconds
2:49
for the best case scenario. You can see these numbers that we picked up out of there were the ones generated previously.
2:56
In the overall time it took to run that was 30, 31-ish seconds. None of this is great, and we can actually dramatically improve this with asyncio.
3:06
Not a little bit, a whole lot.