Async Techniques and Examples in Python Transcripts
Chapter: async and await with asyncio
Lecture: Demo: Make the producer-consumer async (methods)
Login or
purchase this course
to watch this video and the rest of the course contents.
0:00
We've got the execution of our async loop working and we've created our tasks, we've gathered them up we're waiting to run until complete.
0:09
But obviously these are standard old Python methods so what we need to do is adapt these two methods generate_data and process_data so they work.
0:19
If we try to run it now I'm not really sure what's going to happen it's got to be some kind of error, let's find out.
0:24
Run time warning, stuff is not awaited. Well they did start but when we tried to generate and when we tried to gather up the data and process it
0:35
didn't work out so well, did it? No, it did not. Yeah, right here it said you know that didn't work, we called it
0:44
it returned None and this thing said I expect a coroutine not this thing that ran and returned some value. Let's go and convert this, now there's
0:54
two things that we have to do to make our method's coroutines, asynchronous coroutines and there's two Python keywords to make this happen.
1:06
So what we have to do is we have to mark the method as an async method. So we say async like this this is a new keyword as of Python 3.5.
1:14
So now we have two async methods, and if we run them they do sort of run but notice there's no asynchronous behavior here
1:23
they're not actually asynchronous, so that's not great. Because just the fact that we say they're asynchronous
1:30
you know, Python doesn't decide how to slice them up. Our job is to take this code here and say I would like to break it into little pieces
1:40
that can run and the dividing points are where we're waiting. Well, what are we waiting on here? So PyCharm helps a little bit
1:49
and this data structure helps a lot. So there's two places we're waiting on generate_data we're waiting, clearly on line 33, we're saying time.sleep.
2:00
Now you don't want to use time.sleep this puts the whole thread and the entire asyncio loop to sleep.
2:07
There's a better way to indicate I'm done for a while I'm just going to sleep, you can keep working so we can use the asyncio's sleep, okay?
2:17
Same as time.sleep, but now there's two warnings and what are those warnings? It is PyCharm saying you have an asynchronous method
2:24
with two places you could wait you could tell the system to keep running and go do other stuff, but you're not.
2:32
So Python has a cool keyword called await and you just put that in front and these are the markers on line 30 and 33 that tells
2:40
Python to break up the code into little slices. Come in here, run until you get to this point wait and let other stuff happen here
2:48
when you're done with that and you have some time come back and do this and then wait until you're done sleeping which is, you know, .5 to 1.5 seconds
2:57
and then carry on again, okay? So we've converted this to an async method now we got to do the same thing down here.
3:06
So this is async, and this one, what we have to do is for some reason PyCharm doesn't know about the get
3:11
but if you go here you can see it's an async method. So that means that we have to await it when you await the item, it's going to block
3:21
until the return value comes out and the return value is going to be this tuple. Notice before the await, we had errors here
3:27
it was saying coroutine does not have get_item. So when you call a coroutine you have to await it to actually get the value back from it, okay?
3:37
Again, we were sleeping down here so this is going to be an asyncio.sleep and now that it's that style we have to await it as well.
3:44
So pretty simple, the two steps mark the methods as async required and then once you do that, you can use the await keyword
3:52
anywhere you would have waited. You're calling a function that is async. In this case it's kind of silly, sleep stuff
4:00
but we're going to do a much better, realistic example when we start talking to web servers and stuff, real soon. So this is sort of a toy example
4:07
we're going to get some real work done with async and await in just a moment. I think we're ready, our times are awaited and they're async
4:16
and the various places where we get and put data we're awaiting and we have the async methods here let's run this and see what we get.
4:22
Notice we don't need time anymore. Notice a very different behavior, look at this we're generating the item and then we're going
4:33
to wait on something, waiting for more data to be generated or delivered, so then we can immediately go and process it.
4:39
Look at that latency, 0, soon as an item is delivered or almost as soon, we can go and pick it up.
4:47
It took 20 seconds instead of 30, that's pretty awesome. Now we're having 0000 here because the producer is not producing as fast
4:58
as the consumer can consume. So let's take this and make one minor change here let's generate one more of those, I call that a task2
5:06
that task3, and this one we need to tell it it's expecting more data we're not dealing with cancellation and our other types of signals like that
5:14
not yet, task3, okay. So now we should be producing more data than we can consume and that should be even a little more interesting.
5:24
Notice the beginning it's kind of similar but now we sometimes process two sometimes we're processing one. The latencies are still pretty good
5:32
but like I said, we can't quite keep up, so that's okay let's see what happens at the end. Yeah, pretty good, so in the beginning we generated a couple
5:48
had to kind of do a couple double steps to catch up but in the end we were more or less able to keep up after we generated all the data
5:55
especially when one of them finished and the other was still working. Really, really cool, even generating twice as much data
6:03
it took only 22 milliseconds. I believe that would have been 50, sorry 22 seconds I believe that would have been 50 seconds
6:12
if we had done that synchronously. How cool is this? So all we had to do is to break up these functions
6:19
into parts where we're waiting on something else we're waiting on putting items into a queue we're waiting on, well, time, but like I said
6:26
that's a stand-in for database access file access, cache, network, etc. Okay so we just break them up using this await keyword
6:36
we didn't have to change our programming model at all that's the most important takeaway here. We didn't have to, all the sudden
6:42
start off a thread, use callbacks and other weird completion. It's exactly the same code that you would normally write
6:50
except for you have async and you have these await sections. So the programming model is the standard synchronous model that you know
6:57
and that's really, really nice and important. All right, that's a first introduction to asyncio. We create the loop, we create some tasks based on
7:10
some coroutines that we're going to call and then we run until they're complete.