Async Techniques and Examples in Python Transcripts
Chapter: async and await with asyncio
Lecture: Demo: Make the producer-consumer async
Login or
purchase this course
to watch this video and the rest of the course contents.
0:00
You've seen our producer consumer running in synchronous mode. We generate all the data and then we process all the data.
0:09
And of course the first thing happens and then the second because we're not using any concurrent algorithms or techniques here.
0:16
Now also notice I have a produce-sync and a produce-async. Currently these are exactly the same
0:22
so I'm going to leave you this sync program in source control so you can start from scratch and basically
0:28
where we're starting from right now if you'd like. But we're going to convert this what currently is a synchronous program to the async one.
0:34
So in the end async program will be, well the async version. So there's a couple of things we have to do to make our program async.
0:44
Syntactically they're simple. Conceptually there's a lot going on here and we've already talked a lot about that
0:50
with the event loop and slicing our functions up into little pieces. We've talked about the generator function
0:56
and these restartable resumable coroutines. All of that stuff is going to come in play here. But the API, pretty straightforward.
1:03
So what we need to do is two parts. Let's start with the execution of these things. So in order to run asynchronous coroutines
1:14
we can't just call them like regular functions like right here. We have to run them in that asyncio loop and that asyncio loop is going to execute
1:22
on whatever thread or environment that we start it on. And it's our job to create and run that loop. So let's go over here and create the loop.
1:35
And it's really easy. All we do is say asyncio and we have to import that at the top. We say, get_event_loop().
1:43
And later we're going to say, loop.run_until_complete(). And we have to pass something in here. So we're going to talk for just a moment
1:52
wait just a moment to see what we're going to pass there. So we're going to create the loop we're going to give it these items to execute
2:00
and then we're going to have it run until they are completed. The other thing that we need to do is we need a different data structure.
2:09
So right now we're passing this list and we're just this standard Python list and we're asking, hey is there something in there?
2:16
And if there is, let me get it otherwise let me sleep. And we technically could do that again just with the async version.
2:24
But it turns out that asyncio has a better way to work with this. So we can get what's called an asyncio.Queue. It's like a standard queue.
2:33
You put stuff in and it sort of piles up. You ask for it out it comes out in the order it went in. First-in, first-out style okay? The benefit is
2:41
this one allows us to wait and tell the asyncio loop you can continue doing other work until something comes into this queue and then wake up
2:51
resume my coroutine, and get it running. So you'll see that it's going to be really helpful. Now it looks like somethin's wrong. It sort of is.
3:00
Right here you see this little highlight around data and that's because in the type annotations we've said that this is a list
3:07
but we just need to type that as an asyncio.Queue and then just adapt the methods here. Okay so we don't have an append, we have a put.
3:17
We don't have a pop, we have a get. Now we're going to see some interesting things happening here that we're still going to have to deal with
3:26
but we're going to use this asyncio queue. So the way it's going to work is we're going to track the execution of these separately.
3:35
We're going to kick them off we're going to run all of those and then we want to go down here and say start them and wait until they finish.
3:43
So we're going to use another concept to get them started here. We're going to go to our loop and we'll say
3:48
create task and we're just going to pass in here. And we actually call the data with the call the function or the coroutine with the parameters.
3:56
But like a generator function it doesn't actually run until you start to pull on it, make it go and that's what create_task is going to do.
4:04
So we'll say task1 equals this and we'll do exactly the same for the other except we'll call it task2. And then we want to wait
4:13
come down here and run our loop but wait until complete. Now if you look at what this takes it takes a single future or task.
4:21
A single thing that is going to run and then we can wait on it. Not two, one. And that's super annoying. Why that's not a star args
4:30
and they do what we're about to do automatically for us. It's okay, so we'll say final_task or full task
4:35
final_task, and we're going to go to asyncio and say gather and what you do is you give it a list of tasks. task1, task2, et cetera.
4:45
And then we run the final_task to completion. Alright so this is the execution side of things. We create the loop up here on line 10, create the loop.
4:56
We use a better data structure we kick off a few coroutines and we convert them into a single task that we can then run until complete.
5:05
What's interesting is nothing is going to happen until line 21. If we printed out hey starting this loop, starting this coroutine
5:14
starting this method, generating data. None of that is going to actually do anything until we start it here we're going to block and then we'll be done
5:22
and we'll figure out how long it took. So this is how we run coroutines but we don't yet have coroutines down here. So our next task is going to be
5:30
to convert those into actual asynchronous coroutines rather than standard Python methods.