Async Techniques and Examples in Python Transcripts
Chapter: async and await with asyncio
Lecture: Demo: Make the producer-consumer async (methods)
0:00 We've got the execution of our async loop working
0:04 and we've created our tasks, we've gathered them up
0:06 we're waiting to run until complete.
0:08 But obviously these are standard old Python methods
0:11 so what we need to do is adapt these two methods
0:15 generate_data and process_data so they work.
0:18 If we try to run it now
0:19 I'm not really sure what's going to happen
0:20 it's got to be some kind of error, let's find out.
0:23 Run time warning, stuff is not awaited.
0:28 Well they did start but when we tried to generate
0:32 and when we tried to gather up the data and process it
0:34 didn't work out so well, did it?
0:37 No, it did not. Yeah, right here it said
0:40 you know that didn't work, we called it
0:43 it returned None and this thing said I expect a coroutine
0:46 not this thing that ran and returned some value.
0:51 Let's go and convert this, now there's
0:53 two things that we have to do to make our
0:57 method's coroutines, asynchronous coroutines
1:01 and there's two Python keywords to make this happen.
1:05 So what we have to do is we have to
1:06 mark the method as an async method.
1:08 So we say async like this
1:10 this is a new keyword as of Python 3.5.
1:13 So now we have two async methods, and if we run them
1:17 they do sort of run
1:20 but notice there's no asynchronous behavior here
1:22 they're not actually asynchronous, so that's not great.
1:27 Because just the fact that we say they're asynchronous
1:29 you know, Python doesn't decide how to slice them up.
1:33 Our job is to take this code here
1:37 and say I would like to break it into little pieces
1:39 that can run and the dividing points
1:42 are where we're waiting.
1:44 Well, what are we waiting on here?
1:46 So PyCharm helps a little bit
1:48 and this data structure helps a lot.
1:51 So there's two places we're waiting on generate_data
1:54 we're waiting, clearly on line 33, we're saying time.sleep.
1:59 Now you don't want to use time.sleep
2:01 this puts the whole thread
2:02 and the entire asyncio loop to sleep.
2:06 There's a better way to indicate I'm done for a while
2:10 I'm just going to sleep, you can keep working
2:12 so we can use the asyncio's sleep, okay?
2:16 Same as time.sleep, but now there's two warnings
2:19 and what are those warnings?
2:21 It is PyCharm saying you have an asynchronous method
2:23 with two places you could wait
2:26 you could tell the system to keep running
2:28 and go do other stuff, but you're not.
2:31 So Python has a cool keyword called await
2:34 and you just put that in front
2:36 and these are the markers on line 30 and 33 that tells
2:39 Python to break up the code into little slices.
2:42 Come in here, run until you get to this point
2:45 wait and let other stuff happen here
2:47 when you're done with that and you have some time
2:48 come back and do this
2:51 and then wait until you're done sleeping
2:53 which is, you know, .5 to 1.5 seconds
2:56 and then carry on again, okay?
2:59 So we've converted this to an async method
3:02 now we got to do the same thing down here.
3:05 So this is async, and this one, what we have to do is
3:08 for some reason PyCharm doesn't know about the get
3:10 but if you go here you can see it's an async method.
3:15 So that means that we have to await it
3:18 when you await the item, it's going to block
3:20 until the return value comes out
3:22 and the return value is going to be this tuple.
3:24 Notice before the await, we had errors here
3:26 it was saying coroutine does not have get_item.
3:30 So when you call a coroutine you have to await it
3:33 to actually get the value back from it, okay?
3:36 Again, we were sleeping down here
3:38 so this is going to be an asyncio.sleep
3:40 and now that it's that style we have to await it as well.
3:43 So pretty simple, the two steps
3:45 mark the methods as async required
3:48 and then once you do that, you can use the await keyword
3:51 anywhere you would have waited.
3:53 You're calling a function that is async.
3:56 In this case it's kind of silly, sleep stuff
3:59 but we're going to do a much better, realistic example
4:01 when we start talking to web servers and stuff, real soon.
4:04 So this is sort of a toy example
4:06 we're going to get some real work done
4:08 with async and await in just a moment.
4:11 I think we're ready, our times are awaited and they're async
4:15 and the various places where we get
4:16 and put data we're awaiting
4:17 and we have the async methods here
4:19 let's run this and see what we get.
4:21 Notice we don't need time anymore.
4:25 Notice a very different behavior, look at this
4:30 we're generating the item and then we're going
4:32 to wait on something, waiting for more data to be generated
4:35 or delivered, so then we can immediately go and process it.
4:38 Look at that latency, 0, soon as an item is delivered
4:43 or almost as soon, we can go and pick it up.
4:46 It took 20 seconds instead of 30, that's pretty awesome.
4:51 Now we're having 0000 here because
4:54 the producer is not producing as fast
4:57 as the consumer can consume.
4:59 So let's take this and make one minor change here
5:01 let's generate one more of those, I call that a task2
5:05 that task3, and this one we need to tell it
5:08 it's expecting more data
5:10 we're not dealing with cancellation
5:11 and our other types of signals like that
5:13 not yet, task3, okay.
5:17 So now we should be producing more data than we can consume
5:20 and that should be even a little more interesting.
5:23 Notice the beginning it's kind of similar
5:24 but now we sometimes process two
5:27 sometimes we're processing one.
5:29 The latencies are still pretty good
5:31 but like I said, we can't quite keep up, so that's okay
5:35 let's see what happens at the end.
5:44 Yeah, pretty good, so in the beginning
5:46 we generated a couple
5:47 had to kind of do a couple double steps to catch up
5:50 but in the end we were more or less able
5:52 to keep up after we generated all the data
5:54 especially when one of them finished
5:56 and the other was still working.
5:58 Really, really cool, even generating twice as much data
6:02 it took only 22 milliseconds.
6:04 I believe that would have been 50, sorry 22 seconds
6:08 I believe that would have been 50 seconds
6:11 if we had done that synchronously. How cool is this?
6:14 So all we had to do is to break up these functions
6:18 into parts where we're waiting on something else
6:20 we're waiting on putting items into a queue
6:23 we're waiting on, well, time, but like I said
6:25 that's a stand-in for database access
6:28 file access, cache, network, etc.
6:32 Okay so we just break them up using this await keyword
6:35 we didn't have to change our programming model at all
6:37 that's the most important takeaway here.
6:40 We didn't have to, all the sudden
6:41 start off a thread, use callbacks
6:44 and other weird completion.
6:46 It's exactly the same code that you would normally write
6:49 except for you have async and you have these await sections.
6:52 So the programming model
6:53 is the standard synchronous model that you know
6:56 and that's really, really nice and important.
6:59 All right, that's a first introduction to asyncio.
7:03 We create the loop, we create some tasks based on
7:09 some coroutines that we're going to call
7:12 and then we run until they're complete.