Async Techniques and Examples in Python Transcripts
Chapter: async and await with asyncio
Lecture: Demo: Understanding basic generators
0:00 It's high time to write some code
0:02 and much of what we are going to do going forward
0:04 is to write demos and then think about how they worked
0:07 how we can change them, what were the key elements there.
0:10 Let's begin by actually building a little of foundation.
0:13 So, what we're going to do
0:14 is we're going to talk about using asyncio.
0:17 Now, asyncio is quite magical in how it works
0:20 but actually pretty easy in the programming model
0:23 that you're going to use.
0:25 So, what I want to do is give you a conceptual foundation
0:27 that's a little easier to understand
0:29 that doesn't actually have to do with concurrency.
0:32 So, we're going to look at this idea of generator functions.
0:35 If you're familiar with the yield keyword
0:37 you'll know what exactly I'm talking about.
0:39 So, let's go over to our github repository.
0:41 We have 04 section and we have generators
0:44 and we're going to move on to this producer and consumer thing.
0:46 But, let's start with generators.
0:48 Now, the first thing I want to do is open this
0:50 in PyCharm and configure this to have it's own
0:53 virtual environment.
0:54 The generator example won't have external packages
0:56 but, the other stuff will. So, let's get that all set up.
1:01 We'll come over here
1:02 and we're going to create a virtual environment.
1:04 Now, I have a little command that will run
1:05 python3 -m venv and then activate it
1:08 and then upgrade pip and setuptools.
1:11 That all happened. Okay, so, now we can pip list see are
1:16 Python has this virtual environment we just created here.
1:20 And in Windows, you would type where python.
1:23 Okay, so, we're going to hope this in PyCharm.
1:26 So, here's our new virtual environment
1:28 and we can see that it actually is active.
1:30 So, we have our python that is active here.
1:33 So, this is great. Now what we want to do
1:35 is we want to write a very simple generator.
1:41 Why are we messing with generators?
1:42 Why do we care about this at all?
1:44 Like, this is not asynchronous stuff.
1:45 Generators don't have much to do with
1:48 asynch programming or threading, or things like that.
1:51 Well, this idea of a coroutine
1:54 or what you might think of as a restartable function
1:57 is critical to how asyncio works.
1:59 And, I think it's a little easier to understand
2:01 with generators.
2:02 So, what we want to do is write a quick function
2:03 called fib.
2:06 And, at first we're going to pass in an integer
2:08 and it's going to return a list of integers.
2:12 We can import that from typing.
2:15 So, we're going to take this and we're going to generate
2:17 that many Fibonacci numbers. Totally simple.
2:21 We're going to write it as a
2:22 straightforward, non-generator function.
2:26 So, I'm going to return a list.
2:27 We have to have some numbers.
2:29 Put those here.
2:31 Those would be our Fibonacci numbers.
2:33 And, then we're going to have current and next.
2:38 These are going to be 0 and 1.
2:40 The algorithm is super simple.
2:41 While, a line of numbers.
2:45 Say next. Say it like this, current, next
2:48 = next and then current + next
2:52 then numbers.append(current).
2:57 We'll return a lot of numbers.
2:59 Let's see if I wrote that right.
3:03 So, let's print fib of the first ten Fibonacci numbers.
3:07 Go over here, I'm going to run this.
3:09 And, let's just actually set
3:10 each one of these are going to be separate code examples
3:13 so I will mark this directory as a source root.
3:16 Doesn't look up above this for other files
3:18 and we'll go and run this.
3:21 1, 1, 2, 3, 5, 8, 13, 21, 32, 55
3:25 Perfect, it looks like I managed to implement
3:27 the Fibonacci numbers, once again.
3:29 However, there's some problems here.
3:31 Not very realistic.
3:33 The Fibonacci sequence is infinite, right?
3:36 By passing in this number, we are working with a finite set.
3:39 What if we want to work through these numbers
3:41 and go, I'm looking for the condition
3:43 where the previous and current Fibonacci numbers
3:47 excluding 1 and 2, are actually both prime numbers
3:51 may be greater than 1000.
3:52 Who knows. Something like that.
3:54 If you're looking for that situation
3:56 you don't know how many to ask for.
3:58 What if you ask for 10,000.
3:59 Well, that didn't work. Ask for 100,000.
4:01 Uh, did that work? I don't know.
4:03 Ask for 1,000,000. What will be better is
4:06 the consumer of this truly infinite sequence
4:09 could actually decide when they've had enough.
4:11 And, what we can do that with
4:13 is what's called a generator.
4:14 So, I'm going to comment that one out.
4:16 Actually, let's duplicate it first.
4:21 I'm going to change this.
4:22 First we're going to leave it like this here
4:25 except for we're going to say, no longer returns that.
4:29 So, what we're going to do, is we're actually going to go through
4:32 instead of creating this list, right
4:37 and, I can just make this true.
4:41 Let's actually just make the whole sequence
4:43 all at once. It's a little easier.
4:45 So, what we're going to do, is this.
4:48 Now, we're never going to return numbers.
4:50 Instead, what we're going to do is generate a sequence.
4:52 And, in Python, there's this really cool concept
4:54 with this yield keyword of these coroutines.
4:58 These generator methods.
4:59 The idea is, there are effectively restartable functions.
5:02 So, what yield does, it lets you say
5:06 I want to execute up to some point in a series
5:10 until I get to one
5:11 and then I can give it back to whoever asked for it.
5:12 They can look at it, work with it
5:14 and then decide whether this function should keep running.
5:18 It's a totally wild idea, but it's entirely easy to write.
5:20 So check this out.
5:21 We do this, we don't want to print all of them.
5:24 So, now, we want to be a little more careful
5:25 cuz that's an infinite sequence
5:27 for n in fib(): print(n)
5:32 And say, this is going to be comma separated there.
5:34 Then we'll say, if n > 10,000, break.
5:40 Then break and done.
5:43 So, now, we have an infinite sequence
5:46 that we're deciding when we want to stop
5:48 pulling on it.
5:49 Well, let's just run it and see what happens.
5:52 Hmm. If we want one more print in there.
5:57 Here we go. 1, 1, 2, 3, 5.
6:00 And, incredible, it just went until we decided
6:02 to stop asking for it. How does that work?
6:04 So, understanding what is happening here
6:06 is key to understanding how asyncio works.
6:08 So, let's set a breakpoint.
6:10 Actually, let's set it up here.
6:18 Set it right here.
6:21 I'm also going to set a breakpoint right there.
6:24 On a normal function, what would happen
6:26 is we would hit this breakpoint.
6:28 This line would run and it would
6:30 of course, go in here and execute all of this
6:33 and then this code would run.
6:35 But, these are restartable functions, these generators.
6:37 Just like the async coroutines.
6:39 So, let's step through it and see what happens.
6:43 Here we are, we first ran to this
6:45 and that makes perfect sense.
6:46 Nothing surprising there.
6:48 But, watch this breakpoint on line 14.
6:50 If we step over it didn't run.
6:55 What we got back is not a sequence
6:56 but is a generator that we can then iterate.
6:59 So, it's not until we start pulling on this generator
7:02 that it even starts to run.
7:03 So, I'm going to step a little bit.
7:06 I'm going to be over here in current notice.
7:08 0 and 1. This function is starting up.
7:10 It's totally normal.
7:12 Now, we're going to yield, which means
7:13 go back here.
7:17 Oh, I need to use the step into.
7:19 We're going to step in down here
7:21 now n is a 1, the second 1 actually.
7:24 We go down here and we print it out.
7:26 Now, we're going to go back into the function
7:28 but look where we started.
7:29 We started on line 16, not line 13
7:32 and the values have kept what they were doing before.
7:34 So, I'm just going to run this a few times.
7:39 Now if we step in again, again it's back here
7:41 and it's 3 and 5.
7:43 These generator functions are restartable.
7:46 They can run to a point and then stop
7:48 and then resume. There's no parallelism going here
7:52 but you could build this cooperative
7:55 parallelism or concurrency pretty straightforward
7:58 with just generator functions, actually.
8:01 You just need to know how and when to say stop.
8:04 So, let's go through just a little bit more
8:05 until we get to here.
8:12 So, now we have n at this point
8:13 and if we don't no longer loop over
8:16 we break out of the loop
8:18 we start pulling on the generator
8:19 and that basically abandons this continued execution of it.
8:23 So, if we step, we're done, that's it.
8:27 So, the last time we saw these run
8:29 they were, you know
8:30 whatever the last two Fibonacci numbers were.
8:32 And then, we just stop pulling on it
8:33 it stopped doing it's work.
8:34 We didn't restart it again and this example is over.
8:39 So, of course I'm going to put this in your code.
8:41 You can play with it.
8:42 There's nothing that hard about the actual written code
8:45 it's the order and flow of the execution
8:48 that is really special that you
8:50 kind of, get a little bit used to.
8:53 So, I encourage you to take moment
8:54 and open this up in PyCharm or something like that
8:56 that has a decent debugger
8:58 and just go through the steps a few times
9:00 and see what's happening.
9:01 Once you get comfortable with that
9:02 we're ready to talk about asyncio.