Async Techniques and Examples in Python Transcripts
Chapter: async and await with asyncio
Lecture: Demo: Understanding basic generators
Login or
purchase this course
to watch this video and the rest of the course contents.
0:00
It's high time to write some code and much of what we are going to do going forward is to write demos and then think about how they worked
0:08
how we can change them, what were the key elements there. Let's begin by actually building a little of foundation. So, what we're going to do
0:15
is we're going to talk about using asyncio. Now, asyncio is quite magical in how it works but actually pretty easy in the programming model
0:24
that you're going to use. So, what I want to do is give you a conceptual foundation that's a little easier to understand
0:30
that doesn't actually have to do with concurrency. So, we're going to look at this idea of generator functions.
0:36
If you're familiar with the yield keyword you'll know what exactly I'm talking about. So, let's go over to our github repository.
0:42
We have 04 section and we have generators and we're going to move on to this producer and consumer thing. But, let's start with generators.
0:49
Now, the first thing I want to do is open this in PyCharm and configure this to have it's own virtual environment.
0:55
The generator example won't have external packages but, the other stuff will. So, let's get that all set up. We'll come over here
1:03
and we're going to create a virtual environment. Now, I have a little command that will run Python3 -m venv and then activate it
1:09
and then upgrade pip and setuptools. That all happened. Okay, so, now we can pip list see are Python has this virtual environment we just created here.
1:21
And in Windows, you would type where Python. Okay, so, we're going to hope this in PyCharm. So, here's our new virtual environment
1:29
and we can see that it actually is active. So, we have our Python that is active here. So, this is great. Now what we want to do
1:36
is we want to write a very simple generator. Why are we messing with generators? Why do we care about this at all?
1:45
Like, this is not asynchronous stuff. Generators don't have much to do with asynch programming or threading, or things like that.
1:52
Well, this idea of a coroutine or what you might think of as a restartable function is critical to how asyncio works.
2:00
And, I think it's a little easier to understand with generators. So, what we want to do is write a quick function called fib.
2:07
And, at first we're going to pass in an integer and it's going to return a list of integers. We can import that from typing.
2:16
So, we're going to take this and we're going to generate that many Fibonacci numbers. Totally simple. We're going to write it as a
2:23
straightforward, non-generator function. So, I'm going to return a list. We have to have some numbers. Put those here.
2:32
Those would be our Fibonacci numbers. And, then we're going to have current and next. These are going to be 0 and 1. The algorithm is super simple.
2:42
While, a line of numbers. Say next. Say it like this, current, next = next and then current + next then numbers.append(current).
2:58
We'll return a lot of numbers. Let's see if I wrote that right. So, let's print fib of the first ten Fibonacci numbers.
3:08
Go over here, I'm going to run this. And, let's just actually set each one of these are going to be separate code examples
3:14
so I will mark this directory as a source root. Doesn't look up above this for other files and we'll go and run this. 1, 1, 2, 3, 5, 8, 13, 21, 32, 55
3:26
Perfect, it looks like I managed to implement the Fibonacci numbers, once again. However, there's some problems here. Not very realistic.
3:34
The Fibonacci sequence is infinite, right? By passing in this number, we are working with a finite set. What if we want to work through these numbers
3:42
and go, I'm looking for the condition where the previous and current Fibonacci numbers excluding 1 and 2, are actually both prime numbers
3:52
may be greater than 1000. Who knows. Something like that. If you're looking for that situation you don't know how many to ask for.
3:59
What if you ask for 10,000. Well, that didn't work. Ask for 100,000. did that work? I don't know. Ask for 1,000,000. What will be better is
4:07
the consumer of this truly infinite sequence could actually decide when they've had enough. And, what we can do that with is what's called a generator.
4:15
So, I'm going to comment that one out. Actually, let's duplicate it first. I'm going to change this. First we're going to leave it like this here
4:26
except for we're going to say, no longer returns that. So, what we're going to do, is we're actually going to go through
4:33
instead of creating this list, right and, I can just make this true. Let's actually just make the whole sequence all at once. It's a little easier.
4:46
So, what we're going to do, is this. Now, we're never going to return numbers. Instead, what we're going to do is generate a sequence.
4:53
And, in Python, there's this really cool concept with this yield keyword of these coroutines. These generator methods.
5:00
The idea is, there are effectively restartable functions. So, what yield does, it lets you say I want to execute up to some point in a series
5:11
until I get to one and then I can give it back to whoever asked for it. They can look at it, work with it
5:15
and then decide whether this function should keep running. It's a totally wild idea, but it's entirely easy to write. So check this out.
5:22
We do this, we don't want to print all of them. So, now, we want to be a little more careful cuz that's an infinite sequence for n in fib(): print(n)
5:33
And say, this is going to be comma separated there. Then we'll say, if n > 10,000, break. Then break and done. So, now, we have an infinite sequence
5:47
that we're deciding when we want to stop pulling on it. Well, let's just run it and see what happens. Hmm. If we want one more print in there.
5:58
Here we go. 1, 1, 2, 3, 5. And, incredible, it just went until we decided to stop asking for it. How does that work?
6:05
So, understanding what is happening here is key to understanding how asyncio works. So, let's set a breakpoint. Actually, let's set it up here.
6:19
Set it right here. I'm also going to set a breakpoint right there. On a normal function, what would happen is we would hit this breakpoint.
6:29
This line would run and it would of course, go in here and execute all of this and then this code would run.
6:36
But, these are restartable functions, these generators. Just like the async coroutines. So, let's step through it and see what happens.
6:44
Here we are, we first ran to this and that makes perfect sense. Nothing surprising there. But, watch this breakpoint on line 14.
6:51
If we step over it didn't run. What we got back is not a sequence but is a generator that we can then iterate.
7:00
So, it's not until we start pulling on this generator that it even starts to run. So, I'm going to step a little bit.
7:07
I'm going to be over here in current notice. 0 and 1. This function is starting up. It's totally normal. Now, we're going to yield, which means
7:14
go back here. Oh, I need to use the step into. We're going to step in down here now n is a 1, the second 1 actually.
7:25
We go down here and we print it out. Now, we're going to go back into the function but look where we started. We started on line 16, not line 13
7:33
and the values have kept what they were doing before. So, I'm just going to run this a few times. Now if we step in again, again it's back here
7:42
and it's 3 and 5. These generator functions are restartable. They can run to a point and then stop and then resume. There's no parallelism going here
7:53
but you could build this cooperative parallelism or concurrency pretty straightforward with just generator functions, actually.
8:02
You just need to know how and when to say stop. So, let's go through just a little bit more until we get to here. So, now we have n at this point
8:14
and if we don't no longer loop over we break out of the loop we start pulling on the generator
8:20
and that basically abandons this continued execution of it. So, if we step, we're done, that's it. So, the last time we saw these run
8:30
they were, you know whatever the last two Fibonacci numbers were. And then, we just stop pulling on it it stopped doing it's work.
8:35
We didn't restart it again and this example is over. So, of course I'm going to put this in your code. You can play with it.
8:43
There's nothing that hard about the actual written code it's the order and flow of the execution that is really special that you
8:51
kind of, get a little bit used to. So, I encourage you to take moment and open this up in PyCharm or something like that that has a decent debugger
8:59
and just go through the steps a few times and see what's happening. Once you get comfortable with that we're ready to talk about asyncio.