Async Techniques and Examples in Python Transcripts
Chapter: Common APIs with execution pools
Lecture: Demo: Executor app (threaded-edition)
Login or
purchase this course
to watch this video and the rest of the course contents.
0:00
Now you're familiar with the program in its synchronous form let's make it asynchronous. And that let's me introduce an entirely new module
0:08
that is built into Python. So these two are external packages but what we're going to work with now comes with Python. So import concurrent.futures
0:18
and actually we want to get something out of it we'll say from that import Future. So this Future thing here this is a class that's going
0:28
to represent some asynchronous computation. It's going to be the return value of starting one of these asynchronous operations
0:36
and it doesn't matter if it's a process or a thread that it represents. So that's a really great thing. We saw that we have this concept of a pool
0:44
one of these multiprocessing pools before. And the idea of a pool for either threads or processes is really great.
0:51
They take a little bit of work to spin up and get started. It'd be great if they could just run and then we could assign work to them that they pick up
0:58
and go with. So that's the general idea here. So we're going to say from concurrent.futures.threading or rather thread, import ThreadPoolExecutor.
1:10
Now, I can say this and just leave it like this but let me add a little bit as PoolExecutor. So that'll let us reassign what this type means.
1:20
That's going to be really handy. So what we're going to do is we're going to go create one of these
1:24
kind of like we did with the process pool in multiprocessing. So down here, instead of just calling this work let's assign the work to be done.
1:33
So we'll go like this and we're going to use this in a context block, a with manager so we'll say PoolExecutor() as executor.
1:41
Okay, create a variable like that and we're going to work with it. Then we're going to do all this work here like so. This line, instead of doing this
1:49
we're going to come over here and say f that's going to be a Future we're going to say executor.submit(). The work that we want to submit
1:59
is we're going to tell the function, that's the f in it's going to be that and the args is a *args so we can just pass them
2:05
one after another, which is pretty handy. So we'll just say url. Alright, that's pretty cool. And then we can't now print out the work just yet.
2:14
We'll do that in a little bit. Like before, we may want to get a hold of these answers after they're done.
2:19
So let's go over here and say work, it's like this and work on f is this future. Okay, so, that's it. We're going to kick off all of this work
2:30
maybe do a little print statement like waiting for downloads. Just so you know we've already started all of the work. And then when it's done
2:42
we're going to leave that with block on line 28, between 28 and 29. There's not really, I guess there's like kind of the end right there, huh?
2:49
When we leave that with block everything's going to be great. We're going to be ready to roll. Here we go, executor. Let me spell that better.
2:58
Alright, so then we can just print out what we did before. Say for so for f in work: and then we can print almost what we had here
3:06
but instead of title we have our future and we can say go to it and it has a result. Notice it has all sorts of things, right?
3:13
It has cancel, cancel done, whether there's an exception whether it's running, all kinds of great stuff add a callback when it's finished.
3:20
But we're just interested in the result which we could give a timeout or not. So that's going to print out the titles.
3:26
Well, it's going to whatever get_title() returns and right now it's returning the title. Okay, so, here's our pool executor.
3:34
Looks like it's ready to go. I think we've got everything going and we aren't doing any print stuff. We're going to add a little trace here
3:40
just so we can figure out what's going a little bit better in just a second. But let's just run it and see if it works. Running.
3:46
Guess I'll watch the titles and bam! Look at that! Now, our formatting wasn't so great. Let's put this down here.
3:58
Actually, I'll leave it here for you like this. You can uncomment it. And let's put that down here just so we have a printing out.
4:06
And we're going to put a proper end on it. See, if we get them all at once and then all of the results come back.
4:12
So make it a little bigger, maybe, so you can see it run through it's entirety. Now let's run it one more time. Getting all the stuff waiting done.
4:19
So, start the work, waiting, done and then here are the answers. We don't get one back from Google, remember. Now, we're not tying these together.
4:28
It would be great if we had a way to figure out what the url was. Oh, and I noticed we could probably improve on that a little.
4:36
If we could figure out what the domain was and print it here. But we could do it it'd just be a little bit changing the return value
4:42
which I don't know that we're going to mess with. But would be nice to see. Nonetheless, this is working. The question is where is it working?
4:50
What thread is it running on? Or is it even a thread? Is it maybe another process? So, let's go over here. I'm going to add a little debugging.
4:59
I'm going to add a little debugging here. Now, I'm going to import something inside this function. Not normally recommended but it's just
5:06
a temporary sort of trace diagnostics thing we're going to comment out so it doesn't clutter up
5:11
the rest of the file. I'm going to import multiprocessing. 'Cause we actually need that. So we can say, and get the current process.
5:19
So we'll say p = multiprocessing.current_process() And then we can use that process to make some statements about this.
5:28
So we can say getting title form that and we can list out the process ID and the process name and we already have the url.
5:35
So let's go down here and wrap this format a little bit and this is easy pid and p.name, okay. And let's just tell PyCharm everything's cool.
5:47
Okay, so let's run it one more time and just see what we get. You can see we're getting all of these titles
5:53
and got some sort of race condition on our endline that's okay though. So we got all of this coming through
6:03
and you'll notice the process is all the same. It's always 66203 for the moment. Won't be the same every time
6:09
but it should be consistent for a run, right? And it's main process. So that means this get title is running
6:15
on a bunch of different threads in the same process. That's not surprising 'cause we came over here
6:22
and we said we're going to use the ThreadPoolExecutor, right? Pretty cool. And we can make this a little more explicit.
6:30
Make this a, mark that as a Future that's why I import it, so. Make sure the auto complete works. Sometimes it does, sometimes it doesn't.
6:38
This will be sure to help in your editor. Just that requires Python 3.6, so be aware. I think this is working for our threaded version.