Async Techniques and Examples in Python Transcripts
Chapter: Common APIs with execution pools
Lecture: Demo: Executor app (threaded-edition)
0:00 Now you're familiar with the program
0:01 in its synchronous form let's make it asynchronous.
0:04 And that let's me introduce an entirely new module
0:07 that is built into Python.
0:08 So these two are external packages
0:10 but what we're going to work with now comes with Python.
0:12 So import concurrent.futures
0:17 and actually we want to get something out of it
0:19 we'll say from that import Future.
0:23 So this Future thing here
0:25 this is a class that's going
0:27 to represent some asynchronous computation.
0:31 It's going to be the return value
0:32 of starting one of these asynchronous operations
0:35 and it doesn't matter if it's a process
0:37 or a thread that it represents.
0:39 So that's a really great thing.
0:41 We saw that we have this concept of a pool
0:43 one of these multiprocessing pools before.
0:46 And the idea of a pool for either threads
0:48 or processes is really great.
0:50 They take a little bit of work to spin up and get started.
0:53 It'd be great if they could just run
0:55 and then we could assign work to them that they pick up
0:57 and go with. So that's the general idea here.
1:00 So we're going to say from concurrent.futures.threading
1:05 or rather thread, import ThreadPoolExecutor.
1:09 Now, I can say this and just leave it like this
1:12 but let me add a little bit as PoolExecutor.
1:16 So that'll let us reassign what this type means.
1:19 That's going to be really handy.
1:21 So what we're going to do is we're going to go create one of these
1:23 kind of like we did with the process pool in multiprocessing.
1:25 So down here, instead of just calling this work
1:29 let's assign the work to be done.
1:32 So we'll go like this and we're going to use this
1:33 in a context block, a with manager
1:35 so we'll say PoolExecutor() as executor.
1:40 Okay, create a variable like that
1:41 and we're going to work with it.
1:43 Then we're going to do all this work here like so.
1:46 This line, instead of doing this
1:48 we're going to come over here and say f
1:50 that's going to be a Future
1:52 we're going to say executor.submit().
1:56 The work that we want to submit
1:58 is we're going to tell the function, that's the f in
2:00 it's going to be that
2:01 and the args is a *args so we can just pass them
2:04 one after another, which is pretty handy.
2:06 So we'll just say url. Alright, that's pretty cool.
2:10 And then we can't now print out the work just yet.
2:13 We'll do that in a little bit.
2:15 Like before, we may want to get a hold of these answers
2:17 after they're done.
2:18 So let's go over here and say work, it's like this
2:23 and work on f is this future. Okay, so, that's it.
2:28 We're going to kick off all of this work
2:29 maybe do a little print statement
2:31 like waiting for downloads. Just so you know
2:37 we've already started all of the work.
2:40 And then when it's done
2:41 we're going to leave that with block
2:43 on line 28, between 28 and 29.
2:45 There's not really, I guess
2:46 there's like kind of the end right there, huh?
2:48 When we leave that with block
2:50 everything's going to be great.
2:51 We're going to be ready to roll.
2:53 Here we go, executor. Let me spell that better.
2:57 Alright, so then we can just print out what we did before.
3:00 Say for so for f in work:
3:03 and then we can print almost what we had here
3:05 but instead of title we have our future
3:08 and we can say go to it and it has a result.
3:10 Notice it has all sorts of things, right?
3:12 It has cancel, cancel done, whether there's an exception
3:16 whether it's running, all kinds of great stuff
3:18 add a callback when it's finished.
3:19 But we're just interested in the result
3:22 which we could give a timeout or not.
3:23 So that's going to print out the titles.
3:25 Well, it's going to whatever get_title() returns
3:28 and right now it's returning the title.
3:30 Okay, so, here's our pool executor.
3:33 Looks like it's ready to go.
3:34 I think we've got everything going
3:37 and we aren't doing any print stuff.
3:38 We're going to add a little trace here
3:39 just so we can figure out what's going
3:41 a little bit better in just a second.
3:42 But let's just run it and see if it works. Running.
3:45 Guess I'll watch the titles and bam! Look at that!
3:48 Now, our formatting wasn't so great.
3:53 Let's put this down here.
3:57 Actually, I'll leave it here for you like this.
4:00 You can uncomment it.
4:02 And let's put that down here just so we have
4:04 a printing out.
4:05 And we're going to put a proper end on it.
4:07 See, if we get them all at once
4:09 and then all of the results come back.
4:11 So make it a little bigger, maybe, so you can see it run through it's entirety.
4:15 Now let's run it one more time.
4:16 Getting all the stuff waiting done.
4:18 So, start the work, waiting, done
4:21 and then here are the answers.
4:23 We don't get one back from Google, remember.
4:25 Now, we're not tying these together.
4:27 It would be great if we had a way
4:28 to figure out what the url was.
4:31 Oh, and I noticed we could probably improve
4:33 on that a little.
4:35 If we could figure out what the domain was
4:37 and print it here. But we could do it
4:39 it'd just be a little bit changing the return value
4:41 which I don't know that we're going to mess with.
4:43 But would be nice to see.
4:44 Nonetheless, this is working.
4:46 The question is where is it working?
4:49 What thread is it running on? Or is it even a thread?
4:52 Is it maybe another process? So, let's go over here.
4:56 I'm going to add a little debugging.
4:58 I'm going to add a little debugging here.
5:00 Now, I'm going to import something inside this function.
5:03 Not normally recommended but it's just
5:05 a temporary sort of trace diagnostics thing
5:08 we're going to comment out so it doesn't clutter up
5:10 the rest of the file. I'm going to import multiprocessing.
5:13 'Cause we actually need that.
5:16 So we can say, and get the current process.
5:18 So we'll say p = multiprocessing.current_process()
5:23 And then we can use that process
5:25 to make some statements about this.
5:27 So we can say getting title form that
5:28 and we can list out the process ID and the process name
5:33 and we already have the url.
5:34 So let's go down here and wrap this format
5:36 a little bit and this is easy
5:38 pid and p.name, okay.
5:44 And let's just tell PyCharm everything's cool.
5:46 Okay, so let's run it one more time
5:48 and just see what we get.
5:50 You can see we're getting all of these titles
5:52 and got some sort of race condition on our endline
5:58 that's okay though. So we got all of this coming through
6:02 and you'll notice the process is all the same.
6:04 It's always 66203 for the moment.
6:07 Won't be the same every time
6:08 but it should be consistent for a run, right?
6:10 And it's main process.
6:12 So that means this get title is running
6:14 on a bunch of different threads in the same process.
6:18 That's not surprising 'cause we came over here
6:21 and we said we're going to use the ThreadPoolExecutor, right?
6:25 Pretty cool. And we can make this a little more explicit.
6:29 Make this a, mark that as a Future
6:31 that's why I import it, so.
6:33 Make sure the auto complete works.
6:34 Sometimes it does, sometimes it doesn't.
6:37 This will be sure to help in your editor.
6:39 Just that requires Python 3.6, so be aware.
6:41 I think this is working for our threaded version.