Python 3.11: A Guided Tour Through Code Transcripts
Chapter: Concurrency Improvements
Lecture: TaskGroup Demo
Login or
purchase this course
to watch this video and the rest of the course contents.
0:00
Well let's get started over here. We'll write some code to explore these task groups. And let's give it the name groups for our file Again.
0:09
We're gonna start with some existing code and tweak it with this concept because I want it to be something realistic
0:14
So what's the deal with this code sample here? What we're gonna do is we're gonna search two different podcasts.
0:21
We're gonna search talk python and we're going to search python bytes. Let's jump over there real quick. So for example,
0:28
let's say talk python. Now, both the podcasts have great search features. So I could search for pydantic and odm we get a bunch of results here.
0:38
So we get the interview I did with Sam Colvin Colvin on pydantic,
0:44
the updated version 2 some stuff on Sql model, which is the new Sql Alchemy paired thing with FastAPI That's also based on pydantic.
0:56
Right? So we're getting these results. But if you look carefully, there is a want to search with our API
1:00
There we go and we can search the same thing over here. Alright, so it's taken us back. Hold on,
1:10
you click this it'll show you give you some information the keyword you search for and then all these results and
1:16
whether it's an episode or it is a transcript. So in this code sample, what we're gonna do is we're gonna use this search podcast function up here.
1:28
So we've defined this search response which is a pydantic model that will allow us to parse that response.
1:35
You just saw elapsed time keywords, the results which is a list of search items and broken out by just
1:40
episodes, search items, list as well. We've got that code up here, that's totally fine. Notice we're using pydantic and HTTPX.
1:49
We don't have those available to us yet. We'll have them shortly.
1:53
This cleanup function is just unifies the two different search API's don't worry about that.
1:57
Here's where the important stuff is. So we're gonna search the podcast and this is an async feature so
2:02
we can use httpX to asynchronously open the socket,
2:07
we'll close it and then we can await asynchronously getting the search results that Json we're gonna parse it using
2:14
pydantic here and then return it as a list. We're going to do that for talk python and we're gonna
2:19
do that for python bytes. What is the relationships between those two shows besides the fact that I happen to
2:26
work host both of them or co host one of them host the other one. Nothing. There are separate servers. They're separate web apps.
2:33
They're separate search engines so we should be able to search them both at the same time and combine those results
2:38
A perfect use for starting a bunch of tasks that can all run independently bringing back the results when they're
2:45
all done. Alright. So here we are, we're going to go create this task and we're gonna call search and we're gonna try to print out the results.
2:54
How well is this going to go not to Well, first of all, because we don't have the dependencies.
3:00
We don't know what httpX is. So let's first add a requirements for the entire project entire course here.
3:07
So requirements probably thought I was gonna type txt No, no no. Let me show you something cool. This is way better.
3:15
I'm gonna put a requirement.in. And what goes into this file here is just the base dependency. So we have httpx and we have pydantic.
3:26
You want to put those two in here just like that? No versions or anything. And what I would like is a requirements.txt file that has all of the
3:35
dependencies. Not just those two but its dependencies and their transitive closure of their dependencies all pinned with the latest
3:42
compatible version. So what we're gonna do is we're going to come over here,
3:46
I want to say pip, install pip pools, which is a really awesome library to manage exactly that.
3:53
Then we're gonna say pip-compile give it this requirements in and say upgrade.
3:58
It will require generate a requirements.txt. Let's see what we get look at this over here.
4:07
Perfect. So we've got anyio why do we have that because of http core. Http core, why do we have that? We have that because of httpX,
4:15
you can see they're all pinned versions. And if new dependencies come out we can run that command again and
4:20
it'll upgrade the ones that got an upgrade. Even the dependencies of the dependencies.
4:25
So highly recommend pip tools and pip compile here. We still need to get the dependencies.
4:35
So let's do that real quick. Alright. We should be good to go and this warning will go away in a sec. Alright, so this up here looks pretty good.
4:47
Let's go try and run it again. It's still not gonna work. This is going to start the task but it's not going to go well what search do you want to look
4:57
for now? It suggests that this is a search term you can use or set of keywords that will generate
5:03
some results, but not an insanely large one. So let's go with that whack. asyncio feature is in an invalid state. The result is not set.
5:15
Boy, if I were designing asyncio, it would be so different. So one of the things that's annoying, let's say about async Io is what I would like to see
5:25
is like if this is running and you can't give me a result yet, just stop and wait for that to result to finish and then give me the results,
5:33
basically turn that into a blocking call. That's not how it works. If you don't wait for this in some other way it's going to crash,
5:41
and this gets worse. If this thing internally also generates other tasks like you kind of got to build
5:48
this up as a hierarchy. So one way to do this is we can come over here and we can
5:53
say results equals asyncio.gather and give it an arbitrary number of tasks. And this thing itself is in a task that we can await.
6:05
So when this runs we should be able to pretty print out the results and let me just return real quick So we just see that. Try the same search again.
6:18
Awesome. Look at that. So we get here's the results for talk python and here's the results for python
6:24
bytes because the order in which we gathered them. Well, that's cool. But what if after that one started and this one hadn't quite got going yet?
6:33
I decided I want to cancel everything or this one has an error. We don't need the other work to keep going because there's a problem.
6:39
We just got to stop and deal with it. Well, none of that at all addressed here. Okay, so we have these two problems.
6:46
One just running a block of tasks and waiting for all of them including their Children.
6:51
And the other one is cancelation and error handling. And that brings us to task groups. They're super easy to use and their paradigm is very normal,
7:01
very approachable for a python person basically. Do you know how to use a context manager?
7:06
aka with block. So instead of with we say async with task group as tg. Instead of saying asyncio
7:16
create task we say task group create task that associates it with this particular task group.
7:24
And if that thing internally needed to make some tasks we could pass the tax group to it as well
7:29
and it could change those on. Right Okay. So we took away the await.
7:33
We're just gonna run the task group. When we go from line 57-58 we'll know that either there's an exception
7:41
which case we're out of here or all those tasks have completed. Let's try it again. Back to searching for this beautiful search term.
7:53
They like being a mongo dB orm go ah yeah look at that. So it ran it waited for them both to finish because guess what?
8:03
There are a group of tasks working together and then down here both the task.
8:08
None of this weird extra gathering to get the results and then working with the task results later.
8:14
So very very nice. Again we're just touching the surface here. We're not using cancelation, we're not using error handling or anything like that.
8:23
We're just using in a straightforward way but still really nice to be able to run these as a group and then wait just for them to finish.