Modern APIs with FastAPI and Python Transcripts
Chapter: Modern language foundations
Lecture: Async web scraper
Login or
purchase this course
to watch this video and the rest of the course contents.
0:00
Well, we've seen the traditional synchronous style with requests. How's this look if we bring in something new and fancy? And the big
0:08
fancy new thing we're gonna bring in is "httpx". So this is much like requests in the way you work with it, but it actually supports async and await.
0:17
So let's go over here and just see how we're using it. First of all, one thing that's pretty interesting is if we wrote code like, this
0:25
was a synchronous, but it looked just exactly the same as before, you would see it's not so interesting. If I come over here and I run the
0:34
old one, let's just run this real quick, you can see, it's doing exactly the same thing. It's getting an item, give you an answer,
0:39
get the next item, giving the next answer, taking five seconds. But the reason is we're starting off one of these calls,
0:47
and we're waiting for it. So our first thing we've got to realize is what we need to do is kick off a bunch of these asynchronous tasks here,
0:54
right? So when you go to get the titles, we're gonna start all the tasks, so we're gonna go to this thing called an
0:59
"asyncio event loop", you don't need to worry about this in FastAPI but when you work with it directly,
1:04
you do. We're gonna create a task from some asynchronous function, right? This is stuff that FastAPI does for us.
1:10
We're going to start them off and then we're gonna start them all going,
1:13
and then we're gonna wait for them to finish and just process them in the order we started them. But, you know,
1:18
as they all basically are working in parallel. So let's go see this "get_HTML" function. Instead of being a regular function, like this,
1:26
it's an async function with the word async out front, which means we have the ability to use async and await within it.
1:32
So down here we have this, not a regular with block, but a async with block from working with this httpx client.
1:41
And then anytime we call something asynchronous, like if we do this get here, what we're gonna get back is not the answer,
1:48
not the response, but a task or a co-routine that, when finished, would give us the response. And the way to say
1:55
make the program give up its execution to anything else that needs to run while we're waiting and then give me the answer
2:01
when it's done, is that right there. We just say "await this asynchronous call" and then it's up to the program to
2:09
figure out, well, while we're waiting on this, maybe we could start a bunch of other calls.
2:13
Maybe we could be processing the responses from other clients as those API calls get done or those http calls get done.
2:20
So this basically tells the runtime, tells Python, look, "I'm waiting on this stuff. Pick me up when it's done",
2:27
but in the meantime, you can go and run other code without even using threads. So this is really the magic line right there.
2:33
And then we check to make sure it's ok, and we return the text. Remember, it was going "click, click, click,
2:38
request, request, request, response, response" when we ran it before about every five seconds.
2:42
How does it look now? They all start, they're all done. Just like that. It was five seconds before. Now it's less than one second.
2:49
Let's run it a few times. They all start, They're all done. Isn't that awesome? Yeah, it puts a little bit more load on the server,
2:55
but mostly we're just doing all the work while we're waiting, you know, waiting on this stuff to make it through the Internet.
3:01
Right? So this is really, really cool. It is so much faster, right? It's six times faster. Well not that time. Right, there
3:09
we go. Six times faster or so this time around, and all we gotta to do is write these async functions.
3:15
When we had a regular call that happened to be async, we just put await in front, and now it kind of goes back to the way it was before,
3:22
and that's it. I mean, let's just compare real quick. We're not using a client session on requests,
3:28
so it's slightly different. But we have this, seeing a callback URL raise for status and return text. And now we have, oh wait.
3:37
Call URL, raise for status, return the text. It just happens to be we're using this async client to kind of put that all together. Okay, cool
3:44
right? So this is the async foundation, and you can just see it's so much faster, so much nicer, handling these requests.
3:51
And this is the same way that our Web app is gonna work instead of being
3:54
able to handle just one request at a time it could do 10, 20, 100 requests concurrently while maybe it's working with other
4:02
API's, talking to databases, all those things. And working with async and await is what's required to unlock that potential.