Modern APIs with FastAPI and Python Transcripts
Chapter: Modern language foundations
Lecture: Async web scraper
Login or
purchase this course
to watch this video and the rest of the course contents.
0:00
Well, we've seen the traditional synchronous style with requests.
0:03
How's this look if we bring in something new and fancy? And the big
0:07
fancy new thing we're gonna bring in is "httpx". So this is much like requests
0:13
in the way you work with it,
0:14
but it actually supports async and await.
0:16
So let's go over here and just see how we're using it.
0:18
First of all, one thing that's pretty interesting is if we wrote code like, this
0:24
was a synchronous, but it looked just exactly the same as before,
0:28
you would see it's not so interesting. If I come over here and I run the
0:33
old one, let's just run this real quick,
0:34
you can see, it's doing exactly the same thing.
0:37
It's getting an item, give you an answer,
0:38
get the next item, giving the next answer,
0:40
taking five seconds. But the reason is we're starting off one of these calls,
0:46
and we're waiting for it. So our first thing we've got to realize is what
0:49
we need to do is kick off a bunch of these asynchronous tasks here,
0:53
right? So when you go to get the titles,
0:55
we're gonna start all the tasks, so we're gonna go to this thing called an
0:58
"asyncio event loop", you don't need to worry about this in FastAPI
1:01
but when you work with it directly,
1:03
you do. We're gonna create a task from some asynchronous function,
1:06
right? This is stuff that FastAPI does for us.
1:09
We're going to start them off and then we're gonna start them all going,
1:12
and then we're gonna wait for them to finish and just process them in the order we
1:16
started them. But, you know,
1:17
as they all basically are working in parallel.
1:20
So let's go see this "get_html" function.
1:23
Instead of being a regular function, like this,
1:25
it's an async function with the word async out front,
1:28
which means we have the ability to use async and await within it.
1:31
So down here we have this, not a regular with block,
1:35
but a async with block from working with this httpx client.
1:40
And then anytime we call something asynchronous, like if we do this
1:43
get here, what we're gonna get back is not the answer,
1:47
not the response, but a task or a co-routine that,
1:51
when finished, would give us the response. And the way to say
1:54
make the program give up its execution to anything else that needs to run while we're
1:59
waiting and then give me the answer
2:00
when it's done, is that right there.
2:03
We just say "await this asynchronous call" and then it's up to the program to
2:08
figure out, well, while we're waiting on this,
2:10
maybe we could start a bunch of other calls.
2:12
Maybe we could be processing the responses from other clients as those API calls get
2:16
done or those http calls get done.
2:19
So this basically tells the runtime, tells Python,
2:22
look, "I'm waiting on this stuff.
2:24
Pick me up when it's done",
2:26
but in the meantime, you can go and run other code without even using threads.
2:29
So this is really the magic line right there.
2:32
And then we check to make sure it's ok,
2:34
and we return the text. Remember,
2:35
it was going "click, click, click,
2:37
request, request, request, response, response" when we ran it before about every five seconds.
2:41
How does it look now? They all start, they're all done. Just like that.
2:45
It was five seconds before. Now it's less than one second.
2:48
Let's run it a few times.
2:49
They all start, They're all done. Isn't that awesome?
2:52
Yeah, it puts a little bit more load on the server,
2:54
but mostly we're just doing all the work while we're waiting,
2:58
you know, waiting on this stuff to make it through the Internet.
3:00
Right? So this is really,
3:02
really cool. It is so much faster,
3:04
right? It's six times faster.
3:06
Well not that time. Right, there
3:08
we go. Six times faster or so this time around,
3:11
and all we gotta to do is write these async functions.
3:14
When we had a regular call that happened to be async,
3:17
we just put await in front,
3:19
and now it kind of goes back to the way it was before,
3:21
and that's it. I mean,
3:22
let's just compare real quick. We're not using a client session on requests,
3:27
so it's slightly different. But we have this, seeing
3:31
a callback URL raise for status and return text.
3:34
And now we have, oh wait.
3:36
Call URL, raise for status, return the text.
3:39
It just happens to be we're using this async client to kind of put that
3:41
all together. Okay, cool
3:43
right? So this is the async foundation,
3:45
and you can just see it's so much faster,
3:48
so much nicer, handling these requests.
3:50
And this is the same way that our Web app is gonna work instead of being
3:53
able to handle just one request at a time it could do 10, 20, 100 requests concurrently
3:58
while maybe it's working with other
4:01
API's, talking to databases, all those things. And working with async and await is
4:05
what's required to unlock that potential.