Modern APIs with FastAPI and Python Transcripts
Chapter: Modern language foundations
Lecture: Async web scraper

Learn more about this course
Login or purchase this course to watch this video and the rest of the course contents.
0:00 Well, we've seen the traditional synchronous style with requests.
0:03 How's this look if we bring in something new and fancy? And the big
0:07 fancy new thing we're gonna bring in is "httpx". So this is much like requests
0:13 in the way you work with it,
0:14 but it actually supports async and await.
0:16 So let's go over here and just see how we're using it.
0:18 First of all, one thing that's pretty interesting is if we wrote code like, this
0:24 was a synchronous, but it looked just exactly the same as before,
0:28 you would see it's not so interesting. If I come over here and I run the
0:33 old one, let's just run this real quick,
0:34 you can see, it's doing exactly the same thing.
0:37 It's getting an item, give you an answer,
0:38 get the next item, giving the next answer,
0:40 taking five seconds. But the reason is we're starting off one of these calls,
0:46 and we're waiting for it. So our first thing we've got to realize is what
0:49 we need to do is kick off a bunch of these asynchronous tasks here,
0:53 right? So when you go to get the titles,
0:55 we're gonna start all the tasks, so we're gonna go to this thing called an
0:58 "asyncio event loop", you don't need to worry about this in FastAPI
1:01 but when you work with it directly,
1:03 you do. We're gonna create a task from some asynchronous function,
1:06 right? This is stuff that FastAPI does for us.
1:09 We're going to start them off and then we're gonna start them all going,
1:12 and then we're gonna wait for them to finish and just process them in the order we
1:16 started them. But, you know,
1:17 as they all basically are working in parallel.
1:20 So let's go see this "get_html" function.
1:23 Instead of being a regular function, like this,
1:25 it's an async function with the word async out front,
1:28 which means we have the ability to use async and await within it.
1:31 So down here we have this, not a regular with block,
1:35 but a async with block from working with this httpx client.
1:40 And then anytime we call something asynchronous, like if we do this
1:43 get here, what we're gonna get back is not the answer,
1:47 not the response, but a task or a co-routine that,
1:51 when finished, would give us the response. And the way to say
1:54 make the program give up its execution to anything else that needs to run while we're
1:59 waiting and then give me the answer
2:00 when it's done, is that right there.
2:03 We just say "await this asynchronous call" and then it's up to the program to
2:08 figure out, well, while we're waiting on this,
2:10 maybe we could start a bunch of other calls.
2:12 Maybe we could be processing the responses from other clients as those API calls get
2:16 done or those http calls get done.
2:19 So this basically tells the runtime, tells Python,
2:22 look, "I'm waiting on this stuff.
2:24 Pick me up when it's done",
2:26 but in the meantime, you can go and run other code without even using threads.
2:29 So this is really the magic line right there.
2:32 And then we check to make sure it's ok,
2:34 and we return the text. Remember,
2:35 it was going "click, click, click,
2:37 request, request, request, response, response" when we ran it before about every five seconds.
2:41 How does it look now? They all start, they're all done. Just like that.
2:45 It was five seconds before. Now it's less than one second.
2:48 Let's run it a few times.
2:49 They all start, They're all done. Isn't that awesome?
2:52 Yeah, it puts a little bit more load on the server,
2:54 but mostly we're just doing all the work while we're waiting,
2:58 you know, waiting on this stuff to make it through the Internet.
3:00 Right? So this is really,
3:02 really cool. It is so much faster,
3:04 right? It's six times faster.
3:06 Well not that time. Right, there
3:08 we go. Six times faster or so this time around,
3:11 and all we gotta to do is write these async functions.
3:14 When we had a regular call that happened to be async,
3:17 we just put await in front,
3:19 and now it kind of goes back to the way it was before,
3:21 and that's it. I mean,
3:22 let's just compare real quick. We're not using a client session on requests,
3:27 so it's slightly different. But we have this, seeing
3:31 a callback URL raise for status and return text.
3:34 And now we have, oh wait.
3:36 Call URL, raise for status, return the text.
3:39 It just happens to be we're using this async client to kind of put that
3:41 all together. Okay, cool
3:43 right? So this is the async foundation,
3:45 and you can just see it's so much faster,
3:48 so much nicer, handling these requests.
3:50 And this is the same way that our Web app is gonna work instead of being
3:53 able to handle just one request at a time it could do 10, 20, 100 requests concurrently
3:58 while maybe it's working with other
4:01 API's, talking to databases, all those things. And working with async and await is
4:05 what's required to unlock that potential.