Async Techniques and Examples in Python Transcripts
Chapter: async and await with asyncio
Lecture: Concept: async web scraping
0:00 We've seen just how powerful
0:02 asynchronous web requests can be.
0:04 After all, what are you doing
0:05 when you're making a web request?
0:07 You're waiting on the server.
0:09 So, you can wait on a whole lot of servers
0:11 all at the same time, and get a ton done concurrently.
0:14 So, let's review some of the concepts around it.
0:17 Well, of course, we're going to have an asynchronous method
0:20 this get_html method.
0:22 If it's not async, we can't do this.
0:24 But assuming we're within an asynchronous method
0:26 what we need to do is we're going to need to find
0:28 these async with blocks for the aiohttp library.
0:32 So, we're going to create a client session
0:34 and then from that session we're going to
0:36 asynchronously do a get request
0:39 that'll give us a response.
0:41 That one's the one that probably takes the longest.
0:43 Want to do this raise_for_status
0:45 just to make sure that we got to 200
0:47 or some form of success code.
0:50 And, by the way, that does not come up in PyCharm
0:53 when you say resp.raise_for_status() doesn't come up
0:57 but it is there and it works.
0:58 And then finally, we want to read
1:00 the content from the network stream.
1:02 So not just start the response
1:04 but actually get all the content back.
1:06 So, we'll say resp.text and that's also a coroutine
1:10 so we're going to await it as well.
1:11 Convert it from a coroutine into a string
1:14 and send that back.
1:16 So, here's how we do asynchronous web processing
1:19 web request, with aiohttp and asyncio.