Async Techniques and Examples in Python Transcripts
Chapter: async and await with asyncio
Lecture: Concept: async web scraping
Login or
purchase this course
to watch this video and the rest of the course contents.
0:00
We've seen just how powerful asynchronous web requests can be. After all, what are you doing when you're making a web request?
0:08
You're waiting on the server. So, you can wait on a whole lot of servers all at the same time, and get a ton done concurrently.
0:15
So, let's review some of the concepts around it. Well, of course, we're going to have an asynchronous method this get_HTML method.
0:23
If it's not async, we can't do this. But assuming we're within an asynchronous method what we need to do is we're going to need to find
0:29
these async with blocks for the aiohttp library. So, we're going to create a client session and then from that session we're going to
0:37
asynchronously do a get request that'll give us a response. That one's the one that probably takes the longest. Want to do this raise_for_status
0:46
just to make sure that we got to 200 or some form of success code. And, by the way, that does not come up in PyCharm
0:54
when you say resp.raise_for_status() doesn't come up but it is there and it works. And then finally, we want to read
1:01
the content from the network stream. So not just start the response but actually get all the content back.
1:07
So, we'll say resp.text and that's also a coroutine so we're going to await it as well. Convert it from a coroutine into a string and send that back.
1:17
So, here's how we do asynchronous web processing web request, with aiohttp and asyncio.