Consuming HTTP Services in Python Transcripts
Chapter: Screen scraping: Adding APIs where there are none
Lecture: Concept: Controlling your user agent in requests
Login or
purchase this course
to watch this video and the rest of the course contents.
0:01
Controlling your user agent, comes down to basically setting a simple header when you do the requests, so here we import requests and Beautiful Soup
0:09
and we just happen to be targeting a website that we know echo back,
0:12
what it believes our user agent is, but of course, this works for all sites, right,
0:16
the server is going to see your user agent, it just might not tell what it thinks it is, so we can come over here, and what we're going to do,
0:23
is we can actually set the header, User-Agent to a string of your choosing, right, it could be saying I'm on IOS, it could be saying I'm on Android,
0:32
it could be saying I'm a special version of my app, whatever you put here, that is what it's going to get sent
0:37
and the server can interpret that in a variety of ways as we discussed, then all we got to do is pass that along with the url
0:43
as the headers when we do the request, and of course, what does the site see, it sees that our agent is Mozilla-7 whatever,
0:51
Chrome 72 this time, that was fun.