Consuming HTTP Services in Python Transcripts
Chapter: Screen scraping: Adding APIs where there are none
Lecture: Concept: Controlling your user agent in requests
Login or
purchase this course
to watch this video and the rest of the course contents.
0:01
Controlling your user agent, comes down to basically setting a simple header
0:04
when you do the requests, so here we import requests and Beautiful Soup
0:08
and we just happen to be targeting a website that we know echo back,
0:11
what it believes our user agent is, but of course, this works for all sites, right,
0:15
the server is going to see your user agent, it just might not tell what it thinks it is,
0:19
so we can come over here, and what we're going to do,
0:22
is we can actually set the header, User-Agent to a string of your choosing, right,
0:27
it could be saying I'm on IOS, it could be saying I'm on Android,
0:31
it could be saying I'm a special version of my app, whatever you put here,
0:34
that is what it's going to get sent
0:36
and the server can interpret that in a variety of ways
0:39
as we discussed, then all we got to do is pass that along with the url
0:42
as the headers when we do the request, and of course,
0:46
what does the site see, it sees that our agent is Mozilla-7 whatever,
0:50
Chrome 72 this time, that was fun.
