Consuming HTTP Services in Python Transcripts
Chapter: Screen scraping: Adding APIs where there are none
Lecture: Concept: Controlling your user agent in requests

Login or purchase this course to watch this video and the rest of the course contents.
0:01 Controlling your user agent, comes down to basically setting a simple header
0:04 when you do the requests, so here we import requests and Beautiful Soup
0:08 and we just happen to be targeting a website that we know echo back,
0:11 what it believes our user agent is, but of course, this works for all sites, right,
0:15 the server is going to see your user agent, it just might not tell what it thinks it is,
0:19 so we can come over here, and what we're going to do,
0:22 is we can actually set the header, User-Agent to a string of your choosing, right,
0:27 it could be saying I'm on IOS, it could be saying I'm on Android,
0:31 it could be saying I'm a special version of my app, whatever you put here,
0:34 that is what it's going to get sent
0:36 and the server can interpret that in a variety of ways
0:39 as we discussed, then all we got to do is pass that along with the url
0:42 as the headers when we do the request, and of course,
0:46 what does the site see, it sees that our agent is Mozilla-7 whatever,
0:50 Chrome 72 this time, that was fun.