Consuming HTTP Services in Python Transcripts
Chapter: Screen scraping: Adding APIs where there are none
Lecture: What are user agents?

Login or purchase this course to watch this video and the rest of the course contents.
0:01 What is a user agent? As fun as it is to think about like a James Bond type character,
0:07 the user agent is basically just what your browser sends to the web server to say this is the type of browser I am, these are some of my capabilities,
0:16 this is the platform I am running on OS 10, or Linux or Windows or IOS or whatever, okay, so this can actually result in you getting different content
0:26 and it can also result in you getting blocked, so either of those might really matter to you,
0:32 so I suppose I want to go to my site and I am going to do a request from over here in Chrome, so I'll do request and as part of that request,
0:39 one of the headers that gets sent is the user agent, so I might say I'm Mozilla 5.0, Macintosh, Apple WebKit,
0:49 Chrome, 56. Chrome 56 is the latest version, right now. So, it's telling you what version of Chrome,
0:54 the fact that you're running on OS 10 or Mac OS, things like that, and maybe this website sees that and says okay, I love you,
1:00 you are one of my users, here, let me give you some data. The other way in which this could vary is it could say I am IOS
1:07 so you are getting the mobile friendly version of HTML versus I am at desktop browser so you are getting the desktop version of the site,
1:14 okay, so that very much might matter on what you get but either way I would say you are one of my users, here is some version of my site.
1:21 You can come over here and do a request just with requests, and what is going to send across is Python-request 2.13.0
1:27 and maybe the site is going to think, hm, this person is up to no good, I don't know if I want to give them anything, it could also be
1:33 that they give us the wrong version of the page, we want to act possibly very much like a regular browser, so you may want to control
1:39 your user agent and you'll see that it's super super easy to do but it's also very easy to overlook, so let's see how we go about that next.

Talk Python's Mastodon Michael Kennedy's Mastodon