Consuming HTTP Services in Python Transcripts
Chapter: Screen scraping: Adding APIs where there are none
Lecture: What are user agents?
0:01 What is a user agent? As fun as it is to think about like a James Bond type character,
0:06 the user agent is basically just what your browser sends to the web server
0:11 to say this is the type of browser I am, these are some of my capabilities,
0:15 this is the platform I am running on OS 10, or Linux or Windows or IOS or whatever,
0:20 okay, so this can actually result in you getting different content
0:25 and it can also result in you getting blocked,
0:28 so either of those might really matter to you,
0:31 so I suppose I want to go to my site and I am going to do a request
0:35 from over here in Chrome, so I'll do request and as part of that request,
0:38 one of the headers that gets sent is the user agent, so I might say
0:44 I'm Mozilla 5.0, Macintosh, Apple WebKit,
0:48 Chrome, 56. Chrome 56 is the latest version, right now.
0:51 So, it's telling you what version of Chrome,
0:53 the fact that you're running on OS 10 or Mac OS, things like that,
0:56 and maybe this website sees that and says okay, I love you,
0:59 you are one of my users, here, let me give you some data.
1:03 The other way in which this could vary is it could say I am IOS
1:06 so you are getting the mobile friendly version of html versus
1:09 I am at desktop browser so you are getting the desktop version of the site,
1:13 okay, so that very much might matter on what you get
1:16 but either way I would say you are one of my users,
1:18 here is some version of my site.
1:20 You can come over here and do a request just with requests,
1:23 and what is going to send across is python-request 2.13.0
1:26 and maybe the site is going to think, hm, this person is up to no good,
1:29 I don't know if I want to give them anything, it could also be
1:32 that they give us the wrong version of the page, we want to act
1:35 possibly very much like a regular browser, so you may want to control
1:38 your user agent and you'll see that it's super super easy to do
1:43 but it's also very easy to overlook, so let's see how we go about that next.