Consuming HTTP Services in Python Transcripts
Chapter: Screen scraping: Adding APIs where there are none
Lecture: What are user agents?
Login or
purchase this course
to watch this video and the rest of the course contents.
0:01
What is a user agent? As fun as it is to think about like a James Bond type character,
0:07
the user agent is basically just what your browser sends to the web server to say this is the type of browser I am, these are some of my capabilities,
0:16
this is the platform I am running on OS 10, or Linux or Windows or IOS or whatever, okay, so this can actually result in you getting different content
0:26
and it can also result in you getting blocked, so either of those might really matter to you,
0:32
so I suppose I want to go to my site and I am going to do a request from over here in Chrome, so I'll do request and as part of that request,
0:39
one of the headers that gets sent is the user agent, so I might say I'm Mozilla 5.0, Macintosh, Apple WebKit,
0:49
Chrome, 56. Chrome 56 is the latest version, right now. So, it's telling you what version of Chrome,
0:54
the fact that you're running on OS 10 or macOS, things like that, and maybe this website sees that and says okay, I love you,
1:00
you are one of my users, here, let me give you some data. The other way in which this could vary is it could say I am IOS
1:07
so you are getting the mobile friendly version of HTML versus I am at desktop browser so you are getting the desktop version of the site,
1:14
okay, so that very much might matter on what you get but either way I would say you are one of my users, here is some version of my site.
1:21
You can come over here and do a request just with requests, and what is going to send across is Python-request 2.13.0
1:27
and maybe the site is going to think, hm, this person is up to no good, I don't know if I want to give them anything, it could also be
1:33
that they give us the wrong version of the page, we want to act possibly very much like a regular browser, so you may want to control
1:39
your user agent and you'll see that it's super super easy to do but it's also very easy to overlook, so let's see how we go about that next.