#100DaysOfCode: Learn Python with 100 Days of Exercises Course

Hello, and welcome to 100 Days of Code in Python, written by Bob Belderbos, Julian Sequeira, and myself, Michael Kennedy.
Maybe you're wondering what is this #100DaysOfCode.
You've probably seen it all over social media.
In fact, it's really, really taken off and people are finding this concept of 100 days of code really powerful for getting them to focus over a long time to actually get over the hump, and become capable developers, or maybe learn a new language, like, say, Python.
Here's an example of what you might see on Twitter.
Rene Sanchez says, "Day 11.
#100DaysOfCode progress.
Today I worked some more on bite 18.
Find the most common word from codechalleng.es/bites." This code challenge platform they're referring to is actually from your co-authors, Bob and Julian.
We'll talk more about that later.
Here's another Tweet: "Day Five of 100.
Did some short exercises about modules.
Imported modules, did a couple of my own.
Tomorrow IO.
#100DaysOfCode #Python." Way to go, Bricks.
Here we have "Round one, day 101.
Had to do an extra day due to one day off sick earlier in #100DaysOfCode.
Today more Python debugging, tomorrow starts round two." And Jeff says, "Round one, day 19.
#100DaysOfCode.
Did three exercises in the book.
Basically my average, been taking it slow these last few days.
#CodeNewbiePythonIndieDevIndieGameDev" And finally, let's look at one more.
Amit Kumar says, "#Day32.
Another autowebcompat PR pull request, just got merged.
Way to go, Python Tkinter, #100DaysOfCode." So he added some new feature or bug fix to auto Web Compat, very, very cool.
So you've seen this stuff probably all over social media, Facebook, Twitter, and so on.
What's it about?
Well, this is actually a very structured project put together by this guy, Alexander Calloway.
So Alexander, he was studying in business school but also wanted to learn programming.
And he was having a hard time making progress.
So he came up with this idea of #100DaysOfCode.
Here's a quote from him: "The idea of #100DaysOfCode originally came from my personal frustration with my inability to consistently learn to code after work.
I'd find other less involved activities to spend my time on, like binge watching a TV series.
One of those days, I was sitting in a restaurant with my wife, and sharing my frustrations with her.
I suggested maybe I should make the public commitment to learning for at least an hour every day.
I thought it would go for three months, but it turned out 100 days was the right one." How about that?
Well, thank you for creating this project, Alexander.
This is really a great thing for many people getting started, and this is what this course is all about.
We're going to give you lessons and exercises for every one of these 100 days.

There are a lot of things that people do to support each other and encourage themselves to stay focused and keep going on #100DaysOfCode.
But there's really just two main rules and they're really really simple.
The first rule is to code a minimum of an hour every day for the next 100 days.
And I would say taking a coding class like learning the lessons in this course and then coding a little bit counts, right?
You're immersing yourself in code for at least an hour a day every single day for the 100 days.
And if you got sick, like you saw the person before had gotten sick, had to take a day off, that's okay you just add some days on the end.
Have sick days and just move it to the end.
The second rule is a public commitment to making progress and keeping with it.
And they way that works is to tweet or to put onto somewhere like Facebook #100DaysOfCode with an update every day.
The PyBytes platform actually is going to help you a lot with this, but however you want to do it it's code an hour a day and share your progress every day.
Super simple rules, and we hope this course really makes this work for you.

We are going to cover so much content in this course, it's going to be amazing.
You'll learn many, many different things over these #100DaysOfCode.
In fact, there's so many I can't really enumerate all of them, it'll just take too long, but I do want to give you a quick sample into what we're going to cover.
We're going to talk about collections, lists, dictionaries, working with them.
We're going to test our code with pytest to make sure we build reliable apps.
We're going to create games, Dungeons and Dragons style with classes and inheritance and object-oriented programming.
We're going to deal with errors and proper error handling in Python.
We'll do logging to keep a history of what our application has done.
We're going to work with the popular exchange format called JSON, and it's a really great way to exchange data between Python applications and any web service.
Speaking of services, we're going to learn how to call JSON based web services from Python, and if there's no service, can still go to the website and do web scraping.
You can turn any HTML page, anything on the internet, into a data source using web scraping.
Another source that we might go and consume, RSS feeds, really popular among blogs and podcasts, but also other types of subscriptions.
We're going to use the Twitter and the GitHub API to interact with those services automatically from Python.
Want to send an email?
Maybe a new user registered for your site, well we'll see how to do that as well in this course.
Excel has got to be the most popular database in the world.
It's not really a database, but people use it like one, and you may need to program against it.
Turns out, we have the trick for you right here.
Want to automate something on the web?
Go login here, navigate over there, click this button, make that thing happen.
We'll see how to do that with something called Selenium.
You want to write a web application, well we'll do that with something called Flask, it's probably the easiest way to write a web app in Python.
SQLite is a database built into Python, it's what's called an embedded database, and you'll see how to program it, either directly, or from what's called an ORM from SQLAlchemy where you create these classes and you map them to objects in your database, so we'll have a couple of places where we talk about SQLite and relational data.
Graphs are wonderful, they explain so many things, and so we're going to use something called Plotly and draw graphs for you, based on a set of data that you have, and typically when you're doing science like stuff like this, that's done in something called Jupyter Scientific Notebooks, and a good portion of this class will be presented in these notebooks.
Not all of it, maybe about a quarter.
GUIs and Python, they typically don't go together, but in this course, they do.
You'll see in just a few lines of code that we can create a really powerful and cool GUI or desktop application, and this will run on all the platforms, Windows, Linux, and macOS.
And finally, it's fun to consume APIs, but sometimes you want to build them, so we're going to actually take Flask and extend it to create our very own API and put that out on the internet.
This is a ton of stuff right, isn't this exciting?
Well, it's only a small part of what we're going to cover in this course, so I hope you're really excited, Bob, Julian and I definitely are excited to teach it to you, so let's get to it.

You've chosen Python for your #100DaysOfCode.
Maybe you're a Python developer who has lots of experience.
You just want to go through this whole challenge, and that's great.
You probably already know the power and popularity of Python.
But if you're just getting into programming, and you're coming here and saying, "Well, let's try Python for this 100 days.
That seems like a great way." I want to tell you, you have chosen wisely.
So check out this graphic.
This comes from one of the best sources on the internet for popularity in adoption of technology, Stack Overflow.
And the data scientists at Stack Overflow did some predictions and said, "Well, how are the various languages doing over time?
Are they becoming more popular, less popular?" Based on their view into the industry.
And they did this up to mid-2017, and then you can see the gray part where they're projecting out.
One of these languages is unlike the others.
It is just going up, and up, and increasingly up.
Your other best bet is JavaScript, which is barely logarithmically going up.
Java looks like it's topping off.
The rest of them are going down.
So if you're going to focus on something, pick one particular language.
Pick the one that's got all the momentum and the popularity behind it, and that's Python.
Now, you might say, "Okay, Michael, this actually is against all these older languages, C#, Java, and so on.
What about the new languages like Go and Rust?
They're probably even more amazing and more powerful, and growing quicker." Well, let's see.
Yeah, they're growing up, they're going upward, not downward, that's really great.
Swift is going up, TypeScript's going up, Go is going up, but they are nowhere near Python in this graph.
I just want to leave you with these two pictures in your mind that Python is really a great place to put your energy and be learning.
My rough rule of thumb here is I would like to bet my career on things that are going up, not down.
So which one of these do you want to pick?
Well, you're in a good place.

Let's take just a moment and talk about how you're going to experience this course.
Yes, it is #100DaysOfCode, but it's not 100 different topics.
What we've done is we've broken the course into three day segments.
So day one, two, and three, days four, five, and six, and so on.
So, for example, on day 10, we're going to introduce a new topic, teach you how to work with it, the ideas behind it, do some live code demos, and then after that, you'll get a chance to write just a little bit of code, 'cause you're going to spend a long time actually watching the videos.
These vary, they would be between 15 minutes, maybe to 2025 minutes on average.
There's a few that are really involved that go out longer than that.
But basically, the first day of any three day segment on a single topic is going to be mostly learning about that subject, writing a little code.
The next day is almost entirely writing code.
We'll have a video for you, it may be just two minutes long, and we're also going to have in our GitHub repository instructions with hints and tips, and things for you to do.
So you'll be able to follow along there.
Then the last day is really about finalizing your code, maybe putting the polishing touches on the code that you wrote on day one and day two.
Again, we'll probably have some instructions to guide you, but if you want to deviate, that's fine.
This is your #100DaysOfCode.
We're just here to support you along the way.
So you can think of this course as 33 three day journeys through it.
And each one of those journeys will go through a particular topic: SQLAlchemy, Flask, Collections, whatever.

Hi, I'm Bob Belderbos, and I've been programming since ten years.
And for the last six years, I've been using Python, which is now the language of my choice.
I'm very excited to teach you Python in this course.
We're going to cover a lot of angles, so be prepared to learn a lot of Python.
These days I'm a web developer, software engineer at Oracle, and since end of 2016 co-founder of PyBites, together with Julian, where we blog about Python and do code challenges which we lately have transformed in a code challenge platform.
We are super passionate about getting people to code, have a very hands-on approach and really want people to get to the next level of Python.
And that's why we're so passionate about the #100DaysOfCode and this course because we get you to write a lot of Python.
I'm honored to teach you Python and look forward to our journey.
Hi everyone, I'm Julian Sequeira, and welcome to the course.
I've been coding with Python for roughly two years now.
I currently work at AWS, but surprisingly not with Python.
Python is something I've taken up on the side, and something I'm super passionate about and absolutely love doing in my spare time.
I'm co-founder of PyBites, along with Bob.
And everything I do with Python is purely out of love for the programming language.
And I'm totally looking forward to teaching you everything I've learnt over the past couple of years throughout the course.
Hey, welcome to our course.
I'm Michael Kennedy, nice to meet you.
A little background on me, I'm primarily a web developer as well as the founder of the Talk Python To Me podcast and Talk Python Training.
I've been a professional developer for 20 years, and I'm really excited to share that experience with you throughout this course.
Welcome to the #100DaysOfCode and Python.

Do you need a little help with the Python language?
Of course, we'll cover much of the little details and definitely the advanced features as we go through this course, but we don't start from the absolute beginning.
What is a variable?
What is a loop?
Instead, we've included a Python language primer.
So when you look at Python, if it's, you know, a little fuzzy, maybe you've done it a long time ago, but you've forgotten.
You haven't done very much of it.
Well, we put something in here to help clear it up.
So anytime you're confused about something in the language, just jump down to the appendix, Python Language Concepts.
Find that thing, watch the one to two-minute video.
Hopefully, that'll clear things right up.
You'll find this at the end of the course.

We've made all the code we've written over our 100 days worth of topics available to you in GitHub right here.
gitHub.com/talkPython/100daysofcode-with-Python-course.
You can go over here and check it out.
This is also super important because some of the projects that you work on have either data that you've got to start with, or they've got some instructions and all of that you'll find here.
Let's jump over to GitHub and have a look.
Here we are at the GitHub repository.
You have a little bit of a description here at the beginning.
But the main thing you care about is the days.
You can come in here and see the day one to three.
This is the stuff with JavaScript.
Here's day 13 to 15.
This is our text-based games.
Down over here is a Search API.
Let's just check this one out and see what's in here.
Here's demo that we built, and if you need the code for it you can see actually here's what we've written and so on.
These three pieces work together.
But what's most important is this ReadMe.
It's automatically displayed by GitHub when you come here.
You can see it talks about now that you've seen the videos what do you do?
Here it talks about, okay, so you watch this.
And the first thing on the day 43, that's the first day of these three, this is what we're going to do.
On day two, it talks about working with API.
It shows you how to use it with this thing called Postman.
And then finally, it also shows you how to build your app, and then finally it says, okay, on the last day, we're going to make it even better.
For the very grand finale, we're going to open it in a web browser if somebody picks something inside your application.
You can see that as you go through you want to make sure that you go to each day, look at the instructions.
They're going to be there to help you follow along, both the code demo we wrote as well as the steps and data that we put together for you.
Of course, to download this, you'll want to download it.
I would also say star and fork it if you have a GitHub account.
You could either come here and copy this and Git clone it if you're familiar, or if you don't want to mess with Git you can just download the Zip file and it will be a folder you can unzip somewhere to work with.
But definitely download this.
You're going to want it locally.
You're going to want to save it.
You're going to use it throughout this course.

When you dedicate yourself to taking a course, and carefully working the way that the instructor, the author, is working, you are effectively gaining much of the experience that that particular author or developer has gained through their career, through their jobs.
This course is special because it's taught by three people.
That means you get three experiences in one.
And this is super valuable.
Imagine that you have a job at this place.
You get to work with cool VR gear and on hardware and IoT things.
You'll gain one set of experiences.
But if you took a different job, say, you're starting a fashion start-up with your friend, this woman, from college, and you're just working on this coffee shop.
Being scrappy, working, trying to get venture capital and launch your application, you have a totally different experience than this dude in a VR headset.
Or maybe you go the corporate route, work at Microsoft like this guy here.
He's, you know, working on some new programming, language tooling around Python.
These are all super different experiences.
And these experiences are very positive.
They give you a different perspective and more perspectives on programming.
That's awesome.
How's that relevant to this course?
Well, with the three instructors, we each have a slightly different set of tools.
And slightly different way of working.
We're going to show you next, how each of us gets set up.
What you need to follow along with each of us: Julian, Bob, and myself during our particular segments.
You don't have to work like us, but if you want to do exactly what we're doing, we'll show you how we got started.
And we feel this is super valuable for you.
You'll have not just one experience, but three experiences kind of bundled up into one.
And so on the other side of this course, you're going to have a broader perspective.
And that's pretty awesome.

'Kay, guys.
This is Julian here, and I just wanted to quickly walk you through my environment for the duration of the course.
There's actually not much to it, and that's because I'm using Windows, and I really wanted to keep it simple.
Okay, I wanted to show that you don't need to use anything crazy.
Not that there's anything wrong with that, but pretty much bare-bones on Windows, you're able to code anything you want and just get into it, okay.
So the first thing you're going to need to do is go and download Python.
And you can do that at Python.org/downloads.
Okay.
And it will detect you're on Windows if you using Windows, and you can download Python 3.6.4, the current latest version, and install it.
It's just an Microsoft installer file.
It will install to a default path.
You don't have to change anything.
Don't worry about it.
Nice and simple.
And once you're done, you can actually just launch IDLE.
Okay, that's I-D-L-E.
Right?
You type that into your Windows search, and that will actually bring it up into the Start menu.
Right?
Once that's up, you'll see something that looks similar to this, the Python shell, which you can just type in Python code.
Okay?
And you can live code just like that.
This is the Python shell.
Now with IDLE, this Integrated Development Environment, okay, that's what IDLE stands for: Integrated Development and Learning Environment.
With that, you can actually create a Python file.
Okay.
This is actually going to be pretty much a plain text file, but when you save it, it will automatically save it as a .py file.
So your Python file.
That's what I like about using IDLE.
I never have to worry about accidentally using the wrong format.
Okay?
And other than that, you've seen some of my videos that I run my scripts through the command prompt.
Okay?
There's my path to my environment, to where I'm storing everything for this course, and I will just run the scripts straight from there using the Python command.
And that is pretty much the bulk of my setup.
You won't see anything else.
You won't see anything different.
That's it, okay?
So standard command prompt on Windows.
I don't even use PowerShell.
I've go the Python shell here, and I just save the files using the new document section of the Python IDLE.
And that's pretty much it.
So, I say keep it simple.
One day I probably will make the move to PyCharm or something, but for now, I'm happy with this.

I'm using Anaconda for this course, a pre-bundled, Python distribution widely used in the Data Science community.
And it comes with a lot of packages already.
You're not required to use this distribution.
You can also pip install my requirements, which we will see in a bit.
You can download the full version here.
We recommend that you use 3.6.
Really no reason to start a new project in Python 2 anymore.
You can also install Miniconda, which is a smaller version, which only includes the base packages.
And, mainly, what you need to know is for almost all the lessons, I will be using Jupyter Notebooks, which is a great way to experiment with Python code and more in the browser.
It's a great tool to both teach and learn Python.
You can try it out if you want to play a little bit at this point by going to try.jupyter.org, but I encourage you to install it to follow along with my lessons.
To install it, again, the recommended way is to use Anaconda, but you can also use pip install jupyter, and that should get it as well.
And let me show you that quickly.
So, first I need to clone the 100 Days of Code repo.
You cd into that.
At this point you really want to make your virtual environment to work on the project's requirement in isolation to not mess up your global space.
And in every lesson, I have a video how to pip install the requirements for that lesson, but I also have 'em all wrapped together in a requirements file.
So, for all the notebooks, you need Jupyter; and ipykernel, which I will explain why in a bit; and then I listed out the requirements for each lesson.
There are various ways to make a virtual environment.
The classic way is to use pyvenv built in module.
You can also use pipenv, the new way, which should be perfectly fine.
And Anaconda comes with Conda, a utility to manage environments as well.
However, I am used to virtualenv, just the classic one.
So, in this course I am making a virtual environment with this alias: virtualenv -p, pointing to the Python binary that comes with my Anaconda installation, and the name of my virtual environment.
So, let's run that now.
And then you have to activate it.
And that's what I'm doing, that a lot I have another alias, ae, and now I'm in my virtual environment, where I don't have anything installed.
At this point, you can just do it video by video, but if you want to have all the packages up-front, you can do pip install -r, requirements/requirements, and that might take a bit because it's not only pulling the dependencies, but some of the dependencies have other dependencies.
With that done, you can launch a Jupyter notebook like this.
And you can go in today's and do any lesson.
For example, Selenium.
And you can open the notebook like this, and you can follow the lesson.
And you see that the notebook discovered my virtual environment.
If that is not the case, you might have to tweak it a little bit, and that's why I pip install ipykernel and Tornado was pulled in as well.
That should have been enough.
If that's not the case, you might have to run iPythonkernel install <username> <projectname> And this will be venv, or the name of the virtual environment.
And then back to the notebook.
You should have the kernel, venv, or whatever you named the virtual environment, here, and you can switch to that, but I already have it here.
So, then your dependency should work.
It's in your virtual environment and you can follow along with the lesson and make any modifications in the code and experiment and that's how you learn.

Michael here.
It's time to show you how I set up the tools when I'm writing code.
Now, I'm going to be using a Mac, macOS High Sierra for this course.
That's the latest at the time of the recording.
However, you can use Windows, you can use Linux.
They're all basically the same as far as that goes.
I'm going to be using the editor PyCharm and I'm going to be using Python, the one that I got from Python.org not installed any other way.
So let's see how that goes.
First off, when you're on a Mac if you've taken no actions you don't have Python 3.
You'll know if you open up your terminal.
Come over here and type Python3 -V and you would get the same error something like Python3 not found as if there's a Python4.
Someday, maybe, not right now.
So if you type Python3 -V not lowercase v and you get not found you need to install it.
If you get something like, 3.5 or above, you're fine, you're done with Python.
All you got to do is come over here, go to downloads, download right now the latest is 3.6.4.
So download that, it gives you an installer.
Run the installer; you're good to go.
The other tool that I use a lot is something called PyCharm.
It's an editor for Python.
One of the richest and most powerful editors.
And I really think it's great both for beginners and for professional developers.
And it comes in two versions.
You can see the full fledge Professional or the Community Edition.
So you can download the Professional or the Community Edition.
The Professional one costs money.
You can pay for it monthly or you can buy a license.
Whatever, it's like about eight or nine dollars a month.
You can get the Community Edition.
It's free and open source, okay.
This also comes with a month long free trial so you can try it.
If you care about the features, say which one comes in which, you can compare them down here at the bottom under choose your edition.
So it's up to you which one you get.
So we can come down here and download this and install it.
One thing that's cool that you might consider getting is this Toolbox App.
This one will sort of keep track of updates for you.
It looks like this and it gives you access to all of the tools that JetBrains has.
So if you're going to install more than one, this might be handy but you don't have to get it.
Either way, get either PyCharm Pro or Community Edition or get this JetBrains Toolbox which I have up here.
You can see apparently there's an update for my PyCharm Professional.
If I want a data grip, I can just click that and install it.
Once you have it you can run PyCharm and you'll be able to start creating and editing Python projects.
That's it, I don't really have anything else installed for working with the code.
It's just Python, my OS, and PyCharm and we're good to go.

As Michael stated, there are 2 important rules for the 100 Days.
And it is code an hour a day, and track your progress.
Do a daily tweet of what you have accomplished that day.
In order to help you guys do that for this course, we made a feature on our code challenges platform.
So head over to codechalleng.es and sign in with GitHub and go straight to 100DaysOfCode.
So here you see a template where you can start the #100DaysOfCode, but we made it easier and made a direct link to populate the 100 Days for you.
Head over to start talkPython and here you get a grid of 100 Days with exactly the materials of each day.
So for example, Day 1 you start with day times.
Day 1 is a lectures.
Day 2 is the practice, and Day 3 is even more practice.
And you can just click on the Tweet icon and that prepares a tweet for you, so you can just tweet this out or you can modify it as you want.
And then you can start to put things into complete.
So here for example I hit Day 1, done.
And you see that this percent counter then goes up and the items that you've done are under done.
And so, 2.
And here you can document additional learning, so here I learned that time Delta, for example, and here you are at the second lesson.
So this is Day 4 and here you're going to look in the collections module, here.
Watch the videos.
And on 4% and here I did an exercise, and marked complete.
So this is the grid of the whole 100 Days.
All the materials we have prepared for this course, and once you're done, you have a nice love of all the stuff you have coded and accomplished.
For example this is our #100DaysOfCode which I imported from Twitter.
As you see we did a ton of coding and you have it all nicely in one place, and you can refer back to it which we do often.
We go back to our script a lot, pull out stuff we learned, reuse it, and that's why it's nice to have a #100DaysOfCode log.
Apart from that the tweeting is a very easy and we encourage you to do that.
It's also a way to notify us what cool stuff you are building for this course.
Completing the 100 Days is not easy.
It's a lot of work and dedication, but at the end, it's a great accomplishment.
And you will have grown as a developer.
So you should be proud when you see this honor of accomplishment when you hit the hundred percent done.
So we hope this will help you make it easy to keep track of your progress during this course, and if you want to tweet along is stated in one of the rules.
This should make it very easy.
Alright and with that good luck and prepare to learn a lot of of Python.

Good day, this is Julian Sequeira and welcome to the course.
We're going to open things up with playing with datetimes.
Probably not the most interesting thing for most of us and if you're like me, you probably hate it because they can be very finicky.
So, with datetimes I wanted to run us through some of the more basic concepts of it.
Just go with it, there will be more advanced stuff coming up but for now we're going to stick with just the basics to get you through with datetimes specifically around datetimes.date and then datetimes.timedelta.
So, we'll flick through into that, carry on, and let's get started.

Right a quick overview of what we're doing for the next couple of days.
For the first day of your datetimes lessons you're going to watch the videos, okay?
A couple of videos for you to watch to do with datetime, date, and timedelta.
Alright after you've done, after you've completed watching the videos go ahead and just play around in the shell.
So do some timestamp calculations as per the content in the videos.
So we won't dwell on that too much.
The second day I want you to head to our challenges, our challenges platform I should say and sign up with your GitHub account.
It's free and then follow this link here and this will unlock this datetimes challenge, okay?
This bite here is going to be based around parsing dates from logs.
Okay so have a play with it, code in the browser and have fun.
That's your day two.
Then day three.
That is all going to be up to you.
Create something for yourself.
I reckon you should give a Pomodoro timer a chance.
Use datetime for it.
I know you can just use a time module for these examples here but the idea is to include some timestamps.
Do some calculations and see what you can wrap around date time okay so the Pomodoro timer is quite simple.
You can do that or you can do a stop watch.
Anything like that.
So if you have any ideas yourself now is your time to test it out on day three.

Given datetime is part of the Python standard lib, we don't actually have to do any setup here.
You'll see in the coming videos that you will have to do setup steps, create virtual environments and whatnot, but given this is datetime, we don't really have to.
And, I think it'd be best for us to just work in the Python shell here.
This is IDLE, the default Python IDE that it ships with.
So, let's have a play with that.
Now, the first thing we're going to do is we're going to import datetime.
But, we're actually going to do from datetime import datetime, okay?
And, this is just going to make it a bit easier for us when we're typing in the rest of our code.
And, just to get yourself prepared, let's just from datetime import date that's for a bit later in this video.
Alright so what is datetime, alright.
For those who are unaccustomed and unaware datetime is just the Python library module that allows you to deal with dates and times.
Pretty self-explanatory, right?
So, if you want to deal with just the dates so, you know, today's date, let's call it the 23rd of February 2018, not very specific.
Or if you want to deal with the time that you've got to think about that from a programming perspective, there is a difference, okay.
So, datetime allows us to deal with the entire time set, the entire timeframe.
You're talking seconds, minutes, hours, days all the way through to years, okay?
We can visualize that with datetime.today().
If we hit enter, there we go, we can see today's date.
The 24th of February 2018 but we also get this timestamp.
It's 10:17pm and these are the extra seconds here.
So seconds, milliseconds and whatnot, okay?
Now I'm going to show you this, what kind of an object is this?
Well let's go, well first actually we have to assign that to something that way so, we'll just go with today.
Here's datetime.today() alright and then we'll type it out, so type today.
So it's a datetime object, okay?
And that's important because you can't mix these objects.
I'll point that out in just a minute.
So with this timestamp, there is more you can do with that.
And I'll show you that in the next video with timedelta.
Alright, but for now just understand that this is what your standard datetime format will look like.
This is the sort of data you're going to get back.
And this is really useful for times when you want to deal with say, subscriptions or anything like that where it has to do with exact timestamps, or logging or anything where you need to know the time that something happened.
Going by the date of say, the 24th of February is not accurate enough, okay, there is 24 hours within that day so, a lot of things could have happened.
Alright, so we'll move on to the date part here.
So we'll just go today date, we'll create that variable.
Here's date.today(), so you can see straightaway we're not using datetime, we're using the date section okay, we're using the date option here.
So date.today() and if we type that out Today date, we can see the different type of object here.
First one was a datetime and now it's a date object, okay?
And we can see what that looks like with today date.
And we have just the date string, okay?
So we don't have the extra time on the end.
And this is, again, very useful.
So you can see the distinction between the two of them.
Alright let's get ourselves a little bit of white space.
Now one really cool thing that I love about date is that we can drill into it a little more, so we can go today.month is 2.
So you can see we can actually tear it apart a bit.
So today.day is 24 and then today.year, and we get 2018.
So now you can sort of visualize how date can help you in your projects, right, if you're not already using it.
It's actually really cool.
So one really, really cool thing that has come in handy over time, is the fact that you can do a bit of math with your dates, alright.
So we'll go, let's just go something easy.
So Christmas, what's the date for Christmas?
It's the, we'll go year first, so 2018.
It's the month next, so 12.
And then it's the day, so 25th, alright.
Now one thing, if you had a look, this is ...
us specifying a date, this is us assigning a date to a variable.
So now the Christmas variable is always going to have this date assigned to it.
You can see that there, okay.
Now, this is really cool, so We can actually go Christmas, cause we know that's the end of this year, minus, today date.
Kay, and that's 304 days, it automatically called on timedelta, so that's giving away something for the next video but, carry on, 304 days.
Alright, and we can see that visualized a different way.
We can, and this is again giving more away we can go Christmas minus today in days, so .days.
304 days, alright and this is really cool for something such as this, I'm just going to copy and paste here rather than type it all out for you, alright.
So if Christmas is not today date well what can we do?
We can print a certain message.
Again, you can see this is useful for certain other projects so print, sorry there are still this many days (christmas minus today).days, until Christmas.
Okay, and then else ...
We'll copy and paste this as well.
We're going to print some sort of message, alright.
"Yay, it's Christmas." So, by hitting enter, sorry there are still 304 the same value here, until Christmas.
I've obviously left out the word 'days' so that's my mistake, but sorry there are still 304 days until Christmas.
If I happen to wait another, you know, ha ha ha ha that many days, 304 days we would then get this message here.
So this is date and this is datetime.
Very, very tedious at times, I want to say but so useful, so this is a great place to start manipulating your code, manipulate your dates and have some fun with it.
And in the next video we're going to look at datetime.

Okay, just like the previous day, we're going to look at something but we're going to use the Python shell for this one.
And specifically today we're looking at timedelta.
So what is timedelta?
Well, timedelta is pretty much a gap in time measured out.
So, for example, if you want to calculate something such as how many hours from now until a certain point in time, you can use timedelta for that to specify what it's going to be.
A real world example.
How many hours until I go to bed?
Well, let's say it's going to be four hours.
So my timedelta, you can specify for this calculation, is four hours.
And four hours from now could be two in the morning, okay?
So it's different...
That's how you calculate things like that, you use timedelta.
All right, so how do we do that?
Well, we go from datetime import datetime just like usual, from datetime import timedelta.
All right, so let's represent our timedelta as a variable t timedelta and let's work on days and hours.
So let's say we have four days and 10 hours until my next day off work, it's pretty depressing.
And how do we deal with this?
How do we work with this?
Well, let's first of all confirm we have a timedelta object there, excellent.
And what next?
What can we do with this?
Well, we can go how many days in there.
So t.days.
That gives us four days, okay?
One important thing to note here, watch this next one.
T.seconds.
36,000.
So 36,000 seconds is not four days 10 hours.
36,000 seconds is just the 10 hours.
And why is that?
Well, this timedelta is just like...
Imagine the stopwatch on your watch, it's only able to go up to a certain amount of time, right?
Maybe 23 hours and 59 minutes.
So with timedelta, the seconds, it's only able to go up to a maximum of one day, okay?
So we have four full days here, so it's not going to show us the seconds in four full days.
It's only going to show us the seconds in the hours.
So you have to take that into account and your calculation.
Okay?
We could calculate the hours but not like this.
Okay?
It doesn't allow us to do this because it has seconds, it's not going to bother with hours, all right?
So in order to get around this, well, you have to do a bit of maths, unfortunately for people like me.
So t.seconds divided by 60 and divided by 60 again.
Well, because we have 60 seconds in a minute and then 60 minutes in an hour.
And that gives us that 10 hours.
Alternatively, you could write that as t.seconds / 3,600.
Same thing, okay?
That's a really important gotcha because it definitely got me.
back at the start.
So here is an example of a sort of scenario you could use it in, but just keep in mind, timedelta is that gap, it's that sort of way of representing the time between two points in time, okay?
All right, so we have an ETA.
Well, let's just say it's the ETA until I wake up.
So hours equals six.
We're not even going to talk days here, okay?
We can go today.
We'll give ourselves a datetime today, variable, okay?
We're not dealing with just date, we're dealing with day time because we want the time, we want the minutes, the seconds, the hours, right?
So there we go, we've got two variables, ETA and today.
All right?
So today, let's just show you what that is.
It's currently 10:39 p.m., okay?
Let's get rid of that.
All right.
We can go what is ETA?
Is our timedelta, all right?
Now, what next?
We want to add these two together, okay?
So we can go today + ETA, this is the beauty, the absolute beauty of timedelta, we can just add it straight to a datetime object which is so cool and so handy and it makes it so easy.
So today plus ETA.
And look at that time.
It actually changed the date to the 25th because we'd cross over midnight and it says six hours from now is 4:39 a.m., okay?
And this is really, really cool because you don't have to worry about any conversions, you don't have to change anything.
It's so easy.
And even better than that, we can format it, so today + ETA as a string.
Look at that, it's glorious.
We have an actual nicely formatted date string and time stamp.
How awesome is that?
And that's timedelta, that's really the bread and butter of timedelta.
You're dealing with just setting yourself a static time, a static amount of time and then you can add it, subtract it, do whatever you want with it.
And this is really useful in a lot of programs, so keep this one in your belt

Okay, and that was the basic overview of datetimes.
How cool was that?
Not too bad, not too hard.
Nice way to start your #100DaysOfCode on Python, right?
Alright, so let's do a quick recap of what we covered.
There wasn't a lot so this will be pretty quick.
So we began by importing datetime and date.
And we then started to look at the differences between datetime and date.
So a datetime object, well when we ran datetime.today(), it included the date and the time, so we had a timestamp in that object.
Whereas when we ran that with just date, we only get the actual date, the calendar date.
So we the 19th of February 2018, alright.
And we found that you can't actually easily combine the two, do maths between the two.
Okay, not without a lot of conversion.
First we gave ourselves a Christmas variable, and we gave it its' actual date, which is something you can do with date.
You can assign an actual date to an object.
Once we did that, we were then actually able to calculate the amount of days between Christmas and the current date.
So that was just a bit of a little scenario for you to use datetime and date.
Okay, next we played with timedelta.
Now we began by importing timedelta and then we gave ourselves a timedelta object.
So we set the timedelta length as 4 days and 10 hours.
Then we discussed the fact that you can view your timedelta in those days and you can view it in seconds, but you can't view it in hours, okay.
And that's because it only works in days and seconds.
And the seconds only go up to a max of the 24 hours of a day.
They expect you to do the calculations yourself.
And that's what we see here.
t.seconds / 60 / 60, and then we get our ten hours, okay, matches up there.
As a little scenario to try, we wanted to look at the ETA.
We wanted to add the estimated time of arrival onto the current time.
So the current time plus six is that there, that's the object there.
That's the response there I should say, the calculation.
And we were able to add and subtract timedelta from datetimes which is really, really cool and makes it really easy.
And using string on that, converting it to a string, we got a really nicely formatted timestamp here.
Very useful for log files right.
Alright, your turn.
This is where it gets a lot of fun.
What I'd like you to do for day three is come up with something cool for you to make with datetime or timedelta.
Think about perhaps making it a stopwatch, maybe a timer application.
I actually think a really fun one to make would be a Pomodoro timer.
So if you're not familiar with Pomodoro, just go and google it.
But that would be a really cool way of setting specific timestamps that a user could choose using datetime and what have you.
So that would be really, really fun.
Now I know what you're thinking, datetime is a really deep and in-depth topic, but unfortunately we just don't have the time to run it in this course.
So I hope you really enjoyed it.
Move onto the next video, we are keeping it nice and simple for the first day.
Expect things to take it up a notch going forward.
So enjoy, get cracking, don't waste any time.

Welcome back to the 100 days of Python.
In the coming three days I will guide you through the collections module, a very convenient module to work with more advanced data structures.
First we look at namedtuples and how they can make your code more readable and elegant.
Next we look at defaultdict, which is convenient to build up a nested data structure.
Third, counter, saves a lot of code to find the most common thing in a collection, and lastly, deque, which can tremendously improve your performance based on the operations you want to do on your sequence, and for the second and third day I got a movie dataset where you can put the collections module into practice.

Get Pythonic with a collections module.
We are all familiar with dict, list, set, and tuple.
The collections module adds a few other specialized ones that are very useful.
Let's import the modules we're going to use.
A namedtuple is a convenient way to define a class without methods.
We all familiar with a normal tuple, which you can define with parenthesss, and one or more elements.
The thing with the classic tuple, though, is that the order is not really meaningful, so if you print the user name and the user role, user index zero is a user index one, and you already notice that the indexing is not really saying that much.
Is there a more readable way to define these kinds of tuples?
And yes, you can use a namedtuple, so let's define one.
User equals namedtuple, and you give it a name, and the fields or arguments it takes, so name and role.
And let's create a user, with user and I give it a name Bob and role equals coder.
The nice thing, then, is that you can access the entries like this, instead of indexing with zero, one, etc.
So this is much more meaningful And to see that in a print statement.
So, use namedtuples.
It's very easy to set up, and it definitely makes your code more readable.

A second data type I want to show you, today about, is defaultdict.
And it's very useful when you're building up a nested data structure and you have to account for keys not being there.
Now first of all, what's the problem with keys?
Let's define a very simple dictionary, just one user and role, and let's do a lookup by key.
So...
Bob is there.
Julien is...
Oops, not there and that gives you a key error.
There's a way around it by using users get.
Oop, get Bob..
and users get...
Julien...
which returns none.
But how do you deal with that when you're building up a collection?
Now let's get some data.
I'm going to define a list of tuples.
A challenge is done.
And it has...
tuples of name, and a number of the challenge that has been completed.
Let me type that out.
So the goal is to convert this into a dictionary.
Let me show you what happens if I use a normal dictionary.
For name challenge in...
challenges done.
Challenges dictionary...
name append...
challenge.
Oops, it says Mike is not in the dictionary.
In the first round, he is indeed not in the dictionary.
So here is where is you really want to use a defaultdict.
So to define one, challenges...
Equals defaultdict, and you need to define what type the values hold.
So in this case, the key is the user and the value is a list of challenge numbers.
So I put list here and the rest is kind of the same.
For name challenge in challenges done.
Challenges...
Name...
append...
challenge.
I'm almost sure this works.
So, yes, we have a defaultdict which holds lists and here you see keys are Bob, Julien, and Mike and values are list of challenge ids.
So you see here, we work around the key error.
The defaultdict has the mechanisms to use a factory to initialize the data type that the values need to hold and yes, it's safer, it's cleaner, and for this type of task, I highly recommend that you use this data type.

Let's move on with Counter.
Let's say we have a text which is split into words, and we want to count the most common words.
Before I knew about collections, I would write something like this.
There you go.
I had to loop over words, keep a dictionary, see if the key was in the dictionary, if not, initialize to zero.
If it's in there do plus one.
Then I had to loop over the items over the key value pairs, sort them and use lambda to sort by value.
In reversed order and take a slice to get it to five.
Now compare that with using Counter, and its most common method.
It's like magic, right?
One line of code, you pass the words list into the Counter, and you call most common and you give the number of top words you want to see and compare that with all the work I had to do here and how easy it gets by using the collections, Counter.

Next up are deques.
deques are stacks and queues.
They're useful if you want to insert and append on both sides of the sequence.
In this section, I will compare them to lists.
I mean, lists are your best friends.
You will use them everywhere.
They're easy to use and for 80% of your use cases, or maybe 90%, they're just adequate.
But lists come with a downside which if you have to move items around, they get less performant.
And in this exercise, I will create a list and a deque of ten million ints and a function to do random inserts and deletes and then we're going to use timeit, to see how they perform.
So let's create the two sequences.
First I want a list.
I can just use the range.
Let me the get the zeros right.
One, two, three, one, two three.
That's ten million.
And let's make a deque.
You create that big deque.
Range, one, two, three, one, two three.
Next, we create an insert and delete function that takes a sequence...
and we do, for...
Underscore...
In braces..
Index equals random...
Choice.
And a random choice takes a sequence and just randomly chooses one item.
I store that into index and I remove it.
So that index is like a random location in the sequence.
I'm going to remove the item that's at that index.
And I'm going to do an insert of index at index and I'm just going to insert the same value of index, doesn't really matter.
I'm going to use timeit to time this function for both the list and the deque.
Here we have the timeit module.
And we're going to call it with insert and delete on the list, and the list we defined here above.
You can see this took a little bit.
And now let's do the same for the deque which we defined here.
And although it seems to take a little bit as well.
Here we're talking about milliseconds and here we're talking about microseconds.
Here it also run like 10,000 loops.
This one was slower so it reduced to one loop.
So deque performs at a fraction of the list and that's because inserting and removing at both sides of the sequence are more efficient in a deque than a list.
A list has to move all the items around and that's expensive.
I encourage you to look at the docs because there are a few other data types that are interesting and you can read about ChainMap, for example.
An OrderedDict is another good data type to know about.
Although I think in Python 3.6, dicts are becoming ordered by default.
That concludes day one.

So a quick overview of we've learned so far, namedtuples, an elegant and readable way to create tuples, and instead of user index zero, and index one, you can say user.name and user.role.
A defaultdict is great way to buildup a nested data structure, you don't get key errors, because the internals make sure that the value gets initialized before appending to it.
Counter, don't reinvent the wheel, so here at the top you see all the code I needed to get the top five words in a string, and below we did the same, but only with one line of code, very powerful.
A deque, so lists are your best friend, but if you have to insert at them at the start, for example, they get very slow.
So deques are great if you need to insert and remove at both ends of the sequence.
And now it's your turn.

Welcome back to the 100 days of Python and the second day of the collections module.
Today, we're going to get practical with a code challenge.
That will be highest rated movie directors.
We will load in a data set and convert it into a default dictionary of directors as keys and movie namedtuples as values.
If you want to try it yourself on scratch, I encourage you to pause the video now and read through this link and try to code it up yourself.
What I will do in the rest of this video, is to guide you how to get the data loaded into the directors variable.
So parse the CSV, convert it into defaultdict, and also will have a Counter example, how to get the directors with the most amount of movies.
So if you need some guidance, keep watching but maybe you want to try it yourself first.
Welcome back.
I hope you had fun doing the other exercise.
In the next session, I will show you how to load in the data and parse it into a defaultdict.
So we're going to load this data in and the goal is to make a defaultdict where the keys are the directors and the values are a list of movies and every movie will be stored in a namedtuple.
So let's define the namedtuple first.
We've defined a namedtuple called movie with title, year, and score.
Those are the only fields I'm interested in for now.
We need to parse the CSV and load the data into defaultdict.
I'm not going to touch too much upon the CSV part because there's a whole lesson dedicated to that.
I will write out the function and come back and comment it line by line.
And let's see if that works.
And let's get the movies of Christopher Nolan, one of my favorite directors.
Wow.
Look at that.
I can look up a director and I get a list of movies and each movie is a name tuple with title, year, and score.
Okay, let's go back to the code I've just written.
We make a function and receives data which by default is movies CSV which we retrieved here.
I initialize a defaultdict of lists called directors.
I open the data with a with statement.
Then I use the CSV dict reader to parse every line into an OrderedDict.
Every line, I extract the director name, movie title, title year, and IMDB score and store them in variables.
The year, I convert to int.
The score, I convert to float.
With data analysis, there's always bad data and this is no exception.
A value error got raised for some rows.
So when that happens, I just ignore the row.
I'm not interested in incomplete data.
I initialize the movie namedtuple and give it movie, year, and score.
That namedtuple gets appended to the director in a directors named list.
So here you see the defaultdict in action.
I don't have to initialize an empty list for every director up front.
defaultdict handles that all behind the scene.
And then I return the directors defaultdict.
Then I call the function and store the results in the directors variable and then I can look up directors.
So there's a lot of stuff you can do with this data.
Let's do one more exercise.
I'm going to use Counter to find the directors that have most movies in this dataset.
So I use a counter and I'm going to loop over...
the directors...
we stored before.
I can loop over dictionary with items which gives me the value pairs.
Then I'm going to store the director...
in the counter object.
I'm going to sum up the length of the movies.
You can do this by hand, but the nice thing of having a counter object that now I can do counter...
Most common.
Five.
And there you go.
Spielberg, Woody Allen, this is pretty plausible.
So here you got some more practice using the collections data types.

Welcome back to the 100 days of Python, and the third day of the collections module.
Now it's time to get some more practice yourself, so I encourage you to try to use the new collection's data types in your scripts.
We did the #100DaysOfCode ourselves and we made a module index script which lists all the modules we used and the days we used them, so you can go to our log and look up those days and look at the scripts that used the collections in one way or the other.
This was the script to identify if a tip was already submitted to pytip, and here we use the name namedtuple.
And this was the script I was just showed you about the module indexer, and here we used defaultdict and Counter.
So you can look at more examples for where we used those data types, but maybe you can refer to some of your code to start using collections more.
And don't forget to mention 100 days of Python when you tweet out your progress.
That's a great way to keep on track.
Good luck, enjoy, and remember, keep calm and code in Python.

Good day everyone.
This is Julian Sequeira and welcome to Python Data Structures.
So this series of lessons in going to walk you through the basics of lists, tuples, and dictionaries.
So hopefully stuff you've seen before, but this should be a good refresher.
There'll be a couple of videos to watch, but then there'll be some exercises as well.
So, nothing left to say.
Let's get cracking.

Here is your three-day breakdown for this lesson.
It's actually pretty simple.
There's not much to it because we are dealing with data structures.
Super important though, so very important that you get this down.
For the first day, we're just going to watch the videos.
Okay, there's not too much involved with this.
Just watch the videos that we have on lists, tuples, dictionaries, and then just have a play in the Python shell.
There's really not much to do.
So digest the content in the videos and then hang around for day two.
Now the second day, it gets a bit more interesting.
What I'd like you to do is follow this link here, this bites of Py code challenges promotion link.
This will give you free access to this specific bite.
This is a little challenge for you.
We just open it in a new tab.
Okay, I will log in with GitHub.
So you have to have your GitHub account ready.
And there you go.
So I'm already a premium member obviously, but this will unlock this bite for you to work on if you are not already a premium member.
This here is regarding dictionaries.
So have a good play with this.
Enjoy the challenge.
Work on it in the command line within your browser and do that for day two.
Back for day three, this gets a little more tricky.
What I'd like you to do is a bit different as well.
I'd like you to go into this data.py file which is here in the repo.
And I'd like you to just have a quick look at the dictionary and the lists that are in there.
It is pretty much a list of just the United States states and the acronym used for them.
So what you can do then is complete each one of these little tasks, okay.
It will involve you actioning or working against the list in the dictionary, pulling out data and just playing around with them.
So you'll need to pull them into some, whatever script, import them to whatever script you'll be running this from.
And just remember that dictionaries are unsorted so that should make this a little more tricky.
Alright, so that's your day three.
Just playing around with that data.py file.
Obviously if you want to play around with it in any other way, go ahead, feel free.
But this is just a couple of quick, these are just a couple of quick tasks for you to do that should give you around 15 or 20 minutes worth of practice which is what we're aiming for.
So enjoy, these are your three days and let's get on with the video.

Lists are actually pretty simple.
They're probably one of the things you're going to deal with the most in your Python journey and the easiest way to demonstrate it is to just create one right here of numbers.
So let's create a stereotypical list, all right.
We do that by using the square brackets and we're going to create five entries in that list.
So this list contains five items.
Okay.
Now, because we're dealing with numbers specifically, we don't have to put the quotes around it, okay.
If we were dealing with strings that's what we do, but we're dealing with just numbers so let's just leave it plain like that.
So that's our numlist.
We'll just call it back so you can see what it looks like.
There you go.
So it's now a list of five numbers, okay.
1, 2, 3, 4, 5.
One of the cool things you can do with a list is you can actually reverse it.
Okay.
So, now need to write some nifty code that will go through, parse it, and put all the values in back to front.
We can just do numlist.reverse() Call numlist back and there you go, 5, 4, 3, 2, 1.
Now we can do that again.
Okay, and we're back to one, two, three, four, five.
Now, if we actually go back, one thing we can do, is we can actually sort the list.
So numlist.sort(), okay.
And there you go 1, 2, 3, 4, 5 again.
And this is very handy because you can actually sort with letters as well.
Now let's say we want to actually print out all of the values inside num list.
How do we do that?
We can use a four loop, okay?
So we can go four num in numlist, print(str(num)).
So we hit enter and there are our five numbers, 1, 2, 3, 4, 5 So it's pretty powerful.
There's with just this basic ideology you can get a lot of stuff done in Python code.
Now one of the other ways you can actually create a list is to call the list function against a certain string.
Let's say we have a string called - we have a variable called mystring.
And we assign it to string Julian.
Okay, so mystring is Julian.
Well how do we convert that into a list?
We simply call list against it.
So list(mystring) is Julian.
And there you go, you see my name has just been chopped up so that each letter, or each character I should say, is now a string value inside this list.
Right.
So, what can we do?
We can assign that so l = list(mystring).
So we're assigning this here to the variable l, alright.
And we'll just call that back.
And it's Julian.
Now what are some interesting things we can do with this?
Well, there's actually quite a lot.
We can actually reference the values by their position, by their index, inside that list.
So we can go l[0] is J.
We can go l[4] is A.
You can see we got J there, we got A there.
Very handy.
What else can we do?
Well there are a few other functions we can call here.
We can go pop, and what pop will do it's actually going to return the last letter from this list.
So, the letter N is going to be returned.
But at the same time, it's going to be removed from the list.
So my name is now Julia.
Right.
We can then insert it back in.
And we use insert for that.
Now when we insert, we actually, if you look at this tool tip here that's cheating, you can see we have to specify an index and then the object.
So what position are we inserting the letter N into this list?
Well, we're going to insert it into position 0, 1, 2, 3, 4, and 5 5 is going to be on the end, so position five.
And what are we inserting?
We're inserting the letter N.
All right, it's an actual string.
Now when we call the list, there we go, it's rebuilt with the letter N.
Another interesting this we can do is we can actually replace any of these with any other letter.
So we can go l[0], which we know will actually return J, but we can replace J.
So L[0] is going to be B.
So, l is now Bulian.
Okay?
Ya, a little play on words.
Let's go with that.
Now if we wanted to get rid of the B, we could actually delete it or we could pop it.
The difference is, as I've said before, pop will return the letter in that position, where as now with the delete option it will actually delete it.
You won't even know what you're deleting.
It doesn't return anything, it just deletes.
So if we want to delete that zero we just have to type del(l[0] and there's the B gone.
All right?
Next we can do l.insert().
We'll choose index position zero.
And this time we'll put an M in.
l is now Mulian, okay.
And now, even better, with pop, let's say we do want to return something, we can go l.pop(), but now we can actually specify a position, an index.
So we can go l.pop(0), we get the M returned, and the M has also been removed from position 0 in the list.
So those are some cool little nifty tricks you can keep up your sleeve, add them to your book, because when it comes to lists, you'll end up dealing with them quite a lot.
So knowing how to pop and how to delete and insert and append, which is another one I'll show you quickly.
l.append(), let's add S.
We can append, append will always add right at the end into the last position.
So definitely keep all of these handy.
You'll be using them quite a lot.

If you're learning Python you're going to come across two words that may or may not make sense to you, and they are mutability, and immutability.
Okay, type them out on screen, mutability, immutability.
And what do they mean?
Well, mutability means it's something that can be changed, it's an object that can be changed.
You saw in the previous video that we were manipulating the mystring variable, and we were dropping j, we were dropping n, we were doing all sorts of things to that list.
Now, immutable lists, okay, let's just go with that for one second, are lists that cannot be edited, so if you tried to edit it, if you tried to pop out that j, or pop out that n, you'd actually get an error, and they're not actually called lists, they're called tuples, okay, you've probably heard that terminology by now, but we're just covering it again, and I'll just quickly demonstrate the difference between the two.
So we're going to say l, again, for list, is a list of mystring, so we can see mystring is my name, okay, and t for tuple is a tuple of mystring.
So let's just show what the two of them look like.
And the big difference here is you've got the square bracket versus the rounded bracket, and that's the telltale sign that you're dealing with a tuple, okay?
Now watch what happens when we try to edit the two, okay, we can go, l, let's actually overwrite, just like we did in the last video, so, l[0], we're going to assign the word, or the letter, T, so my name is now Tulean, no that wasn't a name I was called in school, so don't laugh, and now if we try and do that to the tuple, we can go t[0], is, let's just go with the B from the other video, and we get this error.
Tuple object does not support item assignment, and that's because the tuple is immutable, we cannot edit it.
We can still talk to it, imagine a hard drive or something, or your SD card in your camera is read-only, that's pretty much what a tuple is, so we can go t zero, we can return that, you know, we can read it, we can talk to it, we can iterate over it we can go, four letter in t, print letter.
And there we go, we can still iterate over it, just like we do with the list, the difference is, we can't edit it, and that is what a tuple is, and that's a demonstration of immutability.

Alright.
Next up we're going to talk about dictionaries, or dicts for short, let's keep saying dictionaries, just to be safe.
And they're made up of pretty much two different things.
The easiest way to demonstrate it is to just create one in its simplest form.
So we'll create a dictionary called pybites, and let's just add a few things to it.
The first thing we need to add is a key, and the second thing we need to add is a value.
Alright.
Looking at it here, there's a separator here, your colon.
And that separates your key, the key comes first, from your value on the end.
Alright, and this here is technically a dictionary.
It may only have one item in it, but it's a dictionary all the same.
Let's make it a little more interesting.
When you're adding a second item into your dictionary, you separate it with a comma.
Alright, so we've got Julian, now let's add Bob, and let's just say he's 33, then we have Mike, let's say he's also 33.
Let's just say I'm being super generous with those ages guys, anyway, there's our dictionary.
So to view what it looks like, we just type in, pybites.
And there it is there, the three different items.
And these link to each other.
So the key is Mike, the value is 33.
The key is Bob, the value is also 33.
Now notice when we printed it out, it printed out in this order.
Well this order is not explicit.
With a dictionary, when you're passing it, when you're listing it out, when you're displaying it, there's no guarantee that the data is going to remain in the order, okay?
That's very important to remember with dictionaries, they are unordered, okay?
Now let's say we wanted to create a dictionary but we didn't know what values we were going to put into it from the start.
We'll just do a quick little side track demonstration here.
You would start off just as you would do with a list that's empty, except you're using two curly brackets instead of the square ones.
So people is now an empty dictionary.
There's absolutely nothing in it.
To add someone to it, this is the tricky part, this is where it gets a bit different.
What you need to do, you need to actually pretty much in square brackets, you need to choose the key, alright?
In this instance the key for a list of people is going to be Julian, and we're going to assign that key a value of 30.
So the dictionary people, we're creating an entry, a key, of Julian, and we're assigning it the value of 30.
We list out people, there we go, we see that exact same formatted dictionary as we did when we explicitly defined it up here, okay?
We can add to it again.
We can go, people, Bob, and we can assign Bob the age this time of 103.
And there we go.
So same thing, right?
This time we just populated an empty dictionary.
But either way it all comes out the same.
Now the way you interact with the dictionaries is a bit different to lists.
The way we view just the keys, just these keys here, forget the values for a minute, is we use keys.
So we can go pybites.keys, and there are our dictionary keys.
Julian, Bob, Mike.
The same things for the values.
pybites.values, and there we go, the three values.
30, 33, and 33.
Now what if we wanted to see all of this?
Now this is all well and good because we're in the shell.
But if this was an actual application you can't just type in the name of your dictionary and expect it to print out, right?
This is just all standard out through the shell.
So the way we would actually print out, first of all, the way we actually see each pair of dictionary items is to use items().
We can see Julian 30, Bob 33, Mike 33.
These are our three different dictionary key value combinations or items.
Now how do we pass over all of that?
Well we do it just how we would with a normal list.
We can go for keys in pybites.keys, print keys.
There we go.
We can do for values in PieBites.values, print values.
Okay?
This is all very similar, you can see the similarities between dictionaries and lists.
But last but not least, what if we want to print out all of this information, not just keys, not just values, without having to run two separate for loops, which would be quite un-Pythonic right?
Well, God love Python, we can go for keys values in pybites.items(), so for keys and values in PieBites.items Print keys, and remember we have to do the string thing here, values, and there we go.
Julian 30, Bob 33, Mike 33.
Now obviously that is not very pleasant on the eye, so we can do our standard string formatting.
So we can go, for keys, values in pybites.items(), print, string is digit years of age, keys, values, okay?
And there we go.
We can see Julian is 30 years of age, Bob is 33 years of age, Mike is 33 years of age.
And that's dictionaries.
It's actually that simple to iterate over dictionary, and that's pretty much it.
You've got your key, and your value.
And you just do it over and over again, just like a list.
So enjoy your dictionaries, and get used to them, because you'll be playing with them.

So that's Python data structures.
I hope you enjoyed that little overview of lists, tuples, and dictionaries.
For the next couple of days, you will be working through the exercises outlined in the three-day overview which was the second video for this lesson set.
But before we get to that, let's just quickly quickly recap everything we've done.
So for lists, lists can be sorted with sort, go figure.
And then worked through how to iterate over a list using a for loop.
We also discovered pop.
Okay, if we remember, pop just returns the item in the index specified.
It actually removes it from your list and then returns it.
del simply deletes it from the list, doesn't even return it.
insert does exactly as its name implies.
It will insert the second argument into the index specified in the first argument.
And then append will just tack whatever it is right on to the end of the list.
Now immutable tuple, okay, we know that this means we can't edit, it can't change, okay.
So we've got a tuple.
We've created it using tuple itself, okay.
Now we print out the contents just so you can see that we have the six items in this tuple, my name, the letters of my name.
And then when we try to manually override it with a simple list substitution there, t[0] we're going to assign the letter m and then we get the fact that it's a type error, it's a tuple.
You cannot change it, it's immutable, okay.
Now onto dictionaries.
We created it manually.
We manually created a dictionary, okay, simple.
And then this is how you return the keys using .keys.
Okay this is just the keys at the front, just the names.
Then we returned the values in the same way that we do with the keys.
We can get just those values, the 30 and the 33s.
And then we can get everything as a whole.
We can get each item, so the key and the value.
We can do that using items().
And then we showed you how to iterate over, over the item itself so we can get the keys and the values, not just one or the other.
And using a forloop, we can then substitute those keys and values into some sort of a string, into some sort of text.
And that's really useful, something you'll probably do quite a lot.
Okay, your turn.
So for this next couple of days, as we discussed in the ReadMe earlier, we are going to work through the code challenges bite that you have a free code for so go ahead and use that.
And then for day three, you're going to work through that data.py file which in the repo.
So if you have any sort of thoughts on that or any confusion on that, just head back to video two where you look at the ReadMe and there's a good explanation there as to what you're going to do for days two and three.
So that's Python data structures.
I hope you enjoyed it.
Keep calm and...

Welcome back to the Hundred Days of Python.
In the coming three days I will show you how you can test the program with pytest.
It's a popular testing framework, often preferred over the standard libraries unittest.
And you will see why.
For this lesson I prepared a guessing game, which lets you guess a number from the command line.
And although is a simple program, it has a lot to offer in showing how to use pytest, for example to validate errors, capture standard output, mocking certain functionality, and more.
By writing the test, I will also show you how you can use coverage, to see how much of your code base, or in this case, the script is covered by tests.
And by the end of this session, it should be easy for you to write tests for your code, an important skill.

Before getting into the nitty gritty, why do you want to test your code?
And here I have a useful link that introduces testing in Python, and from my own experience I think the main thing you want is to have a regression test suite.
As your software grows, it becomes more complex and you need to make sure that all the previous code keeps running.
If you have a suite of tests that are fast and you can run every time you make changes, you have a much more reliable application.
Down here, what I like about this link is that there are rules about testing, and I'll highlight a few.
So every test should test one thing and be small and independent.
One test should not influence the other test.
And set up and tear down, which we will see towards the end with fixtures, is a way to guarantee that.
Tests need to be fast.
Your test suite will be growing, and you will run them often.
You don't want slow tests to delay your development.
Testing should be automated.
Again, because you run them often, it should be as hands off as possible.
Fixing bugs.
If you find a bug in your application you usually want to write a test first to show that the bug, or even document the bug, and then fix it, and then you always have that test to verify that that bug does not occur again.
And there are couple of other items.
This is good link to go through when testing is really new to you.
There are various frameworks in Python to facilitate you writing tests.
We have unittest, doctest, pytest, hypothesis, and tools like talks that lets you test various configurations or environments, unittest2, and mock.
In this lesson we are going to focus on my favorite, which is pytest.
pytest is a framework that allows you to write test for your Python code, and it specifies a set of rules, and it has a couple of features that really helps you write better test code.
Alright, enough theory, let's move on to the next video where I pip install pytest and pytest coverage, and we look at the example for this lesson.

Let's make a virtual environment and install pytest and also pytest coverage, which I will use to show you how much of the code is covered by tests.
Let's head over to my terminal and make a virtual environment, and I'm using Anaconda, so I'm pointing virtualenv to the Python binary in my Anaconda path.
As by convention, I use venv for all my virtual environments, that I can just run this.
Then I have another alias, ae, which will source the venv activate script.
So that's basically enabling my virtual environment.
And we have nothing installed, and let's now install pytest and pytest coverage.
Let's look at that little guessing game we're going to write test for.
It lets me guess 5 times, a number between 1 and 20, and it will feedback if I'm too low or too high.
That's funny.
9 is too high.
Wow, I'm a good guesser.
Right.
You see here, there are some validations we need to do.
So a number I cannot guess again, I need to have a valid number, I need to have a valid range and I cannot just do strings.
So that is all stuff that's in that program I will show you next and we have to write test for.
And we have 5 guesses.
Right.
So here, I didn't guess it, and it bailed out after 5 times.
So that's the program.
Now look at how that looks in code.
So we have to max number of guesses, the start and end of the range, we have a function to get a random number, using the random module.
So we have to constructer that sets the guess's internal set, sets the answer, and it sets when bullion to falls.
Then we have the guess method which takes a user input and see if there was actually a number, if it was an integer, if it was in the range, and if it was already guessed.
And if nothing bails out, then we can add it to the guesses set, and return it.
Then we have an internal validate guess method, which just sees if the guess equals the answer, otherwise, it gives feedback if it's too low or too high.
If a property of number of guesses that's basically just the length of the internal guesses variable, and there's a dunder call which makes that I can instantiate the class and call that object as if it was a function.
That will initialize a while loop, checking if the number of guesses is below the limit, try to get a guess, check for a failure error, print that and continue, validate the guess.
If there was a win or done, make sure that the print guess or guesses, reset the win to true and re-break.
If the while loop exits without breaking, then we know that you have guessed the maximum number of times and you didn't find it.
Although some people are against this, I do find this construct useful in this particular case.
This gives us a nice playground to start using pytest.

Right, before diving into writing tests for that program, let's just quickly look at how to write a test with pytest in the first place, and how to run it from the command line.
And first of all I want to contrast it with unittest, which is in the standard library.
So here at left you have a super simple program, it's almost ridiculous.
But it's Hello name, takes a name and just returns the hello name string.
And look at the amount of code you have to write in unittest to get a test for that running.
Because it's class-based, so you have to subclass another class, write a function, and use a self.assert notation so let's show that next.
Alright, so import unittest, import the program, make a class, inherit from unittest test case, write a method, which needs to start with test to be recognized as a valid test, use self.assert equal notation, call the function, and check for hard coded output.
If the files run as a main script, call the main method on unittest.
Right, and it works.
And it fills if I change the return, and that's good, and notice that pytest can run unittest code, so pytest can be run from the command line, without any switches, if you'll look for files that start with test, and run the methods or functions in there that start with test.
So that means that I can even leave out this main block and it should still work.
Right, but here comes the first benefit of pytest is that you can write test code in a much shorter way.
I don't need unittest, I don't need the class, I can just define functions.
They do need to start with test, and instead of the self.assert equal, I can just use the classic assert.
And this should work.
It does not.
Self is still in the program because it was still in the function.
Same output.
Make it fail, and here is the second advantage of pytest.
The output is much nicer.
unittest was definitely not bad, but pytest, especially if you go into more complex operations and errors, it's a great debugging tool.
So look at that.
Instead of what was it?
Oh that doesn't count one two three, well it was still short because this is an easy program, but you can already see that this is way cleaner.
So that's the simplest of pytest programs and I just wanted to show you how it differs from unittest and how you can run it from the command line.
Just the basic steps to get you up and running.

Alright.
Back to our guessing game.
So, how do we want to test this program?
Ideally, you want to test one function or functionality in one pytest function.
So let's start with the Get Random Number.
I'm going to open a file test on the scoreguess.py.
Again, using pytest I have the convenience to not having to import any module to run the test.
What I do need is to import the actual program.
So from guess import get random number and the game class.
Now, one thing I want to show you in this video is how to mock an object.
Because Get Random Number, as you can see at the right, uses a random integer from start to end.
And random returns to something randomly every time.
So how do you actually test that?
And the way to do that in testing land is to mock an object.
And for this I'm just going to use the unittest patch method on the mock module because it's a perfect fit for this scenario.
So from unittest.mock, import patch.
I actually need to import to random module because that's the one we're going to mock.
And you can use it as patch object and that's to random module.
Just specify the function or method you want to patch.
And then in your test function you can pass in an argument and you can give that argument a fixed return value.
And that's key because instead of having random return something else every time you can give it a fixed value.
So it's kind of an override of what randint() normally does which is randomness.
Now we're saying every time random gets called it gives us 17, and it makes that at least we can call our function, which is get random number, and we know that it returns 17.
Yeah, and this is pretty basic, but it should show you how you can override certain things in your program you cannot really control and I have another example later about the input function where we ask for user input, which is another area that can be anything, so you want to mock that out.
So with this code written, let's go back to the command line and run this test.
And I'm using Control Z on a Mac with foreground to toggle back and forth between my file editor and the terminal.
And here I can just run pytest, and that's funny because the previous example I put in a hello subdirectory and it's actually cool that we see this because pytest is smart enough to look recursively for test files.
So in this case it found two and ran them both.
So I'm going to move this out to somewhere else, because we now want to focus on the guessing game.
And yeah it runs fine, and what I also want to see from now on is how much coverage we have of our tests.
So we installed pytest-cov and to use it was a bit of a trial-and-error for me, but I found this syntax to work well for me.
So I want a coverage report term missing coverage dot current directory.
And the term missing is cool because it starts to index all the-- okay, I actually have to give it something more specific because it starts to look in my virtual environment.
Alright, what I did was in the end moving the files into a subdirectory so we got our venv and we got our guess new subdirectory with the script and the test file in there.
And when I specify a subdirectory in the minus minus scuff argument then it just rounds on our code, and what's cool about the missing argument is that it shows the lines in the code that are not having test yet, which is of course is a lot because we just got started.
But even so, we have 24 percent coverage so we are up for a good start.
And you can then map those lines back to the actual program so this is not tested 29.
This is not tested, et cetera.
One final thing, as I use Vim I'm going to use the coverage command quite a lot so in my Vim RC, which I mapped to VRC to edit it, I have a comma T which maps to save the file or run a command with the exclamation mark and then I run this command and I'm going to-- yeah, I think that's fine because we're going to work in the guess directory from now on and the venv is not sitting there so the dot should work there.
And we can confirm that by going into guess, run a test, run a coverage report with a dot.
Yeah that's fine.
So when I'm writing my tests, I can just hit comma T, it saves the file and it runs my coverage.
So that's a bit of Vim trick or shortcut for Pycharm or another editor.
There must be a similar way to do this but this is my way of running coverage with one keystroke in Vim.

The second thing I want to test is the guess method.
It takes a user input and input is not static, you can change and it can be random.
Even worse, when you run this program it's waiting at the prompt to get input so your test would hang, quickly demo this.
Let's make a game object, and run it.
Actually it's not hanging, it's throwing an error, pretty soon, reading from standard input output is captured, so that's cool.
And look at the output, it's pretty verbose which is a nice feature of pytest.
But anyway, we definitely don't want to use input literally in tests, so I'm going to use patch again.
And I'm going to patch the built-ins input, and this is another way of marking.
Here I can give it side effects.
And a list of expected returns in a row.
Because I'm having all these exceptions here, I'm going to give it a bunch of inputs to go through all these scenarios and see if each scenario throws the value error or accepts the guess as a correct one.
And will also show you how you can check for exceptions in pytest which are important because raising exceptions is a common Python pattern.
So what we're doing here is setting up a sequence of return values as if input was called that many times.
So it will return a 11, then 12 as a string, then bob, then 12, then five minus one, 21 seven and None.
And those are the values that we're going to work with, and you have to give it an argument, it can be called anything.
And now we have some return data from input to work with.
So I'm making a game, and again, the constructor sets defaults for all the internal variables, so that should be fine, and then I can start to make assertions.
And the first two side effects or returns are good.
and again I will show you guess, so guess goes through getting an input, looking for all the bad inputs and raise a value error if so, and if it's good it ends up adding it to it's internal set which is the underscore guesses, and returning the guess.
So 11 is good, 12 is a string should be fine because that can be converted into a int.
The second one, bob, is not a number.
And the way in pytest to check if an exception is raised is to use pytest, and you need to import that, and do a with pytest.raises, the name of the exception and then the statement that would trigger that exception.
So the next return value from input in the row is bob's string, if I call guess with that it should raise a value error and we're telling pytest to check if it actually raises that exception.
And the same is true for the next one with is 12.
If I guess again, the guess is already in the guesses set and the function manually raises a value error.
I can just copy this and it will be the same.
So every new call of guess triggers this input statement or call, which triggers a new value in the return list.
So 12 is done, five should be fine.
Right let's run what we have so far because it's a lot of code and see if it works.
All right that looks good.
So now let's do the complete list again.
And after five comes minus one and 21 which should be two exceptions because they are out of range.
Another good one, and finally None should not be a good one, that falls into this one, if not guess raise a value error, please enter a number.
So we wrap up with...
So that's save, Control + Z, pytest you're still happy.
I mean if it would say...
Game guess returns None when given None it will fail.
So we have please enter a number so it actually raised a value error, and it was not catching that.
Here I do, and it works.
So that's how I use mocking to circumvent this input function waiting for input.
I'm going through all these scenarios by giving various side effects.
Next up, validate guess.

The last video I forgot to run my coverage report, so let's do that now.
So I'm in test, and we see an improvement.
We went from 24% to 56%, and it still shows the missing lines, which are going further down into the script.
So let's continue.
Next up, validate guess.
And the doc string says: verify if guess is correct?
If so print, guess is correct, otherwise it's too low.
Whoops.
Guess is too high or too low, depending what it is.
And it returns a boolean.
So let's write: def test, validate guess.
And I'm going to show you another feature of pytest, which I will explain you in a bit, which is capfd.
And that will capture the standard output of the program and execution.
Very useful because for this method, I not only want to check for the boolean return value but I also want to see if the actual output by the function to print, that the function does because we made this a game, prints to the console and I want to have accurate information for the user.
So let's make a game.
I set the answer.
So let's validate that one is not a winning number.
Validate guess.
So I call this with one.
One goes into this method.
It checks against answer.
If it's answer, true, if it's not answer, false.
And to say false in pytest, you can just say, assert not this method is truthy.
So this return false, not false is true.
So let's run this.
And that one passes as well.
Then of course it's easy to do the same for higher an assertion.
So if it's three, it's still not good, and if it's two, it's good because that's the answer defined.
And now back to capfd.
If I actually want to see what the print is printing to the console, because that's what you see before, if we run the game.
It's printing these kind of feedbacks to the user.
So I want to test if these are what I'm expecting.
And one very useful trick is to redirect the output, standard output, the error I just throw away and I use capfd, which I passed into the function, and I'm calling its readout error: method.
And let's just see what that gives me.
I'm not going to do this for now.
If you want to print inside your test, one way with pytest to show that to the console is to add the -s.
And that actually stands for: shortcut for capture equals no.
So it's not capturing the output, meaning in this scenario it prints it to the console.
So when it's been intermingled into these dots but this is the actual print statement, there's also a new line.
So one is too low.
I captured that in the output variable, which I printed to the console.
So capfd is very cool to capture output printed by your program.
And now it can make assertions on that, as on any other thing.
So I can say, out, and that's strip of the new line, right strip equals one is too low.
Save, control z, pytest, and it worked.
And if I would say too high, it would fail.
And look at that nice output.
So that works and I can do the same for the other two.
So let's just copy this.
Too high and two is the right answer.
So in that case, two is correct.
Let me just not do it 100% correct I you'll see.
Oh, one is too high.
Oh sorry, this is three.
So you'll be targeting a lot between running tests and writing test code, that's something you need to make a habit of.
Still something bad: assert not true.
Wait a second.
Ah!
Typical copy and past error.
Ah.
And here, it's nice how pytest shows that, not only the diff, but also, the actual character that's missing.
So that's now very easy to spot and fix.
There you go, we are green again, and let me run comma T.
Wow we have 68% coverage now.

Alright for the final two test methods that I actually want to run a whole game from end to end.
And I'm going to use the same technique as before because we're still stuck with this input method that requires us to input data which we don't have in an automated way of running test with pytest.
So I'm going to do a patch of the input again and I'm going to just simulate a whole game.
So I'm going to enter 4, 22, 9, 4, and 6.
They're going to play a win scenario and I'm going to give it the input which is the requirement of the patch but I'm also going to capture the standard out as I did before.
So I make a game.
And I need to give it a right answer to make sense of these numbers.
So in this scenario win but at the fifth attempt, which is 6.
So the answer is 6.
I call the game and assert that the game state is underscore win equals true.
Let's run this.
Okay.
So what it actually did, calling game, is it went through all these numbers and when it got to the final one, the fifth attempt, it asserted answers true, so the win was set to true.
And again, you can see in the call, an actual look that when validate guess returns true, which is on the previous test, there is an intermediate variable win set to true and it also sets the inner, or internal variable win, to true.
And that's what we are asserting here.
But what I'm also interested in is how the output looks of the program.
So I can just again do capfd, read out err, and you can also just call this with zero indexing then you don't need the throw-away variable at all.
And I have a bunch of expected states which I'm going to copy in.
Let's actually assimilate this program.
So I have these steps I'm just going to hard code the answer for a minute, 6.
So these are the steps.
So what the test is doing is pretty pretty cool.
So I'm simulating 4, 22, which is not in range.
9, 4, 6.
So here you see the typical program, and that's what we are asserting here with these expected values.
So 4 is too low, number not in range.
9 is too high, already guessed.
6 is correct.
Plus an additional statement of it took you three guesses.
So let's reset this.
Let's clean the output from capfd a little, with a list comprehension.
For line in out split by new line.
Only take lines with one or more characters.
So ignore blank lines basically.
And give me the strip lines.
So I'm stripping off new lines.
Alright.
And then we can just match it line by line.
I can use a zip to look over two sequences.
So you have line and expected in zip output and expected.
So this will look over expected and look over output in parallel.
So the first value of output would match the first value in expected and etc.
Alright.
Look at that.
What's my coverage?
94 percent, very nice.
So they're only a couple lines missing: 83, 87, 88 And 87 I'm okay with because this is just a calling code if this is called as a script, which we've done before, if I call it like this.
And line 83.
Let's see if we can get to this scenario as well.
So this is a lose scenario, where we tried it 5 times and still did not assert the answer.
So let's set up a filled scenario.
Test, game, lose.
That's going to follow the same signature as above.
But I need more stamps.
So let's do a none, which should not count right away.
5, 9, 14, 11, 12, doesn't really matter because what I need to do now is to give it an answer that's just totally different.
So let's make a game.
And the game answer is 13.
So it's not in all my guesses.
But this also test that none doesn't count towards my guesses.
So I can actually do 6 inputs and this would be the fifth guess.
So I'm actually getting here in the first place.
Play the game.
I won't win this game.
And it should pass, right, but the coverage should still be the same because I'm not hitting that line yet.
I do.
Yeah, I did.
So just to recap, this was line 83, which corresponds to this line.
And what happened here, I played, actually what I forgot is that this plays the whole game.
When I launch game, it goes through all these outputs.
And having guessed 5 times, and not asserting this answer, I did get to the L's block.
I can actually show that.
If I turn on non-capture mode.
See?
It just prints the whole thing.
5's too low, 9 is too low, then I guessed 5 times.
Answer is 13.
So I made it to this actual print statement.
And that was the final thing to actually increase the coverage.
If you take main out, we have a 100% coverage of our tests.
You still need to have a critical eye to your code and your tests because one thing is to test it, but, for example, I can get a 100% on this earlier validation of the guesses, sorry, this one, but here I made sure that I'm going into all the different kind of value errors.
Actually let's do an experiment.
If I'm not doing the not-a-number test, would my coverage go down?
Wow, look at that, how cool.
So I took a test out and now it's complaining that 35 and 36 are missing.
And 35 36 is they should be a number, and that's the thing we just deleted.
So I had a string here, and that not-a-number should raise an exception.
And coverage spotted that.
So that's very cool, that you can just pinpoint exactly which code is not being tested versus which code it is.
But again, you still need to have a critical eye of what you're testing.
Because one thing is to have all your lines, some were called, but the other thing is how you call them.
What are you testing?
Are you testing all the edge cases?
So testing is an art in itself.

1 final thing is when to write your test.
I think the motto "having tests is better than no tests" is the most important, but there is a whole style of test driven development, which is to write your test before your actual code, and to drive your design by those tests.
Let's write the Fizz Buzz program, which is a children's game, and it basically is a sequence where numbers divisible by 3 and 5 return Fizz and Buzz, and if they're both divisible by 3 and 5 it returns Fizz Buzz.
So let's write that program, but do it in a TDD way, by writing the tests first.
And I'm going to use the repetitiveness of these tests to also show you a nice feature of pytest, which is parameterize.
So let's do it from the ground up.
Test Fizz Buzz.
So let's give it a number, and it should return Fizz, Buzz, or Fizz Buzz, or number, right?
So if I call it with 1, it should return 1.
If I call it with 2, it should return 2.
3, Fizz.
Actually let me stop there.
The TDD way would be to fill at the earliest possible way.
And just start adding code in small increments.
Alright, so now it is recognized, but it takes zero positional arguments, but 1 was given.
So here 1 was given, but I'm not accepting 1.
So let's just hard-code that here.
And now the return's None.
So let's just return 1 for now.
And the second test fails, so let's then decide to return N.
And then if it's 3, it should return Fizz, okay, so we need some sort of if statement, right?
Alright, that works, okay, let's move on then, 4, 4.
That still works.
5, That's not working, okay.
And let's accommodate that, return Buzz.
And we're green again, cool.
And you already start to see a lot of repetition, right?
Now there's a cool feature in pytest, called parameterize.
And let me just copy over the whole test.
So I can give that a list of tuples of argument, where I call the function with, and the return value.
And then in my test, I can just do this 1 time, call Fizz Buzz with arg, and test it against return.
And look at that, how I put all the parameters in a decorator, and I avoid having to write assert, assert, Fizz Buzz, Fizz Buzz, Fizz Buzz, over and over again.
So this is pretty neat.
I think you will find a use guys for this.
I need to pass them into the function.
And I see that Fizz does not assert Fizz Buzz, and that this particular call, so if it's 15, it should do Fizz Buzz.
There are various ways to solve this.
Let's do here, then return, Fizz Buzz.
As these are returns, I don't need an elif, because these are like, or early return, or continue.
So let's see if this works.
And this works.
And notice that it's also nice that pytest gives a dot for every parameter test.
If 1 would fail, how would that look?
Ah, we already saw that, right?
We still get all the dots, and you see the actual position of the tuple that failed.
And that's also, again, nicely indicated by the output.
Alright, so, this was a little bit of TDD, and also, a nice feature of pytest, parameterize, that you probably want to become familiar with.

Let's review what we've learned so far.
In this lesson we used a guessing game to write pytest.
And the game went like this.
There were a maximum of 5 attempts to guess a number and you got feedback if the number was too low or too high.
We compared pytest to unittest and saw that it's pretty succinct.
For unittest you need a class and self-serve equal syntax.
pytest just lets you write functions.
And this is a very simple example, but you're already see that pytest requires far less code.
pytest has nicely formatted outputs.
For example, if a fail, then we pass.
We saw pytest code to show how much of our code was on our test and where tests were missing it showed the lines in our script.
We saw mocking in action, by mocking out the building input.
Because we wanted to control what the input function returned when playing a game.
We also saw an example of mocking out the randint() function.
We also saw how to test exceptions with pytest, which is pretty easy.
And we saw how to get your outputs from your program using the capfd.
And that's nice because then we can check if the output was as expected.
And here we have a complete game, wrapped into a test.
And then we did some TDD on Fizz Buzz, writing tests, writing a bit of code, writing for more tests, some more code, some more tests and some more code.
And notice here we used the parameterize feature to prevent duplicating a lot of test code.
So it takes a list of typals and in the body of the test function, I can just do one assert.
So, that's how we kept it dry.
And again, the output is very nice.
Alright, that was quite some material to take in.
I hope you are more comfortable with pytest now.
And now, it's your turn.
Keep calm and code in Python.

Welcome back to the hundred days of Python.
This is day two of the pytest lesson.
Yesterday, we went over quite some concepts, and now it's time to get practical.
So here, I pointed to a code challenge, which is, not surprisingly, writing tests with pytest, and it consists of going through your code, see where tests are missing, and add them.
Another option is to test a Flask API, or even contribute to open source.
There's a lot of great work being done, and not all might have test coverage.
It might be a useful way to get into those projects, to start adding tests.
So basically that, if you want to follow along, you can clone our challenges repo and make a branch and PR your work, because we're always curious to see what you come up with.
Another way, if you want to just look at some tests, is to head over to our bites.
Every bite is tested with a couple of pytests, so each bite under the tests tab shows you pytest in action.
And a day generator...
And here is the exceptions being tested by pytest again.
One syntax we did not see is the pytest .fail.
This basically where it raises an exception when it should not have, alright.
Then I want to point you to one more resource and that's Brian Okken's Test and Code.
He has this podcast dedicated to testing in Python and he is the author of Python Testing with Pytest which is a great book.
I only showed you a couple of things.
This book goes in detail on how to set up pytest configuration, fixtures, and a lot more.
So if you're serious about pytest, this is a great resource.
And that's it, today is all about getting practical and tomorrow I'll check back in with you how it is going.
Good luck.

Welcome back to the 100 Days of Python.
Day three of pytest.
And the final day where I want you to get some more exercise.
And in all honestly, I recorded this lesson and something was bugging me.
Fixtures.
I mean, fixtures are described as the killer feature of pytest and I did not cover them.
But no worries, you will learn them today.
I made this article explaining everything you need to know to start to use them in your test code.
And okay, it's a pretty long article, but it's not too hard to actually learn and do.
There's a practical example in this article.
If you read through this, then you will see that it's pretty easy to set up.
But it will make your test code a lot better.
So, I ask you today to come up with a use case, so it can be to set up and tear down a database, but it doesn't have to be a database.
It can also be an object of any kind, a class, or something that requires setUp and tearDown or even only setUp.
But I really want you to try to learn this skill.
As you know from unittest, setUp and tearDown are commonly used, and pytest fixtures are the way to do that.
So it's critical to understand this.
That's another day of practical exercises and I think by the end of this day, you'll have a good grasp on pytest.
Which will go a long way, because as we mentioned before, writing test is a very important skill.
Good luck and let us know what you come up with.
Use the #100DaysOfCode and feel to include Talk Python or PyBites in your tweets.
All right, have fun.

Michael Kennedy here and I'm going to be your guide for the next three days.
We're going to have a lot of fun working with classes and objects.
This is one of the fundamental ways to model things, concepts in your application.
And you'll see that classes very naturally map to sort of real world ideas and more general stuff, that gets more specialized like, say, a car versus a Ferrari.
Right, a car is this general idea.
A Ferrari also is a car but a more specialized type of car.
And we're going to actually build some really fun games.
We'll build one in the demo and then I'll hand off a separate game for you to build.
So we're going to build this little Dungeons and Dragons wizard game.
It comes in just says, "Hello, it's the wizard game." And we have this wizard, Gandalf.
And he will encounter various creatures in his little world.
And he has three options: he can attack, or run away, or look around.
And so you can see we're entering various commands.
R, A and L.
So, first we run away and then the wizard sees a bat.
It's not very strong so he thinks he can attack the bat and win.
He does.
And was very, very close actually.
They basically hide but he had the element of surprise, so he beat the bat.
And then he can look around and see what else is there.
The toad, the tiger; not so scary.
Level 1000 evil wizard then he's probably getting away from that thing.
Alright, so, this is what we're going to build and we're going to build it by modeling these ideas in this little game using classes.

Before we actually write the code and get into the syntax of working with classes, I want to just talk briefly about the idea of two things: inheritance and the difference between classes and objects.
So, in our game we have this concept of a creature, how it'd be like the tiger, that would be, say, the dragon, the bat that the wizard defeated, things like that and in fact, the wizard himself is also a creature.
This creature concept has the basic ideas of what it means to be an actor in the game.
It has, let's say, a level, a name, and it can sort of defend, at least against being attacked.
But we can, say, well, there's special things in the game that have more distinction than that.
So, there's a tiger, maybe the tiger has a special way to defend and so its mechanism for defense, it's a little bit different than, say, a toad or a standard creature.
We have a dragon, maybe the dragon takes into effect whether it can fly, whether it has scales, whether it's fire breathing, things like that.
And this aspect of the dragon means we probably need to model those features that make it different from a creature separately.
So it's like a specialization of this creature.
Now also, the wizard itself.
When you model like this, you're modeling what's called an is-a relationship.
So, the tiger is a creature.
The dragon is a creature and so on, right?
So tigers are creatures.
So we're going to model this type of thing and I'll show you how simple this is to do in Python.
The other important distinction to make over here is, let's look at this wizard concept.
You need to think of these classes that we're going to define.
I haven't shown you how to do it yet, you may know but if you don't know, you got to think of these as blue prints.
Let's think about a tiger for a second.
There's a tiger that's in the San Diego Zoo, there's a tiger that's in the wild, in the jungle.
These tigers were created from the same blue print, from the same DNA.
That's kind of like the class.
But the actual tiger in the zoo and the tiger in the forest, those have different properties and they evolve in different ways over time.
They're not exactly the same thing.
So you'll see, the same thing is happening here in codes.
So we have this line gandalf = Wizard() and this line is going to create a new wizard from the blue print of the class.
It's going to create what's called an object.
And over here we're going to have sort of in memory this thing, it knows it's a wizard and it knows its name is Gandalf, and it's Level 70 and those can change, sort of on their own.
But the evil wizard we're creating, it's going to be a separate thing out there in memory called evil wizard with the name and the level 100.
And once they're created they can be changed and evolve independently like the wizard that is Gandalf can level up to 71 and it would have no effect on the evil wizard.
The thing at the bottom of the two arrows, the wizard Gandalf and the wizard that's evil, those are objects.
The modeling blue print thing at the top, those are classes.
Hopefully that illuminates the confusion around those which is always hard when you're getting started.

Alright, let's write some code.
So, we're going to begin this first part of our demo by simply creating the general skeleton and flow of our application.
We're not going to actually do anything with classes at all, but we're going to get it ready to.
So, let's come over here and add a new Python file.
I noticed there were none at all, and I'll just have this called program.
In here we're going to define a main method, oh, I've got to configure this, hold on.
Alright, now Python is happy.
So, what we're going to do is we're going to do a little print the header and that's just going to show these methods don't exist yet, but just think about the ideas.
Then we're going to run the game loop, okay, and it's going to go around and around and run.
This concept of a game loop you'll find out in a second, but let's put this print header here first.
Now maybe we'll put something more interesting here later, but for now, we'll just do like a little line and something like, wizard game.
Or, something like that, that should look pretty decent and maybe a little divider there, as well.
Then, for our game loop we're going to come in and we're basically going to create the various concepts in the game.
We'll create a number of creatures, we'll create our hero, then we'll just say while true, we'll sort of like ask the user for action.
Then we'll say like if win or exit, and we'll just brake out of this loop and then we'll just say print, goodbye.
Alright, so we just go around and around, and keep asking the user to sort of control the hero.
Do you want to attack?
Do you want to look around?
Things like that.
Now, there's a bunch of writing here that is actually not super interesting for you to watch me type it out.
So, let me just paste a little bit of code here to a more full featured version of what I just described.
There we go.
So, now we're going to come in and we're going to create our creatures.
We don't have any creaturs yet, remember we have to model those with classes and that's one of our primary goals.
We're going to create a hero.
We're going to go around and around, we're going to grab randomly, choose one of the creatures to appear.
We're going to print out details about the creature that has appeared.
Here's the input asking the user do you attack, run away, or look around.
Like what is your hero going to do, and then we just check.
Do they type A?
Do they type R?
And so on.
So, this not super interesting yet.
I'll put a little pass here to show it's not upset with that section there.
It's not super interesting yet, because we don't really have a hero.
As you can see right here, and we have no creatures.
So, let's go and model that next.

Okay, so here's our general program.
Let's put that aside for a minute, and go work on the various creatures that we're going to model.
And I'll call this actors to kind of say, here's where I'm going to put a creature, the wizard, and so on.
So what we need to do is create what we talked about, something called a class, and this is going to be the blueprint of a creature.
So, the way you create a class in Python is really straightforward.
You say class, and then you say the name of it.
Okay, we're going to model everything at the lowest level as a creature, and use a colon to define the block that is the class, like you do all the various blocks in Python.
Now, there's a couple of things which we can do to get started, but, commonly, you want to store pieces of data and behaviors with the creature.
So, for the data part we're going to use what's called an initializer.
So we'll say def init, and there's this dunder init, in fact if I say double underscore, these methods are called dunder because it's double underscores on both ends.
These are all part of what makes up classes, and they all have special meaning.
The one that we care about is this init.
This is very common.
So down here, we might want to have a name for it.
Equals, let's call it, toad, and it might have a level.
And the level of the toad is 1, or something like that.
Now, this is not really helpful because every creature is going to be a toad of level one.
So what we can do is, we can when we create them say, this particular creature is a tiger, this creature is a bat, and so on, and so we could pass in the name here, and we could pass in the level.
So we're going to go like this.
So, here we've defined this blueprint.
What is a creature?
A creature is anything that can be created and given explicitly a name and a level.
We could go farther and give those types, as you can in Python 3, but we're not going to do that right now.
The other thing, this is the data part we have so far, the other thing is going to be some kind of behaviors around it.
So, a creature, can, let's suppose that one creature can be defensive against some kind of attack.
So we could say it like this.
So the creature's going to roll a defensive roll.
And now, you might want this to be somewhat based on the creature's level.
So let's create some sort of randomness.
Let's say roll equals, well I'll go over here and we'll say random.
Now this comes from the standard library so we have to import it here, put like that at the top.
Then we can say randint(), and you give a lower bound, of let's say 1 to 12, and if you want to know whether this goes from 1 to 12, 1 to 11, and things like that, we can say view quick documentation.
And what does it say?
It returns a number such that A, the number that comes back is less than or equal to the upper and lower bound.
Perfect.
So this is our, let's say, 12-sided die.
And then we'll return roll times self.level.
Anytime you want to refer to your own items, your own values, you have to say self dot.
So to get back to this, we say self.level.
Alright, so this is going to be some kind of defense here.
Now let's go and create our various creatures in the game.
There's this little to do here.
We had a couple of things, we had a bat, we had a toad, we had a tiger, we had a wizard, an evil one, and so on.
So we'll type creature and of course we have to import that at the top here, just like any type.
We go like this and if I ask Python to show me the parameters, you'll see it takes a name.
So let's call this a bat, and this is a level 5 bat.
I think we had a toad which is a level 1.
And we had a tiger which was level 12, let's say.
And we had a dragon, which is a level 50.
And for now, I'm going to put it this way, evil wizard was a 1000.
I think that's what it said.
Okay, so if we just run this and we could print out our creatures, we need to make sure that we're running the main.
Remember, up here, at the very top, we wrote this, but at the bottom we have to do our little main.
Like this, to say are we running this script as a program?
Or are we just importing it?
If this is true, we're running it as a program, then we want to invoke our main method.
So let's go over here and say run, and there's some stuff that went a little bit crazy somewhere along the way because we didn't finish it.
But here, you'll see that we've created a creature object here, and a creature object there, and so on.
Our little creature part worked, but then the code that we sort of commented out below, isn't quite done.
So this is close, but remember I said you might want to have special features.
So let's just focus on the dragon for a minute.
You might want to have special features that take into account the scaliness, the fire-breathingness, have a different mechanism for defense, and so on.
So let's say we're going to have a class called dragon.
And the dragon also is going to have a defensive roll, like this, and maybe the dragon also wants to have this information.
Woops, copy that.
So the dragon is going to go like this, and then it's going to copy it, and this seems really, really tedious.
It is and I'll fix it in just a second.
You'll see that we can model this differently.
But when we create a dragon, it will have this.
And let's also say that it has a scaliness factor and breathes fire.
Alright, so we want to store these here, so I'll say, "self dot", that equals that.
In fact, in PyCharm you can hit alt enter and it will even write that whole bit for you because it knows that this is what you should do.
But you probably see a lot of duplication here, and here.
And if I want to change something about this, like add a feature to all creatures, well I'll have to go in and add it everywhere, like this.
So, we'll be able to something a little bit better here in just a second.
But let's go ahead and write this.
Let's take that and let's say we're going to multiply that times the self.scaliness, let's go that equals that.
We'll say if self.breathes_fire, then value equals value times 2.
So it's even worse, stronger if it breathes fire.
Okay, great, so there's that.
Now, this is not really super helpful.
Any change we make over here, we would kind of like to copy them over there, and really, we would like to treat the dragon as just a specialization of a creature.
Remember, this is a relationship.
So we can model that by saying this dragon is a creature, like this, alright?
And when we say that, what it lets us do is actually take on all the attributes here.
Notice this is giving us a little warning, it says, "you need to say super dot", and in pass the name and the level.
And this basically says we're going to run this bit of code, and we don't actually need this stuff here, we're just going to store let it pass on through, then we're just going to store the things that are special about the dragon.
Similarly, this right here, we could say the role is actually whatever the regular creature does.
And then we could say, okay we're going to do that same thing, and we're going to factor in the scaliness and the firebreathing, here.
You can see right here that PyCharm is saying that you're actually getting this detail here from your creature class that you're driving from.
Okay, so this is all well and good.
Let's do one more thing.
Let's have a wizard.
Now the wizard is also going to be a creature, and the wizard actually has no special items or a passing to it, so we can just leave this init thing off.
We're just going to add one behavior to the wizard, which is going to be attack.
So he's going to have a creature he's going to attack, and it's going to return True or False.
So it'll go something like this.
Now notice, both the wizard can get this defensive role, or offensively, but it's fine, and the creature also knows how to get a defensive role.
And we'll just compare those, we'll say return, my role is greater than or equal to their role.
Alright, so what we're going to do is have the wizard attack another creature, and then if we roll something higher than we win, otherwise we lose.
And we're indicating that by returning True or False.
Alright, so we were able to take on a lot of the features and the blueprint of the creature to get this defensive role, and all we're doing is adding an attack mechanism.
Whereas the dragon, we said we're going to change the way defense works for dragons as well as, like, storing additional stuff.
So we're able to model these various actors in our game, all the while keeping a sort of common functionality so they can interact with each other.

Now that we've created our actors in the game, our creature, our dragon, and our wizard, let's go and actually use them to put the behaviors or the implementation of the game together.
So over here, let's also import wizard and dragon.
Now down here, these bats, and toads, and tigers, they're fine, but this one, we're going to create a dragon, and this will be called the black dragon or something like that.
And notice PyCharm says the black dragon actually takes, breathes fire and a scaliness.
Alright, so let's do that.
Let's say this one, its scaliness is 2.
We can make that super explicit and I'll say breathe fire, false.
Okay, so we have a chance to defeat it, but it's still, it's going to be tough.
And here we'll make this a wizard.
This is an evil wizard.
So we can have our standard concept of a creature, or we can have these specialized ones, but we're going to end up treating them all as creatures.
We don't need this.
We'll come over here, and now we're going to create our hero like this.
And his name is going to be Gandalf, and his level is going to be 75.
So, it's going to be a little tough for him to beat that, he should be able to beat this wizard, he's got no chance.
Well, he has a chance, but it's highly unlikely, okay?
Alright, so now we need to randomly choose a creature.
Now let's go up here and say, import random.
There's a really great way in Python to randomly choose a creature.
You could say, create the integer, figure out how many there were, use the index, end to this list of creatures.
Or you could just say choice, and say, give it the creatures, and given a collection here, and we'll just grab one.
Okay, so now we need to print out some information about the creature.
So we can say activecreature., now we're not getting very much help here from our IntelliSense, but that's okay.
So then we say name, activecreature.value, and that's going to print out a nice little thing.
And we should be able to actually run it now.
Let's go ahead and run it and see what happens.
If it's nice and big, not so happy.
Where did we make this mistake?
I typed value, this would be level.
Okay, a tiger of level 12 has appeared.
Okay, so run away.
The wizard runs away.
A black dragon of level 50, has appeared.
Run away.
The evil wizard, definitely ran away.
Okay?
Then we just hit Enter to exit.
So it looks like our actors are working.
All we have to do is have them battle, so if they say attack, we'll just have this.
If a hero.attack(activecreature), if that's true then they win.
So what we actually want to do is make them leave the game.
So we'll say creatures.remove(activecreature) and then we'll say print, something like, the wizard defeated them.
So we'll say just something like that, the wizard defeated such and such.
And we're all good there.
Wizards run away, there's nothing to that.
If the wizard looks around, we just want to show all the creatures.
So that's easy.
We'll just say, for seeing creatures.
Now we'll do a little format string.
Like this, and so we'll say, see.name, see.level.
And notice, all we're using are the features of the base creature.
Not the scaliness, not whether it's breathing fire, we're just talking about the standard creature, which means we can treat these all uniformly.
Alright.
That looks like we've written the game.
I think we're done.
I think the last, final thing to do is to play it, and it seems more fun to play if we play it full screen.
So, let's say this, Python 3 a lot.
Excellent, an evil wizard has appeared.
Shall we try to attack it?
I'm sure we'll lose.
Oh, we're not printing anything if we lose.
Alright, let's do that one more time.
So we'll say, if else print the wizard has been dealt a defeat.
Alright, so the wizard has been dealt a defeat.
Now the wizard doesn't actually lose.
The game's not over.
Maybe it could be, but right now it's not.
So we could attack, first let's look around.
Alright.
So we see the bat, the toad, all the things.
Now if we attack the toad, we should defeat it.
We do, luckily.
If we look around, you'll see now the bat, no, the toad is gone.
This level 12 tiger, we can attack it, and this wizard here, let's run away.
We can look around again.
Now we just have the bat, the dragon, and the wizard.
The wizard we're going to run away from.
The bat we can attack.
Now what's left?
Just the dragon and the wizard.
Alright, we're going to run away from that.
Maybe we can beat the dragon?
The wizard has been, guess I got a spelling there, the wizard has been defeat by the powerful black dragon, has been defeated.
And we probably, it would be nice to see the two roles that came back, but that's okay.
Alright, so if we look around, there's still that.
Run away, run away, run away.
Okay, maybe we beat the dragon?
Yes!
Now if we look around, all there is this, the chances that we beat this evil wizard, not so high, so we're just going to leave.
But we've defeated all the other creatures.
There you have it.
We've built a game and we've modeled it with classes and objects.

Let's quickly review the concepts around classes and remember that classes are the blueprints from which we create objects and those objects are the things that act and take on the data of our application.
So we start by using them in the class keyword and then we just make up a name, this is going to be the name of the blueprint, or the type that we create, here we called it a creature.
And then we add a dunder init, in one of these magic methods here and every method that is on a class has this self message what's called a static method or a class method and they always have self but, but we don't have to explicitly pass those, Python takes care of that for us.
If we want additional premiers, they go after self, so name and the level and we're going to assign new fields to this by saying self.name equals name and the new ones for self.level equals the level.
We also can add behaviors by adding additional functions, so get_defensive_role() for example.

I hope it was fun to watch me write this D&D game, but it's going to be way more fun for you to write one yourself.
And no, we're not going to write the same game, we're going to do something totally different and fun.
So, over here on the GitHub repo, here's the D&D game in case you want to go in and actually look the code we just wrote.
So you can use that to help you come along here as an example.
Now, let's go down a little bit here.
We're going to work on a different kind of game.
Rock, Paper, Scissors.
So, here's our wikiHow on how to play Rock, Paper, Scissors, if you've never done it.
It's a straightforward, fun little game, slightly more complicated than just guessing a number and those sorts of things.
So, it's a pretty interesting game in that sort of three option way.
And so what we're going to do, definitely in the first two days, maybe even into the third day, is we're going to build the standard Rock, Paper, Scissor.
However, if you feel like you get done early and you want something special, like a big challenge, I've also put a link here to this thing called 15 Way Rock, Paper, Scissors.
And, by the way, they even go beyond that so you can have more than just 15.
I think there's like 25 is probably the highest I've seen, but there's all sort of fun creatures and interesting things going on here.
So, let's get to the first day.
So, what you're going to do is mostly just watch the videos, learn what you're going to learn by watching them, and let's just create a project that's going to be the foundation of Rock, Paper, Scissors.
You don't really need to create a virtual environment or anything like that because there's no dependencies, there's really nothing to pip install, which is the main reason to have a virtual environment.
First day, mostly just watch the videos and create that starter project.

Second day, is we're going to write standard Rock, Paper, Scissors.
And you're going to model this with classes.
There's going to be a roll and the roll has a name.
It knows what rolls defeat it.
And, alright, so you store the name that other rolls, that the names of the rolls that you can defeat as a roll and the rolls that defeat you.
Also, we have player concept.
And the player is really just going to have a name.
So it says, player Sarah rolls this.
Player Computer rolls that, and so on.
Alright, so nothing major there.
You could also keep a history of the rolls.
You could show like, sort of replay the game if you wanted, and each player could remember what they played at each stage, that'd be fun.
And then the basic program flow looks like this.
This is not perfectly exactly what you necessarily need.
It's not totally implemented but, we're going to print out the header, we're going to initialize the game here by getting the various rolls, in this case there's only the three: Rock, Paper, Scissors.
And then we're going to get the name of the player, Add the computer and then we'll have an automatic player here, and then we're going to run this little game loop, by passing them off.
And the game loopers need to go round-and-round, until somebody has won.
We'll go around three times, and basically just have the computer randomly roll.
And then have the real player ask them what they want to roll, Rock, Paper, Scissors, and then have them do that roll correctly.
And then finally, you can just sort of do this comparison.
Does the one roll defeat the other?
I don't know, right?
And then just do a little output here, and increment the count so that you'll know, like you just played best of three.
So you'll know who won, figure out who won and then print out so-and-so won 2 to 1 or 3 to 0 or something like that.
If you get a tie somewhere in the middle, this doesn't work so, you know, maybe once you get it working, not considering ties, then come back and address the possibility there might be ties.

Alright third day, if you're not done with the first two days, just finish that up.
Just get your standard 3-way Rock, Paper, Scissors working.
However, if you feel like you want to like, take this to the next level and you got done really quickly, if you've got some extra time left over, try this 15-way Rock, Paper, Scissors.
This diagram is actually really hard to understand, so I put together a battle CSV here that tells you if the attacker is a gun and the attacker attacks a dragon, will a gun defeat a dragon?
Or does the dragon, over here dragon defeat a gun, note the dragon loses to the gun, but the gun defeats the dragon.
So you can think of this like, sort of halfway redundant.
You really only need half this table, but having that table is super helpful.
And here's a little bit of code, we haven't gotten to CSVs yet, but here's a little bit of code that will read that in and you can use it to sort of parse that and probably build from.
Okay, so if you're feeling super adventurous and you've got extra time, work on this Rock, Paper, Scissors 15-way, otherwise just build standard 3-way rock-paper-scissor and hope you have a lot of fun modeling these little games with classes.

Welcome back to the 100 days of Python.
In the coming three days I will guide you through list comprehensions and generators two of my favorite features of the language.
We're going to look at how you can make a for loop with an embedded if statement more Pythonic by using list comprehensions, then we will load in Harry Potter and use list comprehensions to filter out non valid and stop words.
Next up, generators.
We start very simple with the concept of the yield statement writing a simple generator, we look at the stop iteration exception, and how the for loop catches that for you.
We move on to more interesting examples and finally compare performance of lists and generators, because if your data set grows you definitely want to know about generators.
The second day I have practical exercises to train your new gained list comprehension and generator skills.
The third day I will show you solutions to those exercises and I challenge you to take a more advanced generator exercise replicating Unix pipelines and it will be a lot of fun.
So, lets dive straight into those two Python power tools.

List comprehensions and generators.
Let's import the modules we are going to use.
Let's start making a list of names.
We've got a bunch of names and let's loop over the names.
for name in names and we're going to title case each name.
There you go.
Then let's do something more interesting involving an if statement.
So, let's keep the names that start with the first half of the alphabet.
An easy way to do that is to use the strings module, which has helpers like ascii.lowercase.
So here, I used the strings ascii.lowercase, I converted it into a list, and took a slice of the first 13 elements.
Great, and the purpose, by the way, of this exercise is to first do a classic for loop and if statement to later refactor that into a list comprehension.
Right, so, Mike, Bob, Julian, Guitto, but this seems a bit for both, right?
We looked through the names, we do an if statement, and it takes like 5 lines of code.
Before we move on, I have to warn you though, if you see the elegance of list comprehensions, there is no way back and you want to use them everywhere, and that's awesome because it reads like English and I don't know any reason why not to use them.
Let's write a simple list comprehension to do the same as I did here.
The very basic level of this comprehension uses for inside the square brackets, so for name in names.
And before the for, just returns the name.
So, this would just bounce the same list we had before and the nice thing then, is that you can add an if statement after the list.
So here, first character is in first half of the alphabet, that's got to stay and a result, I want title cased, so I can do that here, and now we get the same result.
So, if I call this new names2, I can say new_names asserted, new_names equals new_names2, and they're exactly the same thing.
So look at that, five lines of code, one line of code, and they read pretty well.
You just have to read from the inside out.
Have a loop over the names.
For every loop, I check this if statement and I return the name title case, if that if statement is true.
That's all there is to the basics of list comprehensions.
You can nest them, but they might become unreadable, so I would definitely stay at this level.
The other way to write this is to use map and filter, like the functional programming construction Python, and those work equally as well.
Although, I find this more readable, this more like English.
So, let's move on to another example.

Let's do a more interesting example.
I'm going to load in the text of Harry Potter, split it into words, and use list comprehensions to filter out stop words, or other words that are not meaningful.
So let's load in Harry Potter, and parse the response, which is response.text, I lowercase it, and I split it into a list of words.
And you can see that by just getting a slice.
Cool.
And let's see the most common words so far.
Right, well here are stop words we're not really interested in, and the dataset also has a couple of other characters, that should not really be taken into account, for example, do we have a dash in words?
Right, so we need to filter that out as well.
So let's clean out any non-alphabetic characters first.
So this looks over the words, and any word that contains one or more non-alphabetic, or alphanumeric even, characters gets stripped out, and I do realize that that might lead to empty words in the result list, but next we will have another list comprehension that takes care of that.
So is the dash gone?
And yes its gone, but we still have stop words, for example "the", which we're not really interested in.
So let's do another list comprehension to filter those out, but for that I need a list of stop words.
I already prepared it, and the code is the same as loading in Harry, I'm just going to copy/paste that.
And here you have a list of all the stop words.
Let's wipe those stop words out of the words list so far.
So words equals word for word in words.
If word strip, and that's what I said before.
There might be some empty strings in there, and by checking if word strip is true, you're basically saying, discard any empty strings.
So if you have a non empty string, and the word is not in stop words, then it's a go.
So we need non empty words, and a word that's not a stop word.
If so, store that into the new list.
And then we can do a simple check.
If "the" is still in words, and now it's gone.
Now let's do the counter again, and see if we have a more relevant result.
And there you go, there's the Dumbledore.
I have to confess I didn't read Harry Potter, but this sounds more like Harry Potter.
So, I think this was a great example to show you how you can use list comprehension to clean up data for analysis using few lines of code.

Next up are generators, sometimes building up a big list hits your performance right?
It doesn't fit into memory and you can write a generator that yields values one by one.
Its like a function that pauses itself.
You call it you get one value, it pauses, you call it again, it gets you another value and it keeps your memory footprint small.
Its best to write the simplest of generators next, lets do a number generator, so def num_10 for e in range 5, and then we use the yield keyword and that's it, that's like the smallest easiest generator is, that's stored in a variable, and that's it.
Now you can get the next value from the generator using the next keyword and that's zero and you can loop through them, like this and notice that the for loop took of at one because zero was already used or returned, and another important thing to know about generators is that they consume their sequence once and once you get to the end and try to go beyond that limit you get a StopIteration.
If I now do next gen, boom it doesn't work, it says StopIteration because we've exhausted the sequence right?
And again, for handles this for us, so if I initiate this again, and do again the for loop, we don't get this exception because for is smart enough to catch this for us.
So, that's the simplest generator example I could come up with.

A common use case I find for generators is to build up my sequence.
So, let's define a list of options.
Red, yellow, blue, white, black, green, purple.
And in my older code, I will do something like, And that's fine, we just keep an internal list, and append to it and return it.
Just to show you how you can do this in a more concise way with a generator.
I'm just calling it the same name, but appending _gen.
It's the same for loop, but instead of building up a new list, I'm going to use the yield keyword to yield the values one by one.
Alright, let's see what that gives us, a generator.
And a way to materialize the generator at one go is to convert it into a list, and there you go.
So this is a shorter, more concise way to build up a list or sequence, and it's also faster if your data set grows, because it's evaluated lazily.
And actually to show that in practice, in the next section, I will compare a list and a generator in performance.

I've said it a couple of times now that generators can gain you performance when your data set grows.
So why not see that in action, and define a million years, and loop over them and see which years are leap years.
So let me write it out and I will explain it next.
Okay so, first I have a leap years list that builds up the list of a million years, checking the isleap(), and I'm using calendar.isleap(), which is a nice built in way to do that.
And the second function uses a generator, so it's the same loop, but it yields the year.
So it's not building up the whole list in one go.
So let's use the timeit module tool, time both functions.
And it's taking a bit.
Let's do the same for the generator.
Wow, look at that, that's milliseconds versus nanoseconds.
So the generator is way faster.
And again, that's because it's not taking up so much memory.
It's yielding the years one by one, doing that lazily and saving you memory.
So that's why generator's faster.
And when you're working with large data sets, you should definitely know about them.
And that's a wrap of day one of the list comprehension generators lesson.

Let's look at what we've learned so far.
List comprehensions.
For both ways of doing a loop and conditional, we loop over list and store all the modifications in a new list.
Five lines of code.
The more Pythonic ways to use a list comprehension.
One line of code.
And it reads like English.
We went through another sample cleaning up a word list, and you can do multiple checks.
in the conditional part over list comprehension.
Secondly, generators.
The simplest generator would be something like this.
For in range, yield the value.
Generators, pause.
So, after every call it stops at the yield, and comes back.
Use a generator to build up a sequence.
Here I made a bunch of options for a fictional website.
And instead of building up a list in the function, we use the yield statement to just generate a sequence of items.
And lastly, we look at list and generators, and we saw that when your data set grows your really want to know about generators because the items are lazily loaded, not taking up the whole memory footprint.
And that's it for the basics.
And now it's your turn for day two and three to get more practical exercise.

Welcome back to 100 Days of Python.
The second day of list comprehensions and generators.
Now that we've got some theory down, it's all about getting practice.
I've got some small exercises to get practice.
So, here you're provided with a names list of names and surnames.
And can you write a simple list comprehension to convert those names to title case, and reverse the first and the last name?
Then we use that data to make a simple generator that generates output like this.
So, we initialize the generator.
We look through a range of 10, and call next on the generator.
And every time we call next it returns name one teams up with name two.
And those names are randomly chosen That should not be too hard after yesterday's lesson.
So have fun, and tomorrow we'll show you the solution to these two exercises.

Welcome back.
I hope yesterday's exercise was reasonable for you but starting today I will show you a possible solution.
If it was very easy for you, feel free to skip to the next video where I have some other exercises lined up for you.
Okay, so the first thing we needed to do was to title case the names using a list comprehension.
That should be pretty easy now.
So, name title for name in names.
Oops and names is not defined because I did not run the cell and let's run it again.
Okay, cool.
So, every name is title cased.
And then we have to write a list comprehension, reverse the first and the last name using a helper function.
So, let's define reverse the first, last names and it takes a name, split the name in first and last so this is a nice example of unpacking.
So the name splitted by defaults space, get you two elements and you can assign them directly to first and last.
Then, we return them and I was using a join but in 3.6 you can use f-strings where you can embed the variables, which is very nice.
And let's do the list comprehension to use that function.
Reverse first, last names, name for name in names.
Right.
And yeah, I dropped the title case requirement here but that worked.
Then we move on to generators and the exercise was to generate random pairs of names.
So, name one teams up with name two, etc.
First, define a function.
And, let's get the first names and we can again use a list comprehension for that.
So, we split them again and we take the first element with indexing.
We title case that.
That's nice with Python, that you can chain all these operations for name in names.
So, let's do an infinite loop.
Which I usually do with while true.
I initialize first and second.
And this little while, I'll explain in a bit was that I had to add later.
And I used a random sample to take the first names list and pick two items.
Why you needed the while?
Well, it turned out that I could have two teams of a Julian so the same name came out of random sample.
So, while that's the case, keep picking two names basically.
So that was a little tweak I had to do to make sure that both names were always different.
And then again, I used a f-string to return first, teams up with second.
And let's see if that works.
So, I assign the generator two pairs.
So for underscore in range and the underscore is just a way in Python to say throw away variable I don't care really what that loop variable is.
Print next pairs.
I can adjust to four variable in 10 pairs because that will go on infinitely.
So I'm making sure I'm making next to retrieve one value at a time.
Okay, I did not import random.
And there you go.
Jewel teams up with Julian.
Ali teams up with Bob, etc.
One final thing I wanted to show you is itertools, islice because I said before you can not just loop over an infinite generator, it will probably hang your system because it never ends but islice, you can slice a generator just as you would slice a normal list but that overcomes that problem, so I can just do, itertools.islice give it the generator and the number I want, that gives an islice object and I can materialize those in a list by doing this.
There you go.
Okay, those were two possible solutions of the small exercises I gave you yesterday and in the next video, I will show you some more exercises you can do today.

Alright, you're almost there.
To get some more practice, I put together two smaller exercises, or bites, and one bigger co-challenge.
This one, you will recognize the name's list but it's a little different because you have to take duplicate names out and sort the names and find the shortest first name and, of course, you will be using this comprehensions.
Secondly, what we did not touch upon, is dictionary comprehensions, so, you might look that up and go through bite 26, where you have to do some operations on this dictionary and this set using a dictionary comprehension.
And the co-challenge is Generators for Fun and Profit, a challenge we run some time ago.
And this will be a fun one because you have to turn this Unix pipline into multiple generators.
So, I think that's a great way to get some more practice using generators.
Three exercises, see how far you can get during this third day of this lesson and, of course, share your work.
Put a tweet out with #100DaysOfCode.
It's a great way to get visibility of your work and, of course, we'll be happy to see how you progressed this section.
Good luck.
Have fun.
Keep calm and code in Python.

Good day guys, this is Julian Sequeira, and welcome to iteration with itertools.
This is going to be a three day series just touching on the more common, arguably more common, usages of itertools.
We'll start off with a bit of a coverage of what iteration is, very basic stuff, and then we'll get straight on into itertools.
So carry on, move on to the three day overview for some more detail, and then we'll get straight into the code.

The next three days are going to be pretty jam-packed with content for you guys to consume.
So, for itertools day one, what I'd like you to do is actually just watch the videos.
There are four videos to do, and they involve cycle product combinations and permutations, okay?
I'd like you to pay specific attention to cycle, I'll explain why in a second.
But, pretty much, that's all you have to do for day one.
Nice and easy.
Just grasp the concept and play in the shell.
I cannot stress that enough.
Actually do some live coding in your Python shell, okay?
Do that to really grasp the concepts of cycle product combinations and permutations.
For day two, you're going to create a traffic light script.
Okay, now what this script is going to do, it's actually going to pretty much just emulate or simulate traffic lights.
Red, amber, and green.
Okay, nice and simple, it uses cycle.
Which is why I want you to learn itertools cycle and pay attention to that one specifically.
And you're going to create it before you watch the traffic lights video, okay.
The video is there, it'll be right after combinations and permutations, but please try not to do that until you actually give this an attempt.
Okay, give this a go yourself, and then check the video to see how you went, okay?
Now for your last day, after you've finished your traffic lights, I would like you to have a play with bite 64, bite 17, or bite 65 on the code challenges platform, okay?
These are three bites that are free for you because you're in this course.
And they will actually make you use itertools in some way, shape, or form, okay.
Some of them are easy, this one's intermediate here, and this one's also intermediate, okay.
So have a good play with these three bites.
There's nothing much to it; you'll code within the browser and you'll hit test to run some tests against your code.
A really great way to spend your third day.
Nice and easy, it's all laid out in front of you, you just have to focus on the code.
So if you want to go to bite 64, 17, and 65 using these three links, you'll get them for free, and then you can spend your last day just plain coding.
And that is pretty much your three-day wrap-up.
Go through it, if you have any questions, always reach out.
But, other than that, move on to the

All right, so let's discuss what iteration is in Python.
When we say something is iterable, we're saying that you're able to iterate over it, you're able to move through it one item by item, okay?
So the easiest way to demonstrate this is just to get a simple list.
So we'll create a quick list here of numbers.
Let's go with the numbers 1 through 10.
Okay, oops, what am I doing?
1 through 10.
And we know that, and we know that number is 10 digits, right?
Nice and easy.
Now if we're iterating over that, we're going to run a full loop, so for i in numbers, in number, I should say, print i.
This is something we've all done before, we all know what this is, but what's happening behind the scenes?
Okay, yes we're running a full loop, but really what is that full loop doing to iterate over the number list and give you the numbers?
Well, it's actually calling the iter dunder method, or iter protocol, okay?
So we can see this if we actually drill in to the number list.
We can see that it is actually calling iter or it's capable of being iterated over.
Okay, so we've got iter and dur number and we get true, so it's in there.
This is an iterable item, we can iterate over it, okay?
Now another way to demonstrate an iterable item is to use next.
We've all seen next before, or hopefully you've seen next.
And what happens is when you run next on an iterator, when it finishes iterating over the sequence, over the list or the string or whatever it happens to be, when it finishes this, it then gives you an error, okay, it gives you a StopIteration error because it's only going to iterate through it up until the end and then it will stop.
So now that we know that iter is actually what's being called in the background, we can use that, okay, we can use that with next.
Now if you haven't heard of next, next is a little function you can run against an iterator.
And what it will do is it will pass over, it will iterate over that iterator, that list or that string or whatever it is, and it will continue through it until it hits the end.
When it hits the last item, when it hits the last character, it will actually give you a StopIteration error, okay?
So we'll demonstrate that with a string called, let's call it string, okay?
So it equals iter, we're actually calling iter now over the word string.
Okay, so we know that when you iterate over this word here, over this string, you're going to get the letters one by one, right?
So if we call next on that, we'll get the letter S, okay?
Now let's copy and paste this a few times, just so we can demonstrate.
We get the T, we get the R, I, N, G.
But then, when we run it one more time, we get the stop iteration, okay?
And that is because we're calling next directly.
That's because it's hit the end and it's taken care of.
It gets to the end but then it's not going to go any further because it knows it's already finished.
Now, when you run a full loop, so for character in string, print character, we get the letters, but we don't get this error, we don't get the StopIteration error, and that's because it's actually built in to the full loop so that it's not going to give you that error.
It's expected, okay, so it knows that it's hit the end and it's not going to actually sit there and give you the error.
And that's it, that's a basic coverage of iteration.
You see it's just going through each object, each item, one by one, to get to the end.
Now, there is iter tools, a nice series of functions that are just so cool and make iteration a lot more interesting and a lot easier, so that's what we're going to look at.

So we're going to kick off our foray into EdiTools by using EdiTools.cycle.
This is one of the infinite usages of EdiTools.
Okay, in the since that when you use it, it's just going to keep iterating over the item or series of items over and over again.
It doesn't stop until something tells it to stop, okay?
So, let's have a go.
First thing we're going to do is import EdiTools.
Now we're going to import, I'll explain all of this in a second, we're going to import sis and we're going to import time.
What we're going to make, and this is a really cool one, I absolutely love this one, we're going to make one of those little, cool spinny line things, you know?
Like a little loading line that you see on the command line.
You might actually like this or you might hate it depending how many times you've seen it.
So, to make that, we're going to go symbols, let's create a little item here called symbols.
And it's going to be EdiTools.cycle.
Now what we have to specify between the brackets is, tool tip gave it away, is the iterable, the item that we're going to iterate over with cycle, okay?
Now that could be a variable, could be anything really, it could be any object, but, we're going to use just a specific set in a specific order.
So if you think about this, take a look at how these are lining up.
This looks like a spinner right now.
So it's going to start with a horizontal dash, then it's going to go into the slash, then the pipe, and then the other, then the forward slash, and so on and so forth, okay?
And it's going to iterate through that.
It's going to keep cycling through this and you'll see that it actually starts to spin.
Alright, it gives you that sort of feeling of a spin.
Now, to do this, we need to put it in a loop.
So we're going to give it a while loop.
So while true, and again, dangerous, this is just going to keep going until you control c out of it all.
Snap out of it in some way, shape, or form.
To get this to work the way we want it to work, we need to use the sis module, okay?
So, I won't go into sis now because this is pretty straightforward and you may already know it and it's out of the scope, so we're going to essentially send this data to standard out, okay, on the system, wherever you're running this from.
And the little trick we want to do here is, I'll just show you, we're going, let me type it in first.
We're going to go next, symbols...
Alright so what this line is going to do for every single loop it's going to do the next.
Remember we covered next.
It's going to go through the next iteration of EdiTools.cycle, okay?
And this here, the slash r, is going to negate putting this on a new line, okay so, it's not going to return, it's not going to put it down on a new line, mkay?
Now, sis, this one here we sort of have to put in just for safekeeping.
Just in case, okay?
Oops, not flash.
Flash, so sis standard out flash, this guarantees, this will force whatever you're putting to standard out to appear on the screen because from time to time you might actually get what you're writing to standard out going into a buffer and we don't want that.
We want it to be flushed out of the buffer and onto the screen.
And then we're going to put a time delay.
That's why the imported time up above.
Let's just make it one second, okay?
So, our while loop, it's going to start iterating using next through EdiTools.cycle, okay, it's going to cycle through this infinitely.
It's going to flash it and make sure it appears on the screen and then it's going to take a second, okay it's going to sleep for one second and it's going to do that for, well, for eternity, until we exit out, okay?
So look at that, we're starting to go through here.
Just ignore the two here.
This is a by product of actually running through this in the Python shell.
So what we're actually going to do is we're going to put this into a, an actual Python file and we're going to run it from our Windows command line and you'll see how it actually works.
Okay, with some magic here we now have all of that in a little script and now all we have to do is run Python and what did we call it?
We called it cycle_demo, and look at that.
Look at this funky little spinner over there.
So, that looks kind of boring now...
Mkay, so let's go into this and change this in seconds to be zero point, five seconds.
Let's try it again.
And look it's speeding up, okay it's gettin' quicker.
It's gettin' quicker.
And just, just for the fun of it, let's make it super fast.
And look at that, now it looks like a proper spinner.
So this is a perfect, perfectly awesome and usable example of EdiTools.cycle

Let's take a quick look at itertools product.
Now product actually is short for Cartesian product or it's a Cartesian product, I suppose.
Now, what does that mean?
Well, it's not another language.
It actually means it is every possible combination of values.
Alright, so think of it this way.
Let's say you had a string.
Let's say you had my name and you wanted to see how many different possible combinations you could get of the letters.
Now you can change that slightly.
So, the first thing that comes to mind is that you might think how many combinations of six letters.
So my name, J-U-L-I-A-N.
How many combinations of those six letters can you get?
Now with itertools product, that is exactly what it calculates for you.
That's what it prints out on the screen for you.
So, let me demonstrate that for you.
Okay, so we've done from itertools import product.
For letter, and we'll use my name just because we've discussed that.
For letter in product, now in product, look at that tool tip, we choose the iterable, the item that we're iterating over and the repeat.
Now the repeat, it's actually quite easy.
It's much easier to show it to you.
But the repeat is essentially how many of those letters or those items in that iterable you want to show up as a product.
Okay, so let me show you.
So, we're going to use my name.
Julian.
And then, for the repeat, we're going to say 1, just like the tall tip.
And when we hit enter, or when we, sorry I should say when we print out the letter.
You'll see we, because we repeat it only 1, we're only saying, well we only want one group, 1 grouping, we're only going to use how many iterations of this, how many combinations of this are we getting to a maximum of 1 character or 1 object?
Okay, so you can see here Julian, the J can only show up once.
Okay, because it's only repeated once.
So, if we change this, let's just copy this back out.
And we change this to 2, we can go print, letter.
Now, watch what happens.
We get a lot more as a result.
Okay, so the first thing it does is it takes the letter J.
Well, we're repeating 2.
We're going to use a combination of 2 letters.
We want to maximum of 2, it can repeat twice, okay.
So, we're going to have J with J.
We're going to have J with U.
We're going to have J with L.
We're going to have J with I.
And so on through the list.
And once it exhausts, J being in the first position and this other combination being in the second position it then moves down to the U.
And then it repeats it all over again.
And then, L and so on.
And it keeps going.
Now one thing you'll notice, is that we're talking about the positioning here.
Okay, so U and I appear together here.
But they also appear together here.
I and U.
The difference being obviously that because they're in a different order, it counts as a different combination.
Remember, this is every possible combination.
While yes, they're still returned as U and I you're still getting a U and an I returned, because they're in a different order, because they're in a different technical combination, I suppose.
They are capable of showing up twice.
Okay, what you will also notice, is that J, J, does not show up twice.
Okay, it shows up once, because it doesn't matter.
These aren't treated as different J's.
It is a J, plain and simple.
Okay, so it shows up J, J, just once.
And that is editor's product.
It's so simple, but just imagine trying to code this yourself with a for loop and looking at if statements and everything like that.
It's disgusting, right.
So, this is the power of itertools with just two lines of code you can get every possible combination, the Cartesian product of an iterable.

Next up we're going to look at itertools combinations and permutations.
Now let's look at combinations first.
Now combinations allows you to get the possible combinations of an iterable of the certain, you know, of a certain string or a certain list, okay?
So let's just get our setup here.
From itertools import permutations, combinations, okay?
Let's say we have Mike, Bob and myself.
All right and we'll make a friends list, 'cause we're friends right?
Please say we're friends.
Mike, Bob, and Julian, and we'll split that, not splut, we'll split, okay so we have friends.
We have this list, Mike, Bob, and Julian.
Now with combinations, we can see how many combinations you'll get of the three of us, okay.
Now this will actually give us a generator, so we're going to use list, we're going to force it to be a list okay, so just bear with me here.
So print(list(combinations())), okay so this is now we're talking edit tool stuff and in the brackets we have the iterable, okay?
And the Iris pretty much what we want the combinations to include how many combinations we want.
So for example if we specified Iris two, okay it would say, okay combinations of two.
So Mike and Bob, Bob and Julian, Julian and Mike and so on, okay?
So we're going to actually choose our friends list, all right?
And then we're going to have a length of 2 okay?
Let's close all this off.
And you'll see that we get, well first of all we'll get return, it returns tuples, or tuples.
And look at the combination set that we get there.
We get Mike and Bob, get Mike and Julian, and then we get Bob and Julian.
And what is it that you've probably noticed?
There's no order, okay.
There's no, what if the order mattered?
If you were trying to return this list but you don't like Mike being first every time, what happens if you want it to return a tuple as Bob, then Mike, Julian, then Mike, Julian then Bob.
Well that's where permutations comes in.
So combination gives you just a valid combination, it doesn't care about the order, right?
Permutation will give you not only the valid combinations but also in whatever possible order they can be in, okay and that's where permutations is super powerful, okay?
So we'll do it the same thing, we'll do the exact same things, let's just, may as well copy and paste.
We'll do print(list(permutations()) works in the same way, see we got the iterable and we got the r.
We can go whoops, can't type.
We can go friends and we'll do two as well, just so we're keeping this standard.
And look at that, we now have Mike and Bob, Mike and Julian, but then we also see, Bob and Mike, so over here we same Mike and Bob, and over here now we see Bob and Mike.
Then there's Bob and Julian, then there's the opposite, Julian and Mike, as opposed to Mike and Julian, and Julian and Bob instead of Bob and Julian.
And that's why permutations is awesome.
I mean they're both awesome but this is such a great way of doing it, it's one line of code, it's just amazing, if again, itertools is awesome.
That is permutations and combinations

Alrighty, in this video we are going to create some traffic lights.
As discussed in the ReadMe from the three-day overview, I strongly urge you to try this yourself.
It's not too difficult, it's a nice little challenge for your second day.
This video is going to walk you through how I've created it.
If you haven't done it, if you haven't attempted it yourself, just pause it here, or hit stop and minimize.
Avert your eyes, children, and just give it a try yourself.
This is the best way to learn.
You're going to need itertools cycle, I'll give you that tip, and that's it.
Out of all the stuff we've covered so far, itertools cycle is all you're going to need for this.
Then, just think about how traffic lights work and go from there.
Now, into the code.
We're going to just import some modules here.
For me, I'll discuss this in a minute, we're going to import from time import sleep because we do want our traffic lights to sleep when you're going between the colors.
Now let's import itertools and let's import random because I have an idea.
First thing we're going to need is we're going to need our colors, aren't we?
What are the three colors on a traffic light?
We've got red, green, amber, or yellow, whatever you want to call it.
You know what I mean, I can't type this bit, can I?
And that gets our colors list.
Now, the rotation between those colors.
Again, if you haven't done this yet, hit pause now.
The rotation of going through those lights, we're going to use cycle, remember the spinny thing from the other video.
We're going to use itertools.cycle, but this time we're going to call colors, just like we have here.
We're calling this and we know it's now going to cycle through red, green, and amber because by using split we created a list of these three strings.
So that's what rotation is.
Alrighty, so let's create our function.
You know what, before we start any of that, let's throw this in so we know where we're starting.
What do we want our function to be called?
Let's call it light rotation.
We're going to pass in rotation.
Not that, well, I suppose, we have to, but we'll just do that anyway.
We'll go def light(rotation) We're reading in the rotation.
So, what's this app going to do?
What's this little traffic light going to do?
It's pretty much going to have three if statements.
It's going to be, what do we see when it's amber?
What do we see when it's red?
And what do we see when it's green?
So, we'll create a for loop for color in rotation.
Now, remember, that's pretty much, we're not saying this here, we're saying just for the item in rotation, and rotation is returning each one of these, for the item in rotation, for the color in the rotation.
Let's just make it really simple.
If color equals amber, this is a direct match now, we're asking for a direct match.
At this point, we have to know it's going to say exactly that.
Let's go print.
Caution.
The light is...
Where's percent?
Okay.
And then we just close it off with color.
This is now going to print, if the color is amber, caution, the light is amber.
Then, at this point, we want it to sleep because, if you think about it, when a traffic light goes yellow, or amber, it doesn't just go to red straight away, does it?
It takes a few seconds, so let's just put in a sleep of 3 seconds.
That's it for that.
Just because we're only dealing with 3 colors here, let's just go with an elif statement.
If the color is, now, red, or, elif color is red.
We're going to go print, stop, the light is...
Oops.
S and then we go color.
Now we can do a sleep of, let's go 2 seconds.
And then we can go else because in any other case it's going to be green, isn't it?
Print, go.
The light is...
Oops.
Color.
And we'll do another sleep of 3 seconds.
Believe it or not, that's our app.
This function here is going to go through editor's dot cycle of colors, red, green, and amber, and it's going to say, well, if amber, do that.
If red, do that.
And then, if all else fails, it's going to be green, it's going to be the other option, and we're going to get green.
Let's run that.
Save it, F5 it.
Stop, the light is red.
Go, the light is green.
Caution, the light is amber.
Now, the catch here is that it's going to actually be the same few seconds every single time, so we can anticipate that.
That's not how traffic lights work.
So to make it a little more interesting, a little more complicated, let's get rid of these static sleep seconds.
We know with yellow lights, those are always going to be 3 seconds, or 5 seconds, because it's the standard every time.
This is why I imported random at the start.
Let's make it a little randomly generated timer.
We're going to go return, random.randint.
And what are we going to put the...
See, here we go, return a random integer in the range.
So what do we want the range to be?
Let's go 3 to 7 seconds.
So it's going to return those integers, anywhere between three and seven, so we should just call this in sleep right there and right there.
What that will do is it's going to generate a random number between 7 and 7 every time, and this should change.
You can't predict what that's going to be.
Let's run it.
The light is red, okay.
We have no idea how long it's going to take.
There we go, so it actually took longer than before, same with the light being green.
That was very quick, that was only about 3 seconds.
Amber, default, 3 seconds.
Then we're back on red, and it's sitting there for quite a while, so there you go.
Now this is more of an accurate street traffic light, where it's a bit more randomly generated.
But all of this, due to the awesomeness of itertools.cycle.
There's your traffic light.
Hopefully, your code looked something like that.
Hopefully it was a little cleaner.
But other than that I think we've just made an awesome little nifty app and a little tool, something that you'll probably never use, but that's itertools.cycle.

Well, I hope you really enjoyed itertools, because it's really one of our favorite modules.
It's just so much fun to use, and saves you so much time.
So, without further ado, what did we cover?
The first thing we covered was itertools.cycle.
Okay, so itertools.cycle, we just imported cycle from itertools.
Go figure.
Then, we specified the iterable.
Okay, the iterable that we wanted cycle to go over, okay?
and for our little exercise there, it was those weird dashes, so that we can make a little scroller.
If you were to use a string, such as the word string or my name, Julian, it would then cycle through those letters in the same way that it's cycling through those.
Okay?
And that's why cycle is so easy to use.
It does exactly what its name implies.
It's awesome.
Then, what we did was we threw it in a while loop, while True, so that it was infinite, it would just keep running while this app was live.
And then we pumped that..
Next, we pumped that cycle out to standard out.
That way it would work well on the command line.
Okay?
Using Next, we were able to actually go through one iterable at a time.
One iteration at a time.
Okay, so we know each iteration, each cycle is going to be one of these characters here.
So, Next got us to pull just that one in the first loop.
Then, in the Next loop, it then took the backslash, and then the pipe, and then the forward slash.
Okay?
That's how this works here.
And then we just threw in a little time.sleep, just to slow things down a bit.
Next we have itertools.product.
So again, we imported product, okay?
And then what we did was we used repeat, okay?
We used the repeat inside product to say how many combinations we wanted.
How many times we wanted a single iterable to be repeated or any iteration to be repeated.
Okay?
So by specifying two, when we iterate over my name, we were able to start getting a giant list just like this.
Okay?
So J matched with J, J and then U, and so on and so forth.
Okay?
And that's what product is, it's a Cartesian product, it gives you every possible combination.
Okay?
It doesn't matter in this instance with the doubles, where you've got J and J.
There's no sort of behind the scenes differing index there to say, "Hey this is one J, and this is another one.
"So there needs to be double." No.
It's just a J, okay?
Then we moved on to combinations and permutations, one of my favorites.
Imported, nice and easy.
And then we created a quick list of super friends, Mike, Bob, and Julian.
And this was for us to then use as an example.
So we got the combinations, okay?
In sets of two...
Of Mike, Bob, and Julian.
And you can see we have Mike Bob, Mike Julian, and Bob and Julian.
And then that's when we pointed out that, well, there's no order here.
It takes it into account that, okay Mike and Bob both exist in this one tuple and therefore that's criteria met, that's a combination.
But, if we wanted the order to change, if we wanted to take that into account, that's where permutations comes into it.
We have Mike and Bob here, but then we also have Bob and Mike over here.
So the order actually makes a difference.
And that's why permutations is just as awesome as combinations.
And that was it.
So your turn.
If you haven't yet, I would strongly recommend, go and do the traffic lights, okay?
Try not to watch the video until you've actually done it.
But at this point, you've probably already completed that.
So that was for day two.
So we're looking at the third day of this lesson set.
For this, go back to the three day overview, if you've forgotten what it is that we were talking about, but essentially there are a few bites to go to the Code Challenges Platform, and you have free access to those and all three of them have to do with itertools, and they're actually quite fun.
So if you haven't done that, go and give that a crack for day three.
Enjoy.
Keep calm and code.

Welcome back to the 100 days of Python.
Today we look at an important concept which are decorators.
It might be daunting at first, but they're not that hard to grasp, and they take your Python knowledge to the next level.
First we write a simple decorator and see how we can decorate a function to add additional behavior.
Then we make a quick detour to explain the different ways functions and Python can receive different kind of arguments.
Then we write a second decorator and see how they can be stacked up.
Then we look at some further references, and for the second and third day, I got a couple of practical exercises to hone your newly gained decorator skills.
All right, let's do this.

All right, a quick primer on decorators.
What is a decorator?
I think the best way to explain it is that a decorator can add behavior to a function, so you pass the function into a decorator, it can do something before and or after and returns the newly decorated object.
And it's just one of those common design patterns as described in design patterns the elements of reusable object oriented software.
So let's import the modules we're going to use.
And let's define our first decorator.
Just a very basic one to show the syntax.
Alright, now you can use the mydecorator to decoratate a function, and this is the syntax for that.
I will go into some of the details in a bit before, example, why should you use wraps which is not required.
And the whole aspect of args and keyword args.
For now, at the very basic level, just remember, a decorator takes you function, needs an inner function to pass in the arguments and keyword arguments and calls the function and see this sandwich effect here, so you can do something before calling the function and after it, so for example, when you write a decorator to time your function, here you would start the timing, here you would call the function, and here you would stop the timing.
You can look at it as adding behavior before and after the function.
The function gets called, but additional logic is added around it and that's what a decorator is for.
And then here is the syntax how to use it.
So right before the function you use the at sign, @decorator.
If you have been using any web framework like Flask for Django, you're familiar with this syntax.
As we will see towards the end, there is a login required decorator for example in Django that uses this concept of adding a behavior which is that case is to see if the user is logged in and it adds that behavior to a function which is that case is usually the logic of a web page.
A route to a certain web page.
And keep in mind that this is just syntax, the same thing could be written as...
So here you see actually that myfunction gets passed into the decorator, but this is the common way how you would use a decorator.

Alright, one thing we saw in the decorator was the args and keyword args being passed in and if you're new to Python you might not be 100% aware of all the different options you have to call a function.
So there's this great guide, The Hitchhiker's Guide to Python and it explains very well that you have positional, keyword, and two arbitrary kind of arguments, one is a list called *args and the other is a keyword argument dictionary, which is **kwargs and I just wanted to show a quick example how this all works, so let's define a get profile function and the only thing it's going to do is to bounce the different kind of arguments first let me just call it without anything and it should error because name is a positional argument which is required.
You see that doesn't work and the error is pretty self explanatory so let me then call it with a positional argument and that worked, so the second type is keyword argument which is not required and at set here to the default value you can also set it to False, of course saying active equals False, and that works and then lets look at those two arbitrary kind of arguments which is list and dictionary, so first lets first do the arbitrary argument list so that allows me to, for example, here we do sports define one or more sports.
And I really like basketball.
Oops, yeah this is kind of strange that, well it says positional argument follows keyword argument so if you want to do it this way you have make sure this not to be a keyword argument so now it works.
The reason is that the fourth type of arguments is an arbitrary keyword dictionary which goes towards the end.
So let's pass in some words, and they go last, right?
So we here have the dictionary of Pythonista and Topcoder, and that was the reason we got the error before because the keyword arguments always should go last, and to show arguments in action I define this show args decorator that prints the args before calling the function and prints the keyword arguments after calling the function, so just to put it into the context of the decorators we are dealing with here.
So we have show args and I'm going to redefine the function, and this time not to confuse it with the behavior of the decorator I'm just going to print something else.
Hi from the get profile function okay now lets see what happens if I call and get profile again, now that it is decorated of course I need to run the cell.
And here you see, so in the decorator it first brings the args, then it does the actual function work which is printing 'hi' from the get profile function and then we're at the after stage of the decorator printing the keyword arguments.
So here we see that the *args contains the position arguments, keyword arguments, and the arbitrary argument list, and the **kwargs contains then the arbitrary keyword argument dictionary, so here you see all those arguments actually being passed into the decorator and hopefully now you have a better understanding what *args and **kwargs means and the different kind of ways we can call function in Python

Let's do a more realistic and interesting example, let's write a timeit decorator.
And as we said before, we call the decorated function, and we add some behavior before and after calling it.
Alright, we got that defined, so we have a decorator called timeit.
It receives a function, it wraps the function, and we will see in a bit why that is.
We pass it the args and the keyword args.
We start a timer, we call the function, we end the timer, and then we print how much time that function took.
We return the wrapper, and that's it.
Let's then define a function that we can use this decorator on.
So that in itself is not decorated yet, so let's define it again using the decorator.
And look at that, how cool is that?
So the decorator started the timer, it ran the function, it measured the time, and after the function was complete, it reported back how long it took.
So it took two seconds which of course is not a surprise.
A final note about wraps, so what if I wouldn't have done wraps function?
So let's take that temporarily out.
And let's inspect that function.
Hey, where is my doc string?
It disappeared.
Not good, let's put it back.
Let's define the function again using the decorator.
And there you go, so that's why you should always use wraps from the functools module, to preserve your doc strings.

Next up, let's talk about how you can stack decorators.
And I've found an image of Russian dolls, which might clarify how this "nesting" works.
So let's define another decorator that shows the args and keyword args being passed in, and we're going to stack that together with the timeit decorator.
Alright, let's modify generate report from the last video to take some arguments.
And now, let me show you how you can stack the two decorators we've defined so far, and note that the order matters.
So I put timeit as the outer decorator, because I want to time the whole operation, including the use of the print args decorator.
Let's now call it, but first let's give it some parameters, so let me quickly find a dictionary of some keyword arguments.
And now, all should come together, because when I call generate report with some args and some keyword args, look at that.
We see two decorators in action.
First a timer, starting the timer, doing stuff, ending the timer, and printing how long it took, and then we see the inner decorator, print args, printing the args and the keyword args.
And here you see that decorators can become pretty powerful, because each decorator is doing a specific task, which can be applied to multiple functions, yet it's all abstracted in their definition.
And, yes, this is how you can stack decorators.

Okay, that concludes the basic coverage of decorators in Python.
In this section I provide you some more pointers to study some more decorators today.
I did an article on Twilio, and in this app I used a login required decorator to check if the user is logged in.
And you can see that in the code.
This is what you already saw in this lesson.
The wraps, and the wrapper.
It takes the arguments and just checks if login is in the session.
If so, return to function.
If not, do a flash message and redirect to the login page.
That's similar to what Django is doing, which, I pointed to the source.
You can check that out as well.
There's some more stuff going on so I challenge you to take a look at this code if you have time left today.
And on PyBites we did an article, "Learning Python Decorators By Example," which partly overlaps with what you have seen in the lesson so far, but there are also some more examples for caching, some more decorators in the wild.
And I point you to another article.
Sometimes you need a decorator that takes arguments like speed decorator that takes seconds and it's not always straightforward how optional arguments work so I wrote an article about that.
So if you still have time left today and you want to know decorators a bit more in detail you can read this article as well.
And that concludes the lesson of Day 1.

Alright now it's time to review what we've learned, how to write a decorator.
A decorator takes a function to be decorated.
It adds behavior before, and or, after.
Then it returns the function.
Don't forget to use wraps to the preserve the doc string.
Args and keyword args.
There are various ways you can call a function in Python.
The simplest way is to use a required, positional argument.
If I leave off this argument, I get an error.
The second type, is the keyword argument which can be set to a default.
And then we have two arbitrary sequences which are the list, in this case sports, and keyword arguments which always go last.
Here is an example how you would call this with all the types of arguments.
Let's write a timeit decorator.
It takes a function, starts the timer before calling the functions, calls the function, and ends the timer, printing how long the function took to execute.
We define the decorator.
Here's how to apply it to a function.
You can stack decorators.
Note that the order matters.
As timeit is the outer decorator, that's the one that wraps at the outer level.
Some examples of common decorators.
Here are two from the Flask documentation.
One checks if a user's logged in and the other is performing caching.
Those are ideal examples of decorators because they abstract away common behavior which you want to apply to multiple functions.
Here's another example of a well known decorator called LRU Cache.
You can find those in the Flask and the Python documentation respectively.
And now it's your turn.

Welcome back to the second day of decorators.
Today, you get your hands dirty writing a decorator, and I got an exercise here that you can do on the PyBites Code Challenge Platform.
The goal is to make this work, basically.
So we have a gettext function that takes a text, and you're going to decorate it with a make_HTML that basically adds a tag.
So you can stack it to add various tags, so when I call it like this, it should output p strong, the text of the function, and closing strong, and closing p.
And that's all there is for today.
If that's easy for you, you can already try to look at Day 3.
Good luck.

Welcome back.
For this final day of the decorators lesson, I got another code challenge for you.
It's more an open challenge, which you also can do on the Code Challenge platform, and basically it's to write a decorator of your own choice.
You can just look at your code maybe, refactor things that are repetitive, but I leave you totally free to build something that's useful for your needs.
For example, when we did the #100DaysOfCode, at Day 95, we used a decorator of our own.
By the way, here you see how cool it is to keep a log of your progress, 'cause you can always go back to all the scripts you have written.
So here, Day 95, we used a decorator to cache movie results.
And here we wrote a decorator to store or cache movie results.
It's doing that in the store helper, which you can see here.
So that's an example of how we used the decorator for our own needs.
And that's what I challenge you to do by taking this challenge.
And, of course, we invite you to PR, or pull request, your work.
And there are instructions here to get set up with git and pull request your work, which you can also do here on the platform.
By the way, don't forget to share your awesome scripts on Twitter using #100DaysOfCode and feel free to include us.
Tag Python and PyBites.
We are really happy to see what you all come up with.
So enjoy, and learn a lot about decorators.

Probably time that we talked about error handling.
I'm sure that you've encountered some issues with your Python code and you may wonder what is the right way to catch these errors in Python.
Well, that's what this next three day section is all about.
Have you encountered Python's errors?
Have you seen what's called a traceback here?
This is the report from trying to run the Python program when something actually went wrong on line 21 of api.py.
Let me get a little info here, there's a type error : and this gives us a description of what that is.
The type thing on the left here, the type error is an exception type and it tells us the category of error.
On the right is the actual message of what went wrong within that category.
So None type, object is not iterable.
Turns out that in this case, the data return from the server was empty and we tried to loop over it.
That doesn't work so well.
So we're going to see how to deal with a variety of errors the proper way in Python.

Let's jump right into our demo here.
Over in the GitHub repository, we're going to start with a movie search app.
And this is called "movie search error edition." Later, we're going to actually build this app from scratch.
We're going to talk about the underlying API and all that kind of stuff.
Right now, we're just going to run it and try to solve the errors that it might encounter.
Now, there's two parts here: there's the starter, exactly where I'm starting from, and I'll leave this here, in case you want to play with it.
This one, we're going to involve into the final one.
Now, before we open this up, let's create a virtual environment, there we go, and I'm going to throw it into PyCharm, and use whatever editor you want.
This is one we're using.
This one actually depends upon a package called "requests" So, if we come over here with our virtual environment active, we can say pip install requests or you can just click this little hyperlink thing right there.
Either way, we're going to have to do that before this will run because that's how we're getting to the internet.
So, here's how this program works.
Like I said, we're going to, in a different set of three days, we're going to build this thing from scratch and really focus on the API.
We don't actually care how the API works.
All that matters is, we pass a keyword here, and it's going to go over to a service over at movie_service.talkpython.fm.
Do a search, get back some JSON data, and then return those here as results and we're going to loop over them.
Now, I've introduced some extra errors here, alright, sort of a chance of something going really, really wrong, just to give us some variety.
The most likely error you're going to run into is a network error.
Let's just see if this runs correctly.
How about "Capital?" Cool.
There.
We've gotten three movies back and see their IMBD score right there.
It looks like it's working, except for sometimes it's not.
It actually has these errors baked into it.
But the one that we're definitely going to hit as we come over here, we turn off the wifi, this won't be so good.
So, now if I try to run this, let's see what we get.
Test, maybe?
Uh-huh.
request.exceptions.connectionerror.
And what went wrong, the connection pool had some kind of problem.
Max retries exceeded with URL, caused by ...
We couldn't get to the server, right?
So we turned off the internet, it crashed.
Instead, what we'd like to have happen is our program go, hey, couldn't get to the server.
Is your wifi off?
Check your internet connection.
Are you at a coffee shop?
Then you have to maybe authenticate to the local network, where you say you agree to their terms or whatever, right?
So we want to catch these errors instead of having this nasty crash and give a helpful message to the users.

We've seen that, calling this API, may raise errors.
And notice the WiFi being off, pretty much anything we search for is going to result in an error.
So, we're going to turn the WiFi back on.
But before we do, let's convert this full on crash to something we can catch, maybe log, send a message to the user.
Or we could try again, who knows?
Here's how error handling in Python works.
What we're going to do is, there's a couple options.
We could treat this as a separate section.
Let's say, this part that actually interacts with the data does the request.
We're going to treat this as a block that should either work, or not work here.
So, we can come up here and type the word try: And indent that into the block.
Then if something might go wrong, we'll say, except: And I'll say, print Oh, that didn't work.
And we'll just print that out.
Yeah?
Now Python says this is a little aggressive, it's too broad, but, that's okay.
This is going to work for now, and then we'll refine this a little bit.
So now if I run it, oh that didn't work.
It's a big bad crash, we just say, "You know, that didn't work." We could've logged it.
We could actually loop around and try again.
All sorts of things.
But at least we're giving proper feedback to system here.
Let's turn this back on.
WiFi's back.
Let's try again.
Let's search for capital again.
Boom, it's working.
Here's how the flow goes.
We're going to run every function here.
And if it succeeds, it's just going to run from here to there and then skip over this except block.
But if at any step something goes wrong, like, suppose, say right here, this title doesn't exist, and it's going to be an error the first time through or something.
It's going to immediately stop what it's doing and jump down here.
If this throws an error, we're not even going to run this prank code.
We're just going to jump straight from here to there to deal with it.
So, we can try some stuff, didn't find anything.
Eventually, we'll run into these random errors.
That one didn't work.
So, this is great, but what didn't work?
Wouldn't it be nice to know what is going on?
It turns out there's three or four categories of errors.
One of which is the network is down.
But there's actually others.
So, we're going to adjust this to handle these particular errors.
So, here's how we can stop the errors from crashing our program.
But let's see how we can actually deal with the errors in a more fine grained way.
One message for say, network errors.
Another message for say, if you didn't input any keyword to search for.

So we've stopped the errors from crashing our program but we can't do anything meaningful, we can only say, you know, that didn't work, sorry.
Let's be a little more prescriptive here.
So up the top, we need to work with the various exception types, so we have to import some stuff.
So we'll say request.exceptions down here.
Now in this area, we want to catch the different types of problems that can occur, and the way we do that is we say except: and then we say 'different types', remember when we looked at that trace back, the first thing is the category, or the type of error, and the second part after the colon was the description, just more details.
So what we need to do is, we could have potentially multiple ones of these, we'll say this, we'll say, request.exceptions.connectionerror, okay?
We be able to just go like this, and just do a little print here and say, so maybe we'll say something like, 'couldn't find the server, check your network connection', make this really obvious like that, and print that out here.
Now, it could be if this was like a web browser they could've typed it in wrong, but we've hard-coded the URL, we know that that's correct, so if they can't connect it's not the server doesn't exist, it's really that there's some problem getting to this one known server.
So let's try this again with my internet off.
We'll come down here and search for test.
'Could not find server, check your internet connection.' Oh, how cool, that's way more helpful, we know exactly what went wrong when we want to handle that problem here.
Now the network is back and we have another problem.
If we enter no search term it's going to freak out and throw what's called a ValueError, and say, 'hey that didn't work.' But all we get is, oh, that didn't work.
So we could do a little bit better here, we could say, put this core exception here, and we could put some details like so.
This is kind of going to be our fallback, we don't really know what's going on, so we're going to try this.
Let's try again.
Oh, that didn't work, 'you must specify a search term', and if you're just debugging it to try to figure out what's going on here you'd say, what is the actual thing that I'm getting here?
What is this actual error?
It's a ValueError.
Okay.
So now let's go add an except clause for that.
Now notice this is gray because we're not using it, and when we're not, we can just leave it off like, as we did with the request here.
So we don't need any details, so now, try to run this with nothing, 'error, you must specify a search term.' Now we can really tell the user what's going on.
We have a few more errors that we could potentially deal with, I wouldn't leave this here like that, and these other errors are just kind of random stuff that I threw in there to make the program crash.
One is a StopIteration, and the other is the one that you saw at the opening, which is a TypeError I believe.
So we'll just let those fall through here because there's not anything we can particularly do, they're just sort of random noise in the system to make sure you get some interesting crashes.
These have real fixes, so we have a special message for them.
The final important thing to see here in this whole try except block, is the order in which we specify the errors.
If we change this around and we put this up here at the front, PyCharm's probably going to freak out and say, "no no no, you'll never be getting to these." If we have something general, and this is a derivative of exception, this is not going to work.
It's actually just going to stop.
The way it works is, it runs, if there's an error it just looks, goes, 'is the error this type, is the error this type, is the error that type?' And, if it comes along here, this is going to catch almost everything, and so, even though we have this, you'll see if I run it and I hit enter, that didn't work.
I mean, we still did sort of print this out, but it's not letting us get to the part where we expect it.
So it's super important that this goes from most specific to most general.
So, here we have it; we've figured out what types of errors our program might throw, and then we can handle them independently based on the type.

Let's quickly review the concepts of try and except blocks in Python.
Python's primary error handling style is what's called it's easier to ask for forgiveness than permission.
As opposed to, say, C style of look before you leap.
In C you check, check, check, check, check, and then you just try to do the thing and hope it works.
Typically what happens when it doesn't work is either you get a false sort of return value there or just the program just goes away, it's really bad.
Python is a little bit safer in that it's more carefully captures up the errors and converts them to exceptions.
So if it's going to do that anyway, let's just try and make it work.
And if it doesn't work, well, we'll catch it and deal with it in that case.
So it's kind of an optimistic way of handling errors.
so what we're going to do is we're going to say try and do all the stuff, and we're going to hope that it all just works and kind of assume that it will just go through that block.
But if it doesn't, we're going to drop into one of the specific error handling sections.
Here we have two possible errors, really one that we're dealing with, and the rest is kind of a catch all.
So we're saying except connection error as CE, and then we're going to deal with that.
And in this case we might need to look inside the error to see, well, was there a DNS problem, is there a network problem, did it not respond, things like that, did we get a 500 back from the server, all kinds of things.
So this connection error, we're going to catch and deal with that and then we do this more general except block where here's something we maybe didn't think of, we're going to catch that and at least try to somewhat not crash.
As we talked about before, the order matters.
Most specific goes first, most general last.
If you get that order wrong, you'll never get to your specific errors.
So most specific, most general.
This is error handling in Python.

Now you've seen how error handling works in Python, it's time to put it into action for your code.
Before we get started, I just want to make the point that one of the key differentiators of professional programs and applications as opposed to simple scripts that people just throw together, or code that beginners write, really often has to do with the error handling and ability to continue working when something goes wrong.
This error handling and exception processing in Python that we just covered that is really central to it.
Professional apps still crash of course, we still run into problems, but they do so much less often and when they do we typically have logging and real-time error monitoring with things like Rollbar to let us know and so when they happen we get lots of details and we go back and fix them.
That sort of hardens our app over time.
Right, so your goal is to add this error handling to one of the applications.
So you've already watched the video so that's great.
The thing you're going to do for Day 1 is you're going to go and look through the applications you've already created as part of this #100DaysOfCode challenge.
Or if you want to pick something else that you've maybe written outside of this course you're also welcome to pick that, and we're going to take that application and improve it.
So that's today, you've already done most of the learning and then just pick the app you're going to work on for the next two days.

Day 2, your goal is to discover all the error conditions that you might need to catch, and actually determine the exception type that that results in, in Python.
Are you working with something that talks to a database?
What kind of errors could you get from the database?
Are you talking to something that goes across the network?
What type of errors come across, from say, the network being down or DNS not working, or the network being on, but not being able to reach the host, all those sorts of things.
So come up with that list, and figure out what type, what actual exception type in Python does it surface as.
All right, if it's a connection error it could be something built into the standard library, or it could be something in say, requests, as we saw in our example.
So, you're going to have know exactly what those types are so that you can actually write the probably error handling code.
That's it for today, it might be a little bit tricky to get your app into all the different situations that it's going to encounter, all right?
Some of these errors are hard to trigger, but do your best to figure out all the various error cases you're going to run into.

Day 3, it's time to take those errors that you've discovered and put in specific error handling for each one of them.
So here I've written out a little try-except block that shows the standard error handling in Python, you can use this as a template, take the various types of errors you found on Day 2, figure out how you might either deal with them or at least let the user know, keep your application running.
Add the error handling to your code, make sure that it actually does handle the errors, you know, get the thing to fail in whatever ways you were doing before to find the errors, but now you should have some kind of nice response that's not a full-on crash, maybe it even keeps running and just asks the question again, something like that.
Now that you know how to do error handling, you have some practice with it, when you're writing your applications, be sure to think of the errors that can happen, put the error handling in place, and fail gracefully.

Welcome back to 100 Days of Python.
In the coming three days I will be your guide explaining regular expressions in Python.
First we will look at when not to use them when we can use simple string methods.
Then we will dive into search and match, capturing strings.
Final, compiling your regexes, and advanced string replacements.
Day 2 I will give you some more material to read and practice online, and the third day I have some really good exercises to get your hands dirty with regular expressions.

Let's dive straight into a notebook I've prepared for this class.
It's sometimes said that I have a problem, I use a regular expression knife too, and in a sense that's true, that they're intimidating when you start.
But there are only a few rules so get some practice and you will see that they're not that difficult.
Let's import the re module.
First of all, there are cases that you don't want to use a regex.
The standard string methods are pretty powerful and cover some of your needs already.
For example, I have a text.
'Awesome, I'm doing the #100DaysOfCode challenge' And we want to see if that string starts with 'Awesome' so no regular expression needed for that you can just do text starts with Awesome and True.
Or does it end with 'challenge'?
It does.
Or does the case insensitive version of text has '100DaysOfCode' in it?
Now for that you'd first want to lowercase the string and then you want to see if 100DaysOfCode is in that string.
And it is.
And what about replacing?
So I am bold and I'm taking 200 days of code.
I don't recommend that by the way.
Well you can just do a text, replace 100 text strings by 200 and awesome, I'm doing the 200 days of code.
So for these kind of string operations, you don't really need a regex.
So look at the string operations that are available in Python and use those instead.

Now those string operations were pretty basic, but usually we need something more advanced.
Meet, regex.
The re module has two main methods.
search and match.
match, matches from start to end.
Search can match a substring.
It's best to use an example.
I'm using raw strings by the way, because then I can just use special characters with a single backslash and not having to escape them which makes my regexes more readable.
So again, we have the same awesome I'm doing a 100 days of code challenge and let's...
do a re.search first.
So I'm going to...
match a...
part of that string and you can do this as well with just I am in but the point is to just show how you would make a regular expression and what a match object would look like.
To contrast that with match, this won't work.
Oops.
Match takes two arguments.
So this is None because match ends end to end so I am is not the full string.
So to do a proper match we would be doing, start with awesome, end with challenge.
On text and that works.
And here you see the first part of a pattern which is '.' which matches any character, zero or more of them up until challenge.

A common task is to capture strings using a regex.
Here we have two strings, 100 and 200.
What if we want to extract number days of code out of these strings.
Here's how you would do it.
First, I do a research.
And I use the capturing parentheses.
What this will do, any regex inside these parentheses that matches the string will be stored in a match object, which we can access with groups.
Hashtag, one or more digits, days of code.
As it is searched, re is happy to just match the substrings.
So I don't need make sure that the whole string matches.
If this would be match.
Let's do it actually.
I would have to account for anything that comes before, and anything that comes after.
Of course I need to give it a string.
And let's see what happens.
So first of all we have a match object, and to get the actual matching string I can do groups.
And it gives me a tuple of the matches.
So to get the actual string, I can just use indexing.
And I got 100 days of code.
Now this will work the same for 200.
Let me show search, that was my initial intent.
Search, then I don't have to account for end time, so I those wild cards out.
I'm going to use 200 to show the match object first.
And again, 200 days of code.
So you see the power of regular expressions, this is still very simple.
I can just say one or more digits, followed by a string, and it will match both 100 and 200 days of code.
So that's how you capture strings with the re module.

And now my favorite method of the re module: findall.
findall is useful to match a pattern in a string, and to get all the occurrences of that pattern.
In 100 Days of Code, we wrote a script module in that, which returned three columns: a module, if it was standard lib, and the days that we used it.
As you see in the re module, we used quite a lot.
And it will give a link to the actual scripts at the end of this lesson.
Let's write a regular expression to extract all the days, and findall really shines in this kind of task.
So we do re.findall ...
raw string.
One or more digits, and we have to specify the string as the second argument.
And look at that.
One simple statement and we got all the days.
That's awesome.
Let's do a second example.
Here is some text, and let's extract the words with a regular expression first.
re.findall, if I could type.
raw string.
One or more characters.
Text, and I first need to load that in.
Bang.
Now you can do this, also with text split.
I'm going to make it just the first five one.
So you don't really need a regular expression to split a text string into a list.
Let's say we want to find out the most common words, but only the ones that start with an uppercase character.
So, then you can use a regular expression like, and I'm using character classes which are in square brackets, so let's define an uppercase and then we have one or more lowercase characters or digits, and the plus is one or more, if you want zero more you do an asterisk.
And we want to do that on the text.
And here we have all the words starting with an uppercase.
Now just for the fun of it, let's wrap that in a counter to get the most common words.
So we are going to use from collections import counter, and don't worry I will cover counter more in detail in the collections lesson.
Counter receives a list, so re.findall returns a list as we saw earlier.
So we can just make a counter object, passing that into the counter, and as you see we get some counts here.
And to find out the most common words, we can then do the most common method on that counter object.
And lorem and ipsum are the winners.
So very powerful tool.
I really like findall.
This typical Python example, that in one line of code you can do a lot of good stuff.

Compiling regexes.
If you want to run a regex multiple times, it can be more efficient and convenient to define it in a re.compile statement.
Let's work with some data.
Here I define a list of movies and two extra things in Python.
You can define a multi-line string with triple quotes, and you can build up a list by splitting a multi-line string, or whatever string, on a space or in this case, a new line.
So this gives me a nice list of the first element, one Citizen Kane, second element, Godfather, etc.
And the task here is to identify movie titles with exactly two words.
Before moving along, maybe you want to give this a try yourself.
I hope you had fun working on this little regex problem.
And an extra concept I'm going to show you is the use of re.verbose, which allows you to wrap your regular expressions over multiple lines and add commands, which is great to teach them and makes them more readable.
So let's load in the data and let's start writing a multi-line, medium advanced, regular expression.
And as we're talking about compiling one, the syntax for that is re.compile.
I'm using a raw string, and as explained before, you can make a multi-line string with triple quotes.
And I'm going to write this out because it's quite a large regular expression.
And then we come back and I explain line by line what it does.
So here you go.
To define the start of a string, we use the caret then we need to match one or more digits.
Let me scroll a little bit up to see the data, so the numbering of the movies.
Then we match a literal dot, and note that I escape the dot otherwise it would match any character.
Then we have one or more spaces.
And then I use a non capturing parentheses.
So we've seen capturing parentheses before, but if you add inside parentheses, question mark, colon, it kind of undos the capturing.
Then you can group the various things together without capturing.
And we use a character class then to include uppercase, lowercase, and single quote.
And we want one or more of them, followed by a space.
And I commanded that all at the right.
Then we do the closure of that parentheses.
Then we want exactly two of those.
So just go back to the data, and that's basically a word.
Why don't I do just backslash w?
Turns out that was my first approach, but, and that's the funny thing with parsing data or strings, is that they're always these exceptions.
And singing has this apostrophe or single quote and I had to account for that, so instead of just word, I had to go with a more specific portion of the regular expression.
So two of those because we want to match the ones that have exactly two words.
Then, we do a literal open parentheses, and as with the dot, right, all these characters have a special meaning.
Dot matches all, parentheses are for capturing, so if you want a literal one, you have to escape them.
So here I'm doing the same thing as with the dot, and that's escaping the parentheses.
I want literal parentheses, because the years are in parentheses.
Then the years are four digits, and then I do a dollar which is the end of the string.
Phew, that was quite a regular expression, but the nice thing about verbose is that I could add all these commands, which made it super easy to explain it to you.
So run that cell, and it's now stored in pat.
And pat is just variable name, and now I can use that pattern to loop over the movies and match them all.
So let's do that next.
Four movie in movies.
Print movie.
Just the text.
And then I can use match on the pattern.
So before we did re.match, but now we can do pattern.match.
And I'm interested in match because I want to match the string from beginning to end.
Put in a movie and there you go.
So let's check if this regular expression is actually correct.
Citizen Kane, two words.
Match.
The Godfather.
Match.
Casablanca, one word.
Not a match.
And Schindler's List, this was another tricky one.
In the first iteration, I did not match this because again, I had to account for that single quote which I told you before.
So this one is actually a match because I consider Schindler's as one word.
Vertigo's not and The Wizard of Oz, four words, is not.
So how cool is that?
Let's move on to advanced string replacing.

Welcome back to the last section on the regex lesson.
This video will show you advanced string replacing using re.sub.
For example, we have a string here on doing the 100 days of code, 200 days of Django, and, of course, 365 days of PyBites.
So you're doing a simple string replacement.
It's a bit ugly, right.
So this will work but.
When you start to do things multiple times, you have to think about a better way to do it.
So let's use re.sub to make a more flexible replacement regex.
So re.sub...
raw string and I want one or more digits and I just want to replace those by 100.
And I do that on the text and this works as well and it's more flexible because if there are will be like five or six of these hashtags with different integers, they all would work.
Now let's look at a more advanced example where we also want to capture the thing we replaced.
Say, for the same string, I want to replace the thing after the days off.
So all should be Python, but we want to respect the integer so I want to have 100 days of Python, 200 days of Python, and 365 days of Python.
Doesn't really make sense in a sentence but it does for our exercise.
And the way to do that is to write a re.sub...
raw string.
And let's define the regular expression as a literal hash.
One or more digits.
Hard-code days of one or more alphanumeric characters.
Here, you see these capturing parenthesis again.
I'm going to use that in a replacement part which is the second argument and I can reference that with backslash one.
So what this does is it takes the...
match of hashtag...
digits days off and I'm going to put that in the replacement string.
And then I want to hard-code Python and I want to do this on text.
And there you go.
Awesome, I'm doing 100 days of Python, 200 days of Python, and, of course, 365 days of Python.
That's a wrap.
I hope you enjoyed this and got a better feeling of how to write your regexes in Python.
Of course it's not a complete list of all the features and that's why in day two I will provide you some more resources and things you can check out.

So, let's do a quick wrap of what we've learned so far.
When to not use regexes.
Well if simple string operations do, use those.
Re.search versus re.match, re.match matches the whole string, re.search matches part of the string.
Those are probably the main methods you will be using on the re module.
Capturing parentheses, to access part of the match use parentheses and you can use groups on the matching object to get to the string.
Your new best friend findall, one of my favorites.
To get all the occurrences of a pattern in a string, you can us findall and I showed you two examples.
One is to get all the days or numbers out of the string and it returns a list, and findall combined with counter one line of code a lot of stuff gets done.
Compile for regexes, so you can use re.compile to compile a regex to use over and over again.
And you can add a verbose switch to make regular expressions none space sensitive so you can line 'em out over multiple lines, making 'em more readable.
String replacements, resub.
Again if you can use string replace do that but sometimes the pattern you want to match is more sophisticated and you need a regular expression.
The way to use read outsub methods is to define your pattern.
Again I prefer you using a raw string to not to escape the back slashes.
And the second argument is to put in your replacement.
And here's a little bit more advanced example.
Where you use capturing parentheses to port part of the matching string to the replacement arguments.
And now it's your turn.

Welcome back to 100 days of Python.
This is the second day of the 3 day regEx blog.
And today I want to give you some pointers to get more familiar with regxxes and start to write your own.
So we did an article, 10 Tips to Get More Out of Your RegExes.
I recommend you read through it.
One thing I didn't touch on is greediness, which is important and can prevent nasty bugs.
And if you're still more a beginner in regex land, there's a nice talk at the end by Al Sweigart.
Who by the way wrote an awesome Python beginner book, called How to Automate the Boring Stuff, and it's called Yes, It's Time to Learn Regular Expressions.
And that gives you a very nice overview of regular expressions.
Secondly, there's a nice how-to on the Python documentation page.
Very refreshing how-to.
I read this when I wrote the 10 Tips post, it's pretty dense, but it is a very good primer into regular expression, and what all the specific syntaxes mean, etc.
If that's too much reading, which I can totally get, there's a nice online RegEx tester.
And you just select Python, and you can write here your regular expression, and test it in real time.
For example, I have some HTML with two paragraphs.
And let's experiment a little bit with that greediness I mentioned.
So let's match all of it.
So that's greediness because it takes everything from the starting paragraph tag to the ending one.
If I do a question mark, the match becomes shorter.
It only takes the first paragraph.
And you see all this nice feedback and explanations here.
So that's a really great tool to experiment writing regular expressions in Python, because you get instant feedback.
So that will be the third resource I have for you today.
Just spend 20 or 30 minutes experimenting with regexes, and you'll see that they become a lot easier.
Good luck.

Welcome back to 100 Days of Python.
Wow, you're almost done.
It's the third day of the regex 3 days block.
I hope you're enjoying this and getting a good grasp of writing regular expressions in Python.
So in this third day, let's get you some more practice, and I have some exercises lined up for you.
First of all, we have, on our Code Challenges platform, Bite 2, Regex Fun, where you can solve this problem of extracting course times out of a string, getting hashtags and links, and match the first paragraph.
Then we have mastering regular expressions also as a blog challenge.
And if you like to work more in your own environment, I encourage you to do this one because you get a branch on the code challenges repo and you can just work on your system.
And finally, I mean, those exercise, we think are good practice, but of course feel free to get your own data, and parse it, use regular expressions to clean the data, etc.
It's actually how we got started with code challenges.
We came up with this exercise where we saw this JavaScript course and we saw all these timings, but there was not a total so the first pilot code challenge was, go filter out these timestamps and calculate what the total course time is.
It's not necessarily curriculum stuff, but it gets you to practice.
And with practice comes mastery.
So use any data you want.
The goal is to use more regular expressions.
And don't forget to share your work on Twitter.
You can mention the handle @100DaysOfPython.
Good luck, have fun, and remember: Keep calm and code in Python.

Hey, this is Michael, and I'll be your guide for the next 3 days.
And over those 3 days, we're going to talk about recording the actions of your application, giving you insight into how your app is running, how users are using or misusing you code, and even capturing crashes that might happen.
So, if you run a website, and the user says, "Hey, your site crashed." Here's a 500 server error page from Get Up, for example.
If it crashes, what do you do?
They say, "Well, I clicked this button, "and it just crashed." Great, if you click the button and the button works, well, you're in a really tough place trying to reproduce that, unless you recorded exactly what they did and exactly what went wrong.
So we'll see with logging that that's super easy to do.
So next time they call you up and say, "Hey, something crashed," you can actually go look and see what happened, and that will give you a much better chance of, one, reproducing what they did, and two, solving the problem.
Let's take a look inside the log file for Talk Python training.
So this is somewhat condensed.
This is one log file from one day, but it's compacted so you can see the different scenarios within it.
And it's also edited to be anonymous.
We're actually keeping more information in our log files, but I don't want to share people's private information, so this is what we're going to get.
Now, if you look through here you'll see a couple of things.
There's notices, there's errors.
For example, here's a user action.
This particular user from that IP address has successfully logged in.
You can see information about their browser, and their IP address, and the time, and all that kind of stuff.
We have some more user access.
Here somebody's subscribed to get notified.
They basically gave us their email address and said, "Hey, mail me when there are new classes." Down here, our user happened to have logged out.
They were logged in, and they did some stuff, and now it looks like they're gone.
They were running Windows 10, for example, with Firefox.
That's cool, so these are the types of things you might record, but we also have other interesting stuff.
For example, why is a search engine trying to search a authenticated only lecture page?
So, for some reason, it's trying to go to this page.
Either it doesn't exist, or it's not allowed to get to it.
But, for some reason, people's trying to get here.
So maybe we should go and figure out what's going on, either make that page accessible to Google, or maybe we're linking to it in a way we probably shouldn't.
Look at this appear, though, this is a little nefarious.
We see somebody coming in and trying to go to wplogin.php.
That's WordPress log in.
This page, this site, is not written in php, it's definitely not WordPress, but here somebody's trying to get into it.
And they're telling us that the user agent is Firefox 4.0.
That's unlikely, 'cause that's so old.
Here we can see somebody's actually trying to break in.
If we look carefully, they're doing it again, and actually again.
This happens all the time.
People are just probing for known ways to get into your site, get into your database, and so on.
So knowing what is happening with your application, it's really important.
It might be obvious what the users are doing, maybe.
It's really unlikely that it was obvious that people are trying hack it, that search engines are trying to search hidden parts of it, things like that.
So having this log information is really, really important, and you'll see it's quite easy to add to our apps.

Python has built in log in and you can do log in without any external dependencies.
But I would like to introduce you to this concept to this package logbook.
Now this as you can see here replaces Python's standard library for logging and it does so in a way that makes it really easy to work with logs in our application but also super flexible.
So for example, we can come over here and import the logger and we can just say "I would just like to push all the log messages to standard out", could be a file also for example, now we'll create a logbook and then we just log.info, log.trace, log.error, we sort of categorize the response, the message we'll send that way and you get something like, well see below but also what you see in stock Python.
Why not just use the built in one?
Well, this is nice and clean and it creates a nice hierarchy, really really cool for when you use it your application.
But also it's way more flexible, look at this, how about getting a notification to your phone or pushing notifications or something like that under certain log message situations.
Really cool.
So we'll see the logbook is really powerful and it's actually created by Armin Ronacher who is the creator of Flask, one of the most popular and well liked Python web frameworks out there.

For our application, we're going to return to something you've seen on day 10 and on day 25.
Let's go over here to the GitHub repository.
We're going to go back to our Movie Search application that's going to call a web service and talk to the server; talk to the Movie Search service that we've already talked about in terms of using JSON APIs, and in terms of air handling.
There's no record of what happened with this app, so we're going to extend it further by adding logging to this simple application.
Over here on the logging section, you can see we have a starter movie search in case you want to recreate this for some reason, and then here we have what's going to be the final version.
Before I open this in PyCharm, let's create the virtual environment.
Here we are in the directory we're going to work in.
Create our little virtual environment here, and we'll be done with that.
We're just going to open this in PyCharm.
Here we are in this application.
We should have our virtual environment active; we do.
We don't have anything installed; we have a requirements file that says we have to have requests, so let's go ahead and install that.
In fact, we're going to use Logbook, so I'll go ahead and add Logbook here as well, and then we can say "pip install -r requirements".
Or, I could just click this.
Great, now we have Logbook and we have requests and their various dependencies.
Let's just go ahead and run this real quick here, so: Run the Program.
It's going to go off to the server; let's search for Capital: we found three movies there.
We could search for Action, and we're getting stuff back.
However, if we turn off our internet, we try this again with anything, Boom: "Error.
Could not find server.
Check your network connection".
Recall, over here, we added our try-except block and we have these various pieces there.
What we're going to do is take this application and record a couple of things: what people are searching for, maybe how many results were found, the time of day when that was done, and of course if there's any errors, we're going to record those errors as well.

Okay are we ready to start adding logging to our application?
There's two steps.
One, we have to globally configure how we want to log things.
Do we want our log messages to go to just standard out, so the terminal or console?
Do we want them to go to a file, if it's a file do you want that file based on a date and roll as the days change, or do you want that to just be one big file?
Or do you want to send that somewhere crazy, like, email or desktop notifications, as you saw is possible.
So we're going to configure logging and then we're just going to add the log actions as a separate way.
That actually turned out to be super easy; the only thing that's a little bit complicated is setting this up.
So let's write a function over here that will let us do that.
So we're going to pass in a filename, and it can be, let's give it a default value of None.
It can mean nothing, or we could pass in a filename.
So what we're going to do, is we're going to go over here and we're going to actually set the level first.
So here's how this works; we're going to say our current application is operating at level of notice and above.
So there's like a hierarchy of levels in the logging, there's like, trace, which is just super-verbose stuff, there's error, which you almost never want to skip, but maybe under normal operations you don't want to show the trace.
Only the notices and the warnings and the errors.
So we're going to set this here.
So we're going to use logbook, which needs to be imported at the top, which we've just had PyCharm do, and then we can come over here and let's say, let's set it to 'trace' for just a minute.
So we have it absolutely verbose, then we'll dial it back when we're done.
Alright, so we're going to choose a level, and then based on whether we have a filename or not, we're going to assume the fact that there's no filename, or one was not specified, that that just means 'send it to the console'.
But if there is a filename, we want to do that.
So we'll say this; if filename, we want to put it in the file.
So we'll say; logbook, now there's all these little handlers in here.
There's a file handler, a 'fingers crossed' handler, a Gmail handler, hashing, mixing, mail, etc.
etc.
The one that we want, is we want a filename that is based on the days.
So it has the date involved in the filename.
When the date changes, it automatically creates a new file, and goes from there.
So that's going to be real nice.
So to accomplish that, what we need to use is a time-rotating file handler.
And this takes a couple of things.
We have to obviously give it the filename, we're going to set the level, be the level, and then we want to set the date format so it knows how to roll that out, and it has a default there, and actually that default is totally fine, so we're going to leave this off but if you want to change that it's year, month, day, is how it's going to print that out.
Okay, so we'll go like this, and that's going to create the handler and then we would just say, 'push to application'.
That means every action we do with logging is going to use this underlying system here, so if that's not the case, we're going to say, 'logbook not stream handler', we'll give it a stream, which we need to import this at the top, 'standard out'.
That's just like what happens when you say 'print'.
And the level is equal to level again, and in this case we're going to push that to the application.
That's it.
Long as we call this function, things will be initialized.
But before we get on, let's make our first log message to be something that says, here's how we've configured logging.
So here's a little message that we might have, we'll say, great, the logging is initialized and the level is trace, or something like that, and the mode is either standard out mode or file mode, and we can create one of these logs and we'll have this little startup logger, and then we can just say 'logger.' let's say it's going to be a notice.
Okay, so this should be our very first message, and let's just come down here and say we're going, before we even call main we're going to say, 'init logging but no filename'.
Let's run this and see that we've got everything working.
Woo-hoo, look at that!
We have our time, we have the level, 'notice', this comes from the log category, it's the start-up log, this is the message that we actually wrote.
Logging initialized, level is nine, which is a stand-in for 'trace', and the mode is standard out, that's why you see it here and not in a file, and of course our app still works.

Now that we have logging configured and we are calling that to set everything up let's go and actually create a app level and an API level logger, so let's call this the app_log and this, we already saw me say Logbook.logger and we give it the category so I'll call this App right and then we can use that throughout this part of our program and we'll know not just what the message was but generally where it came from and let's do something similar over in the API section.
So we'll call it api_log to import log book again and this will be API Okay, so when we're doing an accents with the API, maybe the timing and so on we could track that over here.
Let's start with the app log.
Now, I always find there's a super tricky balance between too much logging code getting into your application and making it hard to read being too verbose and not capturing what you need so this is a pretty simple example so it's a little bit hard to find that balance but,you can see that it's pretty easy to read.
Let's deal with the error cases first.
Let's come down here and say app_log It's going to be an error.
So this could either be error or it could be a warning right like our program isn't broken it's just in an environment that it can't operate in.
So I'm going to call this actually a warning.
I'll do a bore in here and it'll say same message.
So we can come out here and refactor that to a variable and just call that message more in the same little message like that.
Actually, we already have a classifier there so we'll say like this...
Same thing here...
You see it's getting harder to read but we will have a record of this so that's pretty cool.
Now this one, like these two we anticipate right.
This is if the internet is off.
This is if they don't type anything and they just hit enter.
This is kind of standard stuff so that's why it's a warning but down here this is like we didn't expect this.
We have no idea what went wrong.
Something went wrong.
We could say this error and give it the message with the formatted exception details added in there right?
Or there's actually a way to say there's some kind of unknown exception.
Just record that as an error so we can actually say exception and just give it the exception like this.
Okay, so let's go and try to run our program now and see what we can get to happen on it.
File just search for T, looks like it works so I'll just run a few searches see if I can get that exception error to hit.
Okay here, there's a very rare chance that some random error's going to be thrown.
Recall that from day 25 and I got it finally after a bunch of attempts to throw a StopIteration exception.
So here you can see it recorded as an error.
It said this isn't a category app and then it just put the trace back right there.
That is...
this section right here.
Let's try some other errors.
We know that it's an error to hit enter and that's going to be a ValueError because we're searching for nothing.
Here you can see error, you must search for a term.
This is just the print statement right there.
And then here, this is the log statement that we put as a warning.
Right, warning,the app says you must search for a search term and that's because we put it in the log as warning 'cause we know it's not technically broken.
The user's just using it wrong.
So the final thing to test is to turn off the internet.
Alright look, we got our standard print and then we got our log message which is warning app at the app level we could not find a server, check your network connection.
Alright, let's just add one final sort of success message here.
Let's go down here and say like app_log this info trace I don't know.
Let's put it at trace so this is super verbose but we can say something to the effect of..
Clean that up a little now let's see how this works.
Run it successfully, search for action.
Perfect, it worked and we logged the fact that trace app search is successful.
The keyword is action.
There were eight results.
Let's search for runner.
Here we go, search, we got six results.
Keyword runner.
Alright, so I feel like we're getting some decent logging at this level.
The other thing I'd like to do before we call this done is to look in here and maybe see if there's something we want to put in here in terms of logging at a lower level just to show how we can work with different parts of our application.

Now we've logged at the general top level of our app, let's log down into our API interaction here.
So let's do a couple things, let's do some time tracking here so we'll import this.
We'll do this time thing so we know how long this takes.
So that'll give us a start and end time we come down here and we say something like this api_log.trace we'll do some verbose things like "starting search for keyword" and then let's put this bookend here, "finished search for keyword".
Some results in some number seconds.
Put colon g grouping, okay.
So what're we going to have?
How many results?
That's going to be len of movies and the duration is going to be t1 minus t0.
So this is great.
We can also come down here and say api_log.warn("no keyword supplied") We could come down here and maybe store the status.
That's probably a traced type of thing.
Request finished.
There's our status code before we potentially throw in exception so we know instead of 500 is it 404, things like that.
Okay, I feel like this is pretty good, let's run it one more time and see if our app is ready.
So we started our logging, that's good.
We're going to search for "test", see what do we get down here.
API started the search for test.
The request finished, status code 200.
That's good.
And then it finished in this amount of time, that's seems like a huge weird number so we'll do some clean up on that number and then it finished and it came over here.
So, that's all pretty cool.
Let's switch this to milliseconds and do an integer times 1,000.
One more to see how that works, quick test.
There we go, that looks a little more clean to me.
193 milliseconds for our request, everything was good.
Let's try some sort of error.
Starting search for nothing, warning no thing.
Now noticed over here we have our API levels and we have our app level and we have our start up code.
So it really tells you like what part of your application is talking to you on this particular message.
Alright, so I feel like we've pretty much added really nice logging to our application.
But we're not quiet done yet.

So far, we've got this almost unreadable log goo mixed in with our standard user input/output.
So it's really, it's nice that we can see what's going on, but it's really not helpful for us here.
So let's change this.
Let's just go down here and, to our program, and when we init the logging, instead of passing in nothing, we're going to pass in a file name, and that's going to go to the timed rotating file handler, the same level rather than the standard out.
Okay, so let's go down here, and let's, we'll just call this movie app.log, something like this, and run it again.
Alright, now it's back to the way it was before in terms of interactions.
There's none of that mess around.
Let's search for action again.
And it runs, and we get our nice output, and, ta-da!
Let's run it one more time.
Let's search for hero.
Ah, we got a bunch of good stuff with heroes and so on.
And let's run it with an error.
Nope, must search for something.
Alright, how about jazz.
Anything there?
Hmm, looks like there is.
Pretty cool, so that's great.
Our app is working again, but notice, notice over here, we now have a movie-app.
Instead of .log, it has 2018-02-23, because that's today, that's when I'm recording this right now, so let's look and see what's in there.
So you can see it's exactly the same messages.
Here's the app is starting up.
Here's the app starting up again.
Things like that, here's another startup.
And these are all the messages.
We started a search for action, we got a 200.
We got eight results, that long.
Alright, and you can see the time of day to the super accurate right there.
Again, we're starting up, and this time we searched for hero and got 10 results.
This time we searched for nothing, and we got a warning, and so on.
Okay, so we have this log file here, and it's not super important, it's less important, let's say, when you're writing a regular app, 'cause you could just put the data and time in the log file.
But if you're writing a super long running service, like a web application that starts and runs basically indefinitely, or some kind of queuing application that's just listening for messages and is going to run, basically anything on the server that starts and just runs, the ability to have it automatically rotate - when it becomes the 24th, the next log message will just start a new file, for example.
So really really nice to have this timed, rotating file handler here.

Now that you've seen logging and logbook in action, let's quickly review some of the concepts.
So, we're going to start by importing the logbook, And if we're going to not use a file, we're going to need to also import sys, so we can get to standard out.
Then we're going to maybe get a file name, that can come from, like a configuration file, the environment, you could hard-code it, whatever you want.
We're going to set a logging level here.
Now I didn't actually demonstrate it, but if we went and switched that to say warnings, you would no longer see any of the messages that were trace, right?
Only the stuff warning and above would appear in the log file, or in the console, and nothing else.
So you can dial this up and down, depending on whether you're running a debug build, or production build, or release build, things like that.
So, set this, and that may also be based on configuration, or version, type of app, right?
Like it's production or development and so on.
Then we have to install a handler, so we're either going to use a stream handler, set that to standard out, set the level of push it to the application.
Or, if you want to go to a file, I recommend the timed rotating file handler, and do the same thing.
Once this is set up, then it's basically ready to go.
So remember this level acts as a filter here, so you can always go logger.trace, logger.info, but only if the level is low enough, will it actually show on these handlers.
Now, you do that, what we just saw, once, at the beginning of your app, and then, anytime you want to log something, you're going to create one of these loggers, like, heres a start up logger.
And we have this little info message, like, hey, we're getting started like this.
Now this just a string, nothing logging about it.
But then we can go to our startup log and say notice, this message, right?
And that's a pretty high level of log message there, this notice level.
And this is going to go, probably near the front, so it'll tell you what level you're logging at, which will help you understand what the rest of the log means.
We have different options, we have notice, info, trace, warn, error, and critical.
So critical is even more of an error than error, right?
We also have exception, if you just want to show a straight up exception.
And then the output looks like this, it has the time, it has the level, here, notice, it has the log category, log name, in this case, app, and then it finally has the actual message.
Really nice, really easy to set up, and it's super, super flexible.

All right, it's your turn to do some logging.
Let's look at all the steps that I'm recommending for you.
So there's a little summary of why you care about logging, but we're going to focus on the three things, the three days, and what you're going to do on them.
Today, you're basically done.
It's watch the videos as usual.
What you're going to do is you're going to pick an application that either you've built in this course or previously built that you think could use some logging.
It would be nicer if it had some logging much like we just added to the movie search app.
So your goal for today is just to pick an app that you're going to study and add logging to over today and the next two days that follow.

Day 2 you're goal will be to take the application you chose yesterday and look at the flow, and think about what type of things do you want to keep track of.
Do you want to track errors?
Would you want to track timing?
Do you want to track inputs, outputs?
All that kind of stuff.
So think about what you're going to try to log and how you're going to do it.
Do you want this to go to a file system, do you want it to go to the console, all those considerations.
So just kind of plan out what you're going to do the next day.

Alright, final day, Day 3, of this block.
You're going to use logbook, and you're going to add logging to your application.
Now this is an external requirement, so I recommend you have a virtual environment dedicated to this application.
Once you have it and you've activated it, you'll pip install logbook to get started right here.
And then, you're going to need to do a one time register of the logging, and then you'll be able to use it over and over.
So here's the basic steps of how we did it in ours.
You could choose other handlers and do other interesting things.
The StreamHandler and timed, rotating FileHandler, those are the two that I like, but pick whatever you want.
And then once you're ready to log something, you can create this sort of once.
You can create as many times as you want, but also can be just queried the NAT level in the application.
And then you're going to use it and, say, "log.notice.
log.warn, log.critical" and so on.
So, take that little bit of information there, and what you planned on logging and tracking the day before, combine them, and make your app production ready with logging.
I hope you enjoyed learning about Logbook and logging.
Be sure to share what you did with us on Twitter, and hashtag it appropriately.
And that's it, you now have this new skill.
You can do really cool logging in your application.
And it's one of those things that pays off in the end.

Welcome back to the 100 Days of Python.
In the coming 3 days I will guide you through refractoring and writing Phythonic code.
One thing is to write Python, the other thing is to really leverage all the great stuff Python has.
The Zen of Python states there should be one and preferably one way to do something, and that's great because in Python there usually is one best way to something, and the more you know these constructs and idioms, the more readable and elegant your code will become.
For this lesson I've prepared a Jupyter notebook with 10 practical examples how you can improve your code.
For example use a with statement instead of a try, accept, finally block.
Or use enumerate to not keep a manual counter or what about refractoring a long if, elif else statement using a dictionary.
We will touch upon list comprehensions, generators, using explicit is better than implicit in your exceptions, string formatting, tuple unpacking, PEP 8, the Zen of Python, and even some common best practices writing maintainable code.
And for day two and three I have you refractor your code or the code of somebody else to really put into practice what you've learned.
So there's a lot to cover, I'm really excited.
Let's dive straight in.

Alright, let's do this.
Let's look at 10 ways to make your code more Pythonic.
Let's start with these typical big if, elif, elif, elif, elif constructs.
You must've seen code like this, right?
You have the typical workout scheme.
We check it Monday, elif Tuesday, Wednesday etc.
And if it's not a day we raise the ValueError.
Now this is pretty ugly but it's also not extensible in the sense that if want another maybe combination of Thursday and Friday to do something we have to add another elif.
What if we change this in using a dictionary.
So that we can just look up the key and return a value?
I got this picture from the Code Complete Book which is an awesome read about code quality.
So to refactor that, let's start with defining a workouts dictionary.
And I'm just going to copy these in because it's quite some typing.
Alright.
And that gives us a workout scheme.
And note that the dates are in random order because it's a dictionary.
By the way there's another way to make this dictionary.
And that is to use zip two sequences.
So if I define a list of days and a list of routines, we can do something like workouts equals dict of a zip and a zip takes one or more sequences.
So days, routines and here you can see we have an equal dict.
Alright, now to go back to this refactoring example.
Now with the dictionary in place you can see how much shorter and nicer this function looks.
So note I can now just do a get on the dictionary.
Looking up the day, and it gives me the routine or None, if today was not found.
So we can explicitly check routine is None.
And raise that ValueError, as we've seen before.
And otherwise, just return the routine.
Let's try it.
Chest and biceps.
What about Saturday rest and call it on nonsense.
Yes I get a ValueError because nonsense is not a day.
Alright, that's our first refactoring.
And let's look at counting inside a loop next.

Next up, counting inside a loop.
So sometimes you want to keep track of an index when you loop over a sequence, and when you come from C or another language you would typically do something like this.
Let's define a list of days and let's loop through them showing the day prepended by the number.
Alright, so that's straightforward.
Now, and this is correct, right?
I mean you can do it like this, it's all 100% correct.
But the more idiomatic or Pythonic way is to use enumerate.
And a way to do it is to wrap your sequence in enumerate, which returns the index and the item in the sequence on every loop.
So I can just, now, print those, and it should give me, oops, obviously, I should not hard code days.
So I'm using f-strings by the way because I'm on Python 3.6.
And yes, this gives me the same result, and there's even a nice little trick with enumerate, which is, you can give it a starting point.
So I can just copy this, and again give enumerate a second argument of 1.
So I want to start the counter at 1, and then I don't have to do this menu plus 1 inside the loop.
And that gives me the same result.

Right, next up is the with statement.
You all have worked with files by now, I suppose.
The way to open and close files is like this.
That's fine, that's not optimal but if an exception occurs between the open and close statements.
Let's demo that here.
So we write hello and for some reason an exception happens.
The problem here is that the file handle called f stayed open.
So if I check now for is f closed?
False, it's still open.
So it's leaking resources into your program which is a problem, right?
One way to avoid this is to use try and use except finally block.
Try an operation, you catch the exception, if there's one and the finally block always execute, so you could put f.close in there to make sure it always closes, right?
You would write something like this.
Let's just trigger an exception.
I divide by zero which is going to give me a ZeroDivision error.
Let's catch that here.
Finally, I will always close my file handle and I do that here.
Open the file, write something, trigger ZeroDivision error, catch it and either working or failing, I always get f.close.
Let's see if the file handle now is closed.
And indeed it is closed.
That's cool, right?
This is much better code, but there's even a better, more pathonic way to do the above and it's to use a context manager or the with statement.
I can rewrite the previous code as with open This is the same code and the nice thing about with is once you go out of the block it auto-closes your resource.
In this case the file handle f.
This raises the exception as we told it to do.
Let's see if f is closed.
And True.
Look at that.
I mean although I like try except finally, it's definitely a good feature.
This is just shorter, more pathonic way to do it.
Use with statements or context manager if you have to deal with resources that have some sort of closure at the end.

Right.
Next up, use built in, learn the standard library.
There's some very powerful built in functions you can use that save you a lot of code.
For example, you run the range of numbers.
Well instead of for e in, or do this classical for loop.
I'm not even sure how to do it anymore.
Now, in Python you can just numbers equals range 1 11 and the first.
The start is inclusive, and the end is exclusive.
So just issue a numbers range of 1 through 11 which doesn't say much but if I convert that into a list there you go, 1 to 10.
And I didn't have to go through a for loop specifying end boundary, etc.
So very nice.
What about sum?
Let's sum up those numbers.
But in Python, you can just use sum, and look at that.
It's easier, and less code, and saves you time.
Let's look at max and min.
So let's create some data.
Again to stay at the gym, I have routines and I have timings.
Let's make a dictionary.
Workout times.
Now see before I can use the zip with routines and timings and put that into the dict constructor to make a dictionary.
So here are the workouts, and the times and minutes they should take.
What I want to do next is to get the workout that takes most time and less time.
Let's do it in the proposed way, if you wouldn't know about max.
Yeah, legs was 55 minutes which was easily visible here.
Now let's see the max building in action, and you will see that we can do this in one line of code.
Workout times.
items.
Again items gets me a tuple, a list of tuples of key value, in this case routine and timing.
Then I can use the key optional arguments to give it a or callable or lambda.
In the lambda, I'm just saying what I want to sort 'em which is the value, the timing, the minutes.
So this is basically telling max, that I want to get the maximum value based on the value which in this case is minutes.
There you go.
Boom.
That returns a tuple of key value.
Look at that.
Compare one, two, three, four, five, six, seven, eight lines of code with one line of code accomplishing exactly the same thing.
You can use min in the same way.
It should get me the core workout of 30 minutes.
I mean it takes the same inner logic, but as it is min it takes the min value.
Look at that.
Less code.
Leveraging the built in functions from the standard library.

Next up, refactoring 5.
Tuple unpacking and intervals.
So, what if you need to do a swap of variables?
We got a and b are 1 and 2.
In other language you have to keep a temporary variable so you store a into tmp then you can override, a with b.
Then you can put the temporary variable back into b and now they're swapped.
Right, so a now became 2 and b became 1.
Well in Python, it just takes one line of code.
Let's restore them.
Let's just do just swap them like this.
And there you go.
No temporary variable needed.
So that's step one, unpacking in action.
Another example of that is the earlier max function returning two values.
Here we had legs 55 minutes, so you can assign them or tuple unpack them by assigning them to two variables.
So in this case, routine and minutes.
There you go.
A function that returns two values can just be unpacked by specifying an equal number of variables before the equal sign.
In this case, routine and minutes.
N tuples, I will just do a quick demo because I already discussed them in more detail in a previous lesson.
That's not really saying that much because if I have to refer to them I would have to access them on day, what is that?
The second?
Okay, so it's zero base so I do 1.
I train, what do I train?
Workout.
Where's my training?
Okay, it's the first element at 0.
Okay, at least I got that right.
Okay, so if you do it that with n tuple.
Let's create one with workout equals workout with uppercase, which is the n tuple.
I always uppercase, unless because I see them kind of as classes without behaviors and classes you uppercase in Python.
You can either give them an args list.
You can also do that more verbose with a keyword argument list.
Now the print statement becomes a lot cleaner because I don't have to think about indexes.
I can just do workout.day, workout.routine, workout.duration, and it's much easier to type.
It's much clearer to the reader of your code what you're actually referring to and hence you'll probably make less mistakes.
N tuples, they're very easy to define and use.
It makes your code much more readable.
I would encourage you to use them whenever you can.

Alright.
List comprehension generators.
I did a whole class on this topic.
So, I just want to quickly recap what we learned here.
Because it should really be mentioned in refactoring.
Because it's one of those candidates to make your code more readable, and Pythonic.
Let's get the days starting with a T and let's do the old style keeping a list.
There you go, Tuesday and Thursday.
And again, this is fine, right?
However, you can write this in a more concise way.
Let's use a list comprehension.
So look at that, one, two, three, four, five lines of code reduced to one.
Awesome.
Next, let's do a quick generator example.
So, just to recap, let's make a random day generator.
So, we need some random function, and we're going to use choice.
So while True initiates an infinite look, I'm just going to yield the count, and a random choice of days.
Let's initiate that generator.
I call it daysgen.
And you can see that that's the generator object.
And a generator I can call next on.
And it gives me a random day.
Lets call it again, and I get Monday again, Tuesday, Saturday.
So it's random.
Okay?
I can use it in a loop.
So let's get five more days.
There you go.
Remember that the index was at four, so now it's five, six, seven, eight, nine random days.
And then the last nice thing to know about generators is you can use itertools islice.
And that lets you get a slice of the generator.
Because if I now put this generator in a list, my system would hang because this never ends.
It keeps on adding values, consuming memory, and there's no way to stop.
There's not a StopIteration, or a stop clause in here.
islice is nice.
It can just get a slice of the generator.
Just like you do a normal slice on your list, like first 20 items.
This is similar but works for a generator.
So, let's take an islice of daysgen, and I want to start it at position 100 and stop it at 105.
And then it's safe to put this in a list, because this is a finite sequence.
And there you go.
And that wraps up list comprehensions and generators.

Right, next stop: string formatting and concatenation.
And pretty important because you will be doing this a lot in your code.
It's funny the other day I bought some clay with my daughter and we were making Python figures and obviously we were very bad at it and then we got better.
And I found it a nice analogy with string formatting.
As the Zen says there's preferably one best way to do it and there are various ways to do string formatting but in the end there is a best way which is now the f-string in 3.6 but if you're not on 3.6 then at least you can use format that's basically the gist for me of doing formatting the right way.
But let's also demo that with some code And I hope you agree that those last two figures were definitely better than the first.
Alright, total hours is six print.
the course takes plus total hours to complete.
Okay a type error, not good so I cannot add an 'int' to a 'string' and vice versa.
So I need to make this a string and then it works.
But yeah, it's not the best way to do it.
And the best way is using f-strings in Python 3.6 and look at that I can just embed the variable.
This can even take operations, its pretty awesome but yeah, this is faster and it's much easier to read and I don't have to concatenate or even doing any type conversion.
So go with f-strings, but maybe you're not on 3.6 for some reason and then you can use format.
So you would write this as in the same curly braces and then you use format on that string and give it a variable.
And you don't have to worry about type conversion because format is, sorry, not using f-strings anymore format is smart enough to convert this 'int' into a 'string' or whatever is needed to make this work.
And then last note about concatenation so as said before, building up a string like this is not the way, and its slower if you're working with a lot of data and how true is that, right?
Every day you get to write Python is a happy day but, it's not happy for your performance and readability I would say, it's not really looking nice in this case you really want to use join so you want to really have your strings in a sequence or a list, and then just join them together and you can specify what to join on and there you go, here's my same string as above but using the Pythonic way of joining and, you know, you can do what you want you can also join it on dash, even better I can leave off the spaces here and give join a space.
So I'm going from multiple spaces or worry about spacing in the first place by doing that in one place at the join level so much better.

Number 8.
PEP 8 and the Zen of Python.
Any Python developer should become familiar with PEP 8 and use it in their code.
So here's the Style Guide for Python and you really should read this end-to-end and make a habit of formatting your code in the proper way, using variable names with underscore, so all these conventions.
There's even a recent initiative, pep8.org, which should make this even easier to digest.
And it's really nicely formatted and gives you some more context.
And, of course, if you do import this from your Python REPL, you get the Zen of Python.
And the more you write Python, the more you see how this applies to the language and it's design.
It's really where you start to better understand and appreciate the language.

One thing that the Zen is saying is explicit is better than implicit and I want to show you that concept with a couple of examples here.
For example, if you do a from module import *, it imports everything in your namespace if this would import full or bar and you define such a variable or function that will override it and that leads to very obscure box.
So don't do import *.
Make it explicit what you're importing.
So from module import full, bar.
Another coding horror is something like try and except pass.
Now this is like the black hole or monster that absorbs all possible errors, muting them away and you won't see anything in return.
So whatever happens, even if I were to interrupt the program with a control C which would raise the keyboard interrupt exception, that will be silenced as well so I cannot even kill my program basically.
So don't do this.
No bare exceptions, make it explicit.
So a little bit better will be for example, at least I get some information, or let's divide it by 0, I'm still shouting and I get another error.
And you see these operations lead to different exceptions so the best way to do it is to explicitly name the exceptions.
Again, try to divide number one by number two and let's go through the different exceptions we might get.
Here for example, if I get a ZeroDivision error, I just return 0.
That's my handling of the exception.
If I get a type error, and here I just print a message but I don't really have a way to allow this and so I just reraise the exception.
And if something else goes wrong you still can instead of doing accept pass or accept print something you can at least say accept exception as variable and browse that variable.
And you can send this to a log file for example.
To have at least a little bit more information when you're debugging the problem.
So now it's from the cell and call it with a string.
And here you see that it entered the type error and I get the message related to type error and it reraised the exception and if I divide by 0, here we don't have to crash anymore because I explicitly handled ZeroDivision error.
Printed the message and return 0.
Great, and that concludes number 9.
Explicit is better than implicit.

Alright number 10, bonus, the last refactoring item.
I'm just going to talk over some general coding best practices because of course Pythonic code is important and pleasant to read but it's often also a matter of sticking to the best of role coding practices.
For example, make sure your units of code are short.
Typically, a function or method should be around like 10, 15 lines max.
It can be very Pythonic but if you start to write functions of hundreds of lines of code, it's not good.
It's not easy to maintain.
So this is your 10 guidelines to make your code more maintainable.
Some other ones I want to highlight here, write code once duplication is a monster.
You really want to prevent having the same code defined in multiple places.
Keep unit interfaces small, so that means when you have functions the number of arguments to get if you keep that to a reasonable level, it's easier to maintain.
For example, if you have a function that takes 10 arguments, not easy to maintain.
You could consider passing around a class or refactoring to a class modules.
Again, that's the part of name spaces of the Zen.
It's good to have similar behaviors or similar objects grouped together in files.
To not have one file holding 10 classes, split those out in files, right?
Which makes it easier to reuse them in other programs.
Testing of course, you want to automate your testing that when you make changes you have this regression suite that you can just run in seconds and highlight if you introduced a new failure.
Which, if your system becomes more complex, might happen.
This is an automatic way to catch that and keep the quality of your software high.
In day 10 you learned about pytest and how to write tests so that's really important here.
There's also a lot of habit around this so if you get into the habit of writing clean code, keeping to high standards, it means that when you go into your code base, you leave the campground cleaner than you found it, right?
I love the anology in the pragmatic programmer book about having a small crack in a window if I remember correctly?
Leaving that unattended leads to broken windows and the same is true for your code base.
If you let small bad habits creep in it might actually lead to a lot of damage in the long term.
It's really about having good habits around writing clean code, run the PEP 8 checker on your codes, see if it's formatted properly, the right conventions are used, etc.

In this video I'll provide you some extra resources to read up about refactoring and how to make your code more Pythonic.
Read about the topic of refactoring, we have a whole dedicated post on the importance of refactoring code.
Errors should not pass silently, as we've seen.
So a lot of this we've discussed in this notebook, but it shows you some more examples.
This is an important one, there's some Python mistakes the beginner is likely to make.
And we listed them here so you can know about them in advance.
There's string formatting we discussed and Julian had a very nice article here.
From going from the terribly un-Pythonic method to the nicer way.
And I think he come to the same conculsion that you really want to use f-strings if you can, if you're on 3.6.
There's an awesome talk about beautiful idiomatic Python, by Raymond Hettinger.
I learned a lot from this talk, there's a lot of good tips and tricks in it.
And we've made a summary in this blog post.
And when we talk about modules, and splitting your code, it becomes important to know about packaging your code.
So here we have a post about how to do that on a practical project, like her Karma Bot.
And it's not that hard, I mean basically you have to make an init.py file and know how to import your stuff.
That's important.
The imports mights be confusing, but with this article it should become clearer.
So when we're talking about splitting code and making it maintainable, know how to work with modules and packaging is important.
And generic code quality resources, I have two.
First of all Martin Fowler's Refactoring Book, and yes, the examples are in Java, not Python.
But it's about the general principles.
And this book really teaches you how to write better code.
It shows you very practical examples how you can refactor bad code into much more maintainable code.
It's a great read.
And of course, Uncle Bob's Clean Code, a classic.
And also shows you a lot of ways to write professional quality code.

Lets go over what we've learned so far, 10 refactorings.
One.
Having a big if elif else block is not really maintainable and looks pretty ugly.
Rather, use a dict and just look up the items.
Much better.
Next, keeping an index when looping over a sequence.
In Python, you can just use enumerate.
And look at that, you can even give it a start value of one and you get the same result but it looks way more concise.
Next, context managers.
Don't use files like this.
If an exception gets raised, the file handle won't be closed.
This is a bit better because the finally block will make sure that your file handle will get closed.
Yet, much better, is to use a with statement which automatically closes the file handle after it's done.
Four, use builtins and learn about the standard library.
For example, here we saw a pretty verbose example how to get the maximum duration of a couple of workouts.
Not the best way.
You can just use max and min which are built into Python.
Here are two examples how you can do it in one line of code.
Tuple unpacking and named tuples.
You need to swap variables and using a temp variable?
Or what about indexing a tuple?
By indices?
Not optimal.
Use tuple unpacking.
For example, you can just swap the variables around.
No need for a temporary variable.
Or, to access to access elements in a tuple, make an attribute thanks to a named tuple.
Now you can just print, work out that day, work out that routine, and work out that duration.
And it's way more readable.
List comprehensions and generators.
Once you need to build up a sequence, you don't really need to build up that list.
You can use a list comprehension.
That's way shorter.
Or use a generator which when your dataset grows, will be more efficient.
Here's another generator to get a random day.
String formatting and concatenation.
Don't concatenate strings like this.
It's ugly and less performant.
Rather, use f-strings if you're on three dot six or later.
Otherwise, use format as a more elegant way to format your strings.
Another thing we saw was string concatenation when you build up your long string, don't do this.
It's not efficient.
Python will make a new string object with every operation.
You rather want to put all the string objects in a list and just join them together.
And here you see an example of that.
PEP 8 and the Zen of Python, your new best friends.
You will refer back to them, a lot.
First of all, to understand Python a bit better, why things are designed the way they are, read through Zen of Python and, I guess, print it out and put it next to your screen because it's a very concise list and it really explains a lot about Python.
And equally important, look at the PEP 8 style guide and try to abide those principles because they make for more readable code and more consistency across code bases.
There's an awesome resource under PEP 8.org that makes it easier to understand.
And I forgot to mention in the lesson, you probably have a shortcut under PyCharm or Vim that you can just run PEP 8 checks or Flake 8 upon save and catch those errors early on.
Definitely something where you want to go from average to awesome.
Explicit is better than implicit.
That's literally quoted from the Zen.
Don't do try-except paths, like never do that.
Don't mute all the exceptions, being kind of a black hole, beam into an obscure box, it's just bad.
Rather, be explicit in catching your exceptions.
So in this case, dividing num one by num two.
You can have a ZeroDivision error or a type error or any other exception and all have their specific message and handling.
Code quality overall.
Which the Software Improvement Group has nicely wrapped into ten principles.
In short, keep your methods small, don't pass along too many function arguments, keep your code organized in modules or files, and automate your testing.
Of course, there's more to it.
It's a whole study in itself.
So you can read their building maintainable software book but I also recommend Clean Code by Uncle Bob and Refactoring by Martin Fowler.
That's really where you want to go from coding horror to writing awesome code.
Your code can be very Pythonic, but if you're writing methods of like 50 lines or more, you still want to go back to general code quality principles.
Alright, now it's your turn.
Keep calm and code in Python.

Welcome back.
In this second and third day, I encourage you to take our Code Challenge 30.
It's The Art of Refactoring: Improve Your Code.
And, the task of today and tomorrow is to get some of your code, run tests and do some refactorings, making it more Pythonic.
Also, I would recommend to start looking at Flake8, or Pylint.
I personally use Flake8.
And integrate that into your editor.
And have a check upon each save.
Or, what I did, for example, in the vimrc, I have a shortcut:, f.
And when I press that, it runs Flake8 and it opens a new editor window where it highlights my PEP 8 violations.
So that way I keep my files clean during development.
Optionally, you can look at Code Challenge 35 and use Better Code Hub to look at your code quality overall.
Using how we use the tool to improve a couple of our projects.
And don't forget to tweet out your progress.
You can mention Talk Python and PyBites in your tweets, and we would love to see what refactorings you come up with, or what your favorite Pythonic concept is you learned from this lesson.
Alright, good luck and have fun.

Hello, it's Michael Kennedy, and I'm going to be your guide for day 37, 38, and 39.
And this time we're going to work with structured data called CSV files.
So anything that can be stored in something like Excel or Tabular, data like that that you might work with in some kind of Excel spreadsheet.
Much of the data you'll find out on the internet lives in this form, and we're going to find some really, really interesting data sets, and we're going to build some programs to ask and answer some pretty powerful questions.
Let's get started.

Let's talk about our data sets real quickly.
You're probably familiar with CSV data, but just in case you're not, it looks like this.
It's a plain text file and there's a set of headers across the top that tell you what each column is.
So here we can see we've got data, an actual mean temperature, an actual minimum temperature, and an actual maximum temperature.
And then we have a bunch of columns that correspond to that data.
Now I told you we're going to work with some interesting data sets and it's true, I found a good collection here for us to play with.
Now you may be familiar with a place called FiveThirtyEight.
It's like a data-driven journalistic news site where they gather up a bunch of data and they use it to write articles and do investigative journalism type things.
And it turns out every article they have, the data that they use to derive those conclusions is available online on GitHub, so that's pretty awesome.
So over at github.com/fivethirtyeight/data, that is where we're going to be working for the next three days.
All right, so let's jump over to my web browser here and we'll just have a quick look through all the data.
I told you there's a lot, look at the size of that scroll bar.
There's a ton of options here.
So let's skip down past all the folders and just go to this section.
So here you can see all of the articles written by FiveThirtyEight and then the corresponding data that goes with it.
So let's just grab one here, American Chess is Great Again, and if we come back, click on it and you can see here's the actual article that they wrote about, and you can see here's the graphs that they drew based on the data and so on.
But here is the actual data, so you can come over here and it actually describes what it is and so on.
If you click on it, you can see here's all the data that they were using to make these conclusions.
So what we're going to do in this section of the course is we're going to take one set of CSV files and use that for our demos and ask and answer interesting questions.
And then for the next three days afterwards, you'll be turned loose to build your own investigative journalism app.
You'll choose one of these CSV files and come up with a set of questions, and it's going to be a ton of fun so I hope you're ready to get started on that.

For the learning section of today, what we're going to do is we're going to write an app that takes a CSV file, and answers a set of questions around it.
The data that we're going to use is from the 538 data set, data/us-weather-history.
It's not very clear what this means, but this is the weather history from different cities.
You can even actually see here's the the data that they, the code that they used to visualize it.
Here's the web scraping, they go to Wunderground which is the weather site to get the data, pretty awesome.
So we're going to use this data set, this is the Seattle weather data for 2014 and 2015, I believe.
Yeah, there's the dates right there.
This should look familiar, right?
Date, actual mean, actual man, actual temp and so on.
So we're going to take this data and we're going to work with it.
Now, let's make sure we start with the raw, we do not want GitHub HTML, you want the raw.
So we're going to save this.
And I'll just save this as seattle.csv.
Now let's go write some code to work with this data file.

Alright let's create our little application for our demo that will let us work with the CSV data.
I'm over here in the actual GitHub repository and we're going to create our application here.
Now I want to create a virtual environment in here before we get started and maybe I'll even name it venv So, I'm going to go to that same folder in my terminal here and I'm going to run the command to create the virtual environment before I open this in PyCharm.
Alright.
If you were going to continue working in the terminal here and you wanted to say, run Python commands you would do this on Mac and that would activate it, and if you were on Windows you would just say venv\scripts\activate.bat like that.
Either way, I guess you would use backslashes wouldn't you?
But, until you get started here I'm not going to worry about this because I'm going to work in PyCharm and so if I throw this over here or on Windows or Linux say file, open directory it'll open this up and I'll go ahead and let it add the GitHub root doesn't matter so much for you guys but I'm going to of course check all the stuff in.
Now, we can just go to our virtual environment and say let's just ignore this, it doesn't really matter and we're just going to get started like we have been by creating a program.py this is going to be our little way to explore, and this is going to be the top level thing that we want to work with.
So, I'm just going to print out kind of the basic structure of what we're going to do in terms of working with this data and then we'll actually go write the code to implement that.
So, I'm going to define a main method here and just for a minute I'm going to do the pass.
We'll do this little structure that is very common in Python that says only directly execute this code if it's being invoked directly, if it's being used as a library.
Don't run main just leave the other functions here.
So this is the common pattern and we're going to print out a few things.
We'll print out a little header value.
Alright, so weather research for Seattle 2014 to 2015 and we'll just put a blank line like that.
And then we're going to need to initialize the data.
Spelling is hard that's why PyCharm can fix it for us.
Okay, so that's going to be great and once we get down here, we want to answer the questions.
What, say, the hottest five days?
And then we'll say to do show the days.
And we're going to do this for a couple of different ways.
I'm going to come in here and I want to answer the coldest five days and the wettest five days.
So, this is our goal is to run basically answer these questions and we're going to do that by reading that CSV file.
Before we do, let's just really quickly run and make sure this works.
Hey, it tells us basically here are the days, but doesn't yet show them to us.
So we're going to need to get the data.
Let me actually make a little folder to organize that here.
I'll call this data.
And into that data file, I'm going to drop this thing we downloaded in the previous video.
Drop that there.
PyCharm will put it over in the right place and we can look.
Does it look correct?
Yes.
Apparently PyCharm can help out with CSV files.
I don't really care.
But I do care about what the header values are going to be.
We're going to work with that later.
So maybe go ahead and copy that preemptively.
Now, I think we're pretty much ready to write the code that is going to read that file and then provide the data so we can answer these questions here.

Our goal is to read that CSV file.
We technically could do that right here, but I would like one part of my application to just be dedicated to reading, and parsing, and working with the data.
And another to do with this, kind of, UI business.
So let's make another Python file called research, and we'll put that at the top of level of our program here.
Now we're going to want to work with the CSV file.
So let's come over here and define a function called init.
And this is going to basically load up the data, the data file and then get it into memory and we can answer questions about it later.
We're going to have this global shared piece of data like this, and we're just going to make that a list.
So we're going to initialize this, and now what we need to do is we need to determine this file.
It would be very tempting to say file name is just ./data/seattle.csv, or back slashes if you're on windows.
Now, be careful, backslashes and strings don't mean quite what you think they mean.
This, if you said something like \t, notice how it changed colors, or \n.
Backslash and strings is used as a, what's called an escape character, and it means \n actually stands for a return, and \t for a tab and so on.
You can use double backslashes to say, no no, I just mean the backslash.
All right, so be a little careful there.
But, even so, that withstanding, this, and the spelling as well, this is not going to be working out really well.
It'll work if we'd run it actually, but only if the working directory is this folder.
If you run it from the command prompt, or terminal, and you're somewhere else, this isn't going to work.
So we're going to take a slightly more complicated step to determine this location.
So we're going to come up here and import the os module, which let's us ask cool questions.
And we'll say folder, let's call it basefolder, = os.path.dirname of where.
Well, one of the things we have to work with that's in variant is where this file is located.
And what we want to do is go to where this file is located, go into the data folder, and go here.
And there's a way to always get to that in any Python file just say dunder file like this.
So, this is a file, dir name, we'll drop the file part, and just keep the directory, and then we need to say filename, here is actually going to be os.path.join, and we want to put together the basefolder and data, no slashes, and seattle.csv, okay?
So this will do that in a platform independent, location independent way.
And it's really, you know, not that much work above what we were maybe going to type in the first place.
So let's just open this, we'll use a with block, we'll say open file name, and we can give it a read, and one can even say encoding=UTF-8, that's the safest guess, and we'll say as fin, that'll name our variable.
Let's just do a quick print, forget CSV for a minute, we just want to see what's in here.
So we can say, read lines, or just read, look at all the text.
So now, let's call this function, it's not done, it's not even close, it doesn't do anything with the data, it just pulls it in as text, but it's going to verify this one aspect of what we've done, reading the file.
So let's go over here, and let's, before we do any of this, let's go do our initialization.
So we want to go to the very top, say import research, all right, and then down here we can say research.
forget data, we're not going to work with that directly, but we'll say init.
Now, let's run this and see what we get.
Boom, we saw a whole bunch of CSV data scream by, just like that.
How cool is this?
So this is totally working well.
We've got our research, it can find the file, and I can tell you it will find it from anywhere on the computer.
It doesn't matter, it's going to just use the location relative from here to there, based on this thing that we did.
All right, so that's how we load the file up and get a file stream, that's one half of the problem.
The other half is to take that giant string and convert it into data we can work with.
That's the next step.

We were able to read our file as text, and that gets us really close, but it doesn't really let us work with it.
And now, technically, we could do, like a crazy bunch of string operations and parse this.
But, it turns out it is entirely unnecessary.
So we can import the csv module.
This is a built in in Python.
I'll call this reader, and I'll say csv.DictReader.
There's a couple of kind if readers.
DictReader is the best one.
And you pass it this file stream.
And what this thing is, is it is something that you can loop over, and every time you loop over it, you get the values out, okay.
So, let's do just something really, really quick, and, just for row in reader, and just so you can see it working I'll print out, let's say.
So if we run this, looks like it's working, and what we get is something called an Ordered Dictionary.
Doesn't matter that it's ordered, all that matters to us I can ask it for the date, or the actual mean temperatures.
Like, if just want to print out this part, I can go, because this is a dictionary, let's comment that out, I can just print here, row.get the actual mean temperature.
Great, it looks like we already got a lot of data, and we could work with it.
There's one super, super challenging problem, though.
If I really want to work with this data, you very likely want to ask, is this thing less than that other thing, right.
Is this temperature less than that?
So we can answer our questions.
What's the hottest day, what's the coldest day?
Things like this.
So, let's print out the type, right.
This is going to tell us the actual underlying data type here.
Strings, everything that comes back from CSV files are strings.
Even if they look like dates, or they look like, you know, numbers or something.
And so in order to actually work with these, we're going to have to do something a little bit different.
So, we're going to actually write a function that takes this sort of raw row that we have here, and instead of just saying we're going to store the text, we're actually going to convert the numbers to numbers, the dates to dates, things like that.
So we can actually answer questions like, is this number less than that number?
Alright, the string less than or greater than is not the same thing as numerical less than or greater than, right?
Like, 100 is less than 7, for example, if those were strings, but obviously not as number.

Now, we've actually parsed the CSV file and we're ready to go.
But we saw that everything that comes back is actually just a bunch of strings.
And we can't really do data analysis.
When you want to do numerical operations or date time operations.
But the data type is a string.
So what we have to do is that we need to write a function that will take one of these rows.
And kind of upgrade it to its types that we know exists in there.
So let's go down here, we're going to write a function called parse_row And we'll give it a row and it's going to return some kind of item.
So, first of all, let's write one that just actually upgrades the values.
And we'll do one more step beyond that.
So, we know, if we look over here.
That we have a date, we have an actual mean and all of these things.
So, let's start by coming over here and we're going to upgrade the rows date.
Let's upgrade the temperature, it's a little simpler, first.
So we'll come over here and we'll say actual mean temp, that's from the header.
Now, see, this value is actually the result of converting it to an integer.
And we also have the actual min temp.
Now, you want to be careful here, of course that you don't cross those over but also, that you're using integers with the integers and the floats where there are floats and so on.
Now, this is not fun to watch me type this out for each one, and there's a lot of it, it's quite tedious so let me write this out.
And then we'll come back to it.
Alright, here we are.
So now we've taken everything that we found in the header and we've done a conversion from strings to numbers.
Sometimes those are integers, sometimes those are floats we paid careful attention to when that was the case.
If you're unsure just use floats and then we're going to return this row but by getting the value out and then replacing the value with the integers we should be able to come over here and print this.
So if I do this and, kind of what we did before we do a little type of that.
We print those.
Now, if I write on it, Actually...
Excuse me if I do this parse_row.
And we kind of upgrade this row here And notice these are now integers if I change the column to the actual precipitation you'll see these are floats.
Here they are.
Those look like floats, don't they?
Great, so we've upgraded this and, it's already in a really good shape.
There's one other thing I want to do to make it nicer.
And that is to use a data type that's built into Python that helps you work directly with sort of these values in a really simple way, and it's called a namedtuple.
So let's go to the top here and we'll say import, collections and inside Collections there's this thing called the namedtuple.
So we can actually define another type something better than this basic dictionary which is cumbersome to work with.
You got to do the .get, the value might not be there, things like that, so let's define a record, like a weather record, and we're going to do that by saying, collections.namedtuple.
Now, you give it two pieces of information.
One, you...
You basically replicate this, so you tell it, I'm calling you this, but, your name is what I'm calling you it's sort of so it knows what its name is.
And then the next thing you give it is simply this giant thing up here, okay, like so.
Now, PyCharm says, whoa whoa whoa, what're you doing, that's like line, or column 200, this is insane.
So why don't you do some line wrappings, just so people can read what's going on, right.
And this Python will turn that back into one long string because there's no comma separating it.
These might look like they're separating it, but they're on the inside, not the outside.
Okay, so this is going to let us define a type, so come down here, and I could actually upgrade this, so I could say, r equals record, and if you look and see what it takes, it takes a date, a temperature, all the temperatures and so on.
And I could say date equals this, mean temperature equals this, min temperature equals that.
So I could say date equals row.getdate comma.
Did I say data?
Oh, no, date, yeah, date.
So date equals that and actual mean temperature equals row.get, this is starting to feel tedious isn't it?
And we got to do this for every one of those.
It turns out these rows are dictionaries, there's a shorthand to say this statement, for every element in the dictionary.
So if I want to say, if date is in there, go get the value assigned as date.
If mean, actual mean temp is in there, go get that value and assign it to this, and the way you do that is you say **row.
Star star row just says, well, do what I erased, right.
Set this value to the value from the dictionary, set this argument value to the value of the dictionary, and because what's in the dictionary is literally what we put right here, this is going to match exactly and this will work.
Alright, so now let's return this little record thing instead and we can rename it better to record.
There we go.
And so we'll call it record here.
Now we go over here record and say ., and notice it gives us a nice, beautiful list of all the stuff that's in there, we don't have to do this style over here, I'll just say, I don't have to know it's a string and is the actual precipitation, I just say record.actual_precipitation.
Then into print out the value, then we'll do it again.
So now it should work just the same, boom, it does.
Okay, whew, so we've now converted our data, the last thing to do is to store it.
So instead of printing out, which is kind of fun, but generally not helpful, we're going to go to our data and say, data, append record.
Alright and just in case somebody goes and calls this a second time, right, we don't want to over do this, whoops not record, we'll say data.clear.
So we'll reset the data and then we'll load the new data from this file just in case you run it twice, probably not going to happen.
Alright, so now we're basically ready, let's just check and see that this worked.
Let's go over here and just print research.data, just to see that we got something that looks meaningful.
And look at that, we did.
Record, the dataset, the mean is, we actually didn't parse the date, but just keeping it simple.
Actual mean temperature and so on, you can see this goes on to the right for very long.
But it did exactly what we wanted.
So it looks like we're off to the races.
Now the final bit, actually this is the easy part, let's answer the questions now that data is super structured to work with.

Alright we're ready to answer the questions, let's go through this again.
I'm going to move this down just so it fits right here, so we initialize the data.
Boom, we're doing that.
The next thing we need to do is say research dot let's say hot days.
Now what we're going to do is we're going to write a function here that is going to give us all the days, hottest to coldest.
And we come over here and say days equals this, then we could loop over and say for d in days, and that would show all of them.
But we want just the top five, and so the way you do that in Python is using something called slicing.
So you can come over here, and say I would like to go from index 0 to index 5.
Alright so this gives us a little subset of our thing, and when it starts at zero or ends at the length you can just omit it.
So we can write it like this, and that will process the first five of em'.
And then let's just say, print d for a second.
So we're going print out what that looks like.
So let's go write this hot days function.
Alright so we want to go through our data, and figure out the hot days.
And it turns out, the easiest way to do this is just to sort it.
So we can say return sorted data.
Now that's going to sort it by, jeez I don't know, maybe the first element, alphabetical, by date, I'm not really sure.
So we want to control what it's sorted by, and in here we can say there's a function that is going to take basically one of these items, one of these rows, and tell us what we're going to use to sort for it.
So we're going to say key equals lambda this is a little in line function.
Lambda says there's a function, then the next thing we put is the argument.
So let's say r for record, we do a colon and then we have the body of the function, the implementation.
Where do we want to sort by for hot days?
Let's say with a max temperature for the hot days.
Actual max temp is what we want.
So we're going over here and I'll say r.actual_max_temp.
Now this is close, this is going to actually give us the lowest value first, and then the highest value to the end.
So the default sort is going to go from low to high, how do you reverse it?
There's two ways, we could say reversed=True, I'll spell it correctly, that's one way, or another way a little more flexible, is to just put a negation here and say multiply that by a negative number.
So take your pick, you can do it either way.
While we're here let's write the cold days function, that should be easy.
On the cold days take away that.
And let's do wet days.
And this time we're not going to sort by that, we're going to sort by actual precipitation.
And again wet days are where we have more, so we want to sort that in reverse.
Now the format is a little off so reformat the code.
So PEP 8 is happy.
Alright so those three functions actually should do what we need.
So let's go over here and we'll just print out the five hottest days.
Now this is probably not how we want to view it, so let's print this out slightly differently.
So I would actually like the index, like this is the number 1 hot day, this is number 3 hot day, so I can come over and say idx, and instead of just looping over the days, I can innumerate them and then loop over.
And that will give us the index and the day, for each time.
So then in here we'll put something like a little dot to say what day it is and we'll say the temperature in terms of Fahrenheit on such and such day.
And we'll just do a little string format on this.
So what this is idx and it's zero base, you don't talk in terms of 0 1 2, you talk in 1 2 3 so plus 1.
And then we need the day, dot.
Now notice we're not getting any help here, let's go back to these real quick.
So we can come over here and tell Python this is a record.
Now if we do that, sorry not a record, you can tell it it's a list of record.
Now this is a Python 3.5 feature, and if we come over here we import this typing.List.
So at the very top, we have from typing import List, and we put this here, and I'll go ahead and while we're here just put it on the others because they're all the same type.
We come over here and now we go back to this, and I say let's try that again, d.
Oh yeah, now our editor is smart, now it can help us.
What we want here, we want the actual max temperature, and then we want d.date.
Alright, let's go and run it, see if this whole thing's hanging together.
Boom, look at that, hottest five days, 96-94-92-91-90.
And those are your dates, that's awesome, right?
See how easy it is to answer the question.
The challenge was not actually answering that question, the challenge was taking the data, reading it in, and then converting it to a workable format and storing it in our record.
Once it's stored like that it is easy.
Let's go ahead and do the same thing for the coldest days.
So we'll say that days is not the hot ones but now it's research.cold_days, should do exactly the thing.
And let's do a little print in between just so there's some spacing, and you guessed it, the wet days, same thing, just ask for the wet days.
Let's run it, so there's our hot days, the cold days look really cold, well doesn't look that cold does it?
Did I get the cold?
That might be the high on the cold days.
Let's go down here yeah.
Yeah, yeah, careful.
This going to be actual min temp, like so, and this is going to be actual precipitation.
There we go.
Let's say, inches of rain, there we go.
Oh yeah, now that's starting to look cold isn't it.
23 Fahrenheit, that's like negative negative 5 Celsius for those of you who are on the Celsius scale, this is like 36-37.
Alright so pretty hot, pretty cold.
I notice the sorting is correct.
The weather stays, the heaviest amount of rain during that time was 2.2 inches, or 5 centimeters.
And then it goes down pretty quickly from there.
So hopefully you really got a good sense of how to take the CSV file in, read it, parse it, convert it to a usable format and then just quickly answer questions.
We stored it in a list and sorted the list, there might be other data structures you need to use for your data.

Recall there were two basic steps that we had to go through to work with this CSV data, and they were both pretty simple.
We're going to start by importing the csv module, so that will let us actually create this dictionary reader to run through it, and we're going to open a file stream that we hand to the dictionary reader to actually get the data.
Then we just create one of these dictionary readers based on the file, and gives us this thing that we can loop over and each item that comes out of the loop is going to be one of those rows in there.
And we get this row back and we work with it one at a time.
Now you saw the data's always text, so you may need to upgrade it to something more usable if you're working with numbers and dates and things that are not just plain text.
The other thing that makes this really, really nice, makes a big difference in terms of working with this data, is to give it a little more structure, and we saw that we could really easily use something called a namedtuple.
This comes out of the collections and then we're going to create the namedtuple here and we call it whatever you want, we make up the name.
I said I'm going to call it record, so record it is.
Just remember it appears in two places, and in that second part, we actually put the basically the header from the CSV, it just so happens that format works perfectly there.
And then we can, if we have some data, we can create one of the by setting the keyword values.
So we set all the values there.
We actually have a shorter way to do it if the dictionary exactly matches the items, which it does in this case.
So we could use the **row, because often there's a ton of keyword values to set, and this will just do it in a super, nice shortcut here.

Now that you know all about working with CSVs, it's time to do some data journalism.
You're going to come up with an amazing question, and find a data set, and answer some questions about it.
So, let's get you the steps here.
First day is, you're basically done with the first day.
It's more or less to watch the videos, but the final thing to do, I hope you're inspired, is to head over to the GitHub repo, fivethirtyeight/data.
And sort of look through there and find one of the data sets that looks interesting to you, and think about answering some questions.
So, here's what an example of that may look like.
Alright, so you might want to write these three things down.
So, here's the goal, is, I found, maybe I should put data set first in terms of the order, but I found this data set on where you live in the United States you eat different things on the U.S.
holiday Thanksgiving.
So, if you live in the American South, you'd have one type of thing, if you live in the North, in like the Northwest, you eat something different.
It also varies by income, so pretty interesting.
So, you go over here and the goal is going to be to predict, you know, ask two questions of the user, and then predict what they have for Thanksgiving.
So, you ask them where do they live and how much money does their family make, and you're going to use this data set to basically, generate a set of things they're going to eat, right.
You're going to eat turkey, and stuffing, and mashed potatoes, and things like that.
And they answer the questions differently, that menu that you provide to them might be different.
So, here's sort of the steps that you need to do for those few things, alright.
I think that's going to be fun.
You don't write the program on day one.
You've already watched all those videos and listened to me talk, so you're goal is to just find the data set and have a question.

Day 2 or Day 38, if you're adding them, is to actually create the skeleton of that program and just open a CSV reader to that file.
If you can just load it up and just print out the rows, you're done with day 2.

And on Day 3 you're going to want to actually work with the data.
So you need to transform it into a way that you can work with.
If you need to work with that particular field, I think my Thanksgiving example might not actually need it because of the way that that works.
Depending on your data you might have to, you know, sort of upgrade it to the numbers or numbers and then, you know, ask the user questions and try to basically provide an answer based on the data.
When you're done, share what you learned as always, and thanks, hopefully you enjoyed this section.

Hi everyone and welcome to JSON in Python.
This is Julian Sequeira and I'm going to be walking you through some of the more interesting ways of dealing with detailed JSON output.
So JSON, if you're not familiar with it, stands for JavaScript Object Notation and it's pretty much just a way of formatting data, okay.
One of the most common ways of seeing JSON when you're working with Python is through contacting APIs.
Through working with numerous APIs out through the web.
And one of the things about that is that you actually get really complex dictionary nested situations going on.
So the JSON output, if you haven't seen it before, just looks like a little dictionaries and lists and if you get really deeply nested lists and dictionaries, it gets really complicated to try and pull out that data, which is why JSON can sort of screw with you, pretty much.
So, what we're going to do is we're going to go through that in the next couple of days, have some fun with it and see what other cool APIs you can talk to by the end of it.
So, let's move on.

And here's how your next 3 days are going to look.
So, for the first day working JSON, you're going to watch two videos.
You're going to inspect some JSON schema, and then you're going to watch how to pull down and decode JSON data.
Now, the data set you're going to be pulling down does need an API key which I don't think you'll have, and you don't have to get it for this; you can just follow along.
I've also actually put the data set that we pull down into the repo, so within this repo you can find it.
Now for Day 2, you're going to look at how to pass the nested dictionaries within that same data set.
So for Day 1 you were just playing around with it with what you found in the video.
Now for Day 2, that's when you drill down further into the actual JSON nested dictionary part.
Now once you've watched the video on how to do it, I'd like you to spend the rest of this day, Day 2, playing with it.
So you're going to actually just use that data, pop it into your script, into your shell, whatever it is that you're going to be using, and then go ahead and just play with it.
Follow the same advice and have a go.
And Day 3 is your turn, so what I'd like you to do on this one, little more interesting, is I would like you to go to our PyBites Code Challenge 16.
We'll bring that up here, and that's all about querying you favorite API.
A lot of APIs, as we know, will return JSON data, so what you can do is, and a good example is the OMDB API.
You can use that to return JSON-formatted data on your favorite movies and then come up with an inventive way to present or request that data.
So have a read-through, have a go, and share whatever it is that you come up with.

Okay, so to start we're going to look at a basic example of JSON output or JSON schema.
Okay so it is pretty complex at times, especially when it starts getting really deep.
So just look at this basic example to understand it.
So from Python perspective, it looks pretty simple for now.
This is a really basic one again and it's about a sort of person object so you've got a title of a person.
You got the type, it's an object.
And then what are the properties of that?
Well there is a first name, a last name, an age, a description, and so on and so forth.
What really gets people and gets me sometimes with a JSON is once it's decoded in Python, all of these become nested dictionaries and lists.
And it makes it quite difficult when you're writing loops and what not to try and drill down to those nested dictionaries.
So looking at this, if this was formatted like a dictionary in Python, your first level of the dictionary has the key title value person, key type value object.
Key properties but the value of properties is another dictionary and that dictionary goes all the way down to here, okay.
And within that dictionary, you then have another key called first name whose value is yet another dictionary, okay.
And then that closes off here and then you have another key whose value is yet another dictionary and so on.
When you break it down visually, it makes a lot more sense.
That said, it is still a little bit of a crazy process trying to drill down, okay.
And we're going to work through that in the next couple of videos.
So this is just a nice basic example of what JSON schema looks like.
So have a good look at this.
Wrap your head around it and move onto the next video.

So, for this video, we're going to look at some JSON data that is returned by an API.
Okay, we're not going to just download a file and parse that.
What I'd like to do is show you that a lot of APIs respond with JSON data, okay, JSON formatted data.
So, what we need to do in order to look through all of that is, first, we need to import json, obviously, so we can decode the data.
We also need to import requests, because that's how we're going to get the actual data.
All right?
And, for later on in this video, we're going to get pprint, all right, and we'll discuss that in a bit.
Now, there will be some magic here, because I don't want you to see my API key, but, essentially, we're going to go r., r is requests, .get.
Okay, let me do some magic here.
Copy and paste the key and get this JSON data pulled down, and then we'll carry on.
All right, now I've just given us some white space here to work with, get it out of the screen.
So, r has been pulled down, the actual JSON has been pulled down, and we're going to assign that to the data variable.
Now, data = json.loads, all right?
Loads allows us to actually decode the JSON output.
Okay, so the JSON output is pretty ugly, and using json.loads, that allows us to actually decode it, and make it readable.
So, we're going to load in r.text, okay?
Now, if I was to hit data, and enter, and show you that, it would probably kill my session because there's so much data in this.
So what I've actually done is, I've copied and pasted this output into a text file for you.
And that is here, okay?
So you can see this is all the data for my character in the game.
Now, you can see the sort of nested dictionaries I was discussing before.
Okay, you've got, at the highest level, you've got a dictionary key there, and a value, and then we move on to the next one, and so on.
Keep on filtering down, until we get to mounts.
Now mounts is just, long story short, mounts is a sort of creature you can ride on in the game to get around.
So, under the mounts key, we have a large dictionary here, called "collected," okay?
And within that collected key, we have our list of dictionaries as the value.
So, you can see how far it drills down, so we've got the parent key here, whose value is another dictionary, with the key to a list of more dictionaries, okay?
And each mount, each animal that you ride on in this output, has these keys, okay?
So it's a whole array of data and it just goes on and on and on and on, and then you can see number collected, we drop out of that list here, at the end, and we get back into that, we go up one level, and we see number collected, number collected, number not collected, and then we go up to the parent level.
I want to call it the parent level, the top level, and we see realms and whatnot.
Okay?
So that is what that data looks like, the stuff we just pulled down, so how do we work with that data, okay?
Well, to look at that data, we need to treat it just like a dictionary, okay, there's a little bit of difference, and we'll get to that in the next video, but, for example, we now know what the data looks like, so it allows us to figure out how we're going to flesh it out in the Python code.
So for item in data.items, I'm just going to do a standard, sort of, dictionary parse here.
We're just going to print item, okay, and watch what happens.
We get all of that data here and it is, honestly, disgusting.
It does not format correctly, okay.
Now, I've just scrolled that out of the buffer to get the distraction away.
We can then go for item in data.items, okay, same as we just did, but this time, we're going to use pretty print, or pprint, okay, and the beauty of pprint is that it actually knows what JSON data looks like, and it formats it nicely for us.
Okay, and this is the same output we saw in our Notepad file just then.
Wait for it to continue flooding my screen, probably overloading my computer, and it's going to look just like this.
Okay, so there it is there, and that's pretty much how I got that Notepad file in the first place.
So this is JSON data.
This is importing it into Python, into your Python application, and this is pretty much printing the entire thing in a nicely formatted way.
So, in the next video, we will cover how to drill into this, but for now, have fun.

Okay, so carrying on from the last video, we have this gargantuan bit of JSON response, and it's been formatted nicely by pretty print, to look just like this.
And what we want to do, the aim of this exercise here is to take the names of these mounts.
These in the game I've collected all of these mounts in this list here.
In this list of dictionaries, and I want to get the list of them.
So how do we do that?
Well it's easy enough using key and value calls for a dictionary, to get this data here, so we're not going to cover that.
What we want to do, is talk to the data that's nested below these lists and dictionaries.
So we need to talk to mounts, the mounts key, then we need to talk to the collected key, and so on down to what we want.
So how do we do that, how do we do that in Python code?
Well the first thing we need to do is we need to start off our normal for loop.
So we're going to do for item in data, but instead of doing items as we normally would for keys and values, we're actually going to do some tags, so the key that we want to look at specifically is mounts, isn't it?
So let's just double check that.
Go back to the file, and sure enough yes it is mounts we want to talk to mounts.
And this is where things are slightly different and this is where things get a bit out of hand when it comes to handling nested dictionaries, you really have to investigate the JSON output, okay?
So we're going to go for item in data mounts, well let's just pprint item and see what we come up with.
Well, we get collected, num not collected, and number collected.
And where did those come from?
Let's go back to the file.
So we have collected here, that's the first key, okay?
We're going to scroll all the way to the bottom, and then we have the next key, number collected and number not collected.
And that's exactly what popped up here.
We just wanted the first key for mounts.
But we want to drill into the rest of this stuff.
We want to get further into mounts.
And this is where we further talk to collected.
So you can see we are now going to start staggering our way down, almost like a staircase.
So now we're going to go for item in data mounts, and we know that's collected, okay, well what's going to happen now?
I'll give you a clue so we don't flood the screen.
Printing item is going to print all of this stuff here, and we don't want to flood our screen yet again, okay?
What we do want however, is something in every single one of these dictionaries, so in this list, we have this dictionary, which comes down to here for the Albino Drake.
And underneath that we have this dictionary for the Amber Scorpion which ends here.
And we want the name; each one of these little dictionaries has a name key.
So we call 'em that.
So we've drilled down now with our code to mounts and collected, and now we're going to start talking to these keys here.
So, we're not going to pprint item, we're actually going to pprint the items, but only the name tag.
So we've got for item in data mounts, we're drilling down from the top level to collected, and then pprint the item with just the name tag.
And watch this.
There we have all of those mounts from the game, in the order that they are in those dictionaries, and that's all we got, which is exactly what we wanted, okay?
So we go back in here, you can see we got that name, Albino Drake, Amber Scorpion, Argent Hippogryph.
And so on.
All the way down here.
And it's really, really cool.
So that's how we've just worked our way down and passed our way down through the steps of the nested dictionary.
So what can we so with this?
Let's make it interesting.
Hop into this, now what's something that differs between these mounts?
Well this mount here, the Albino Drake is a flying mount because it's set to True.
But the Scorpion is not flying.
The flying is set to False.
And this is a boolean operator.
It's just True or False.
Not a string, if you considered it a string it would have had the quotes around it.
This is completely boolean, is aquatic and ground and everything.
How 'about we just try and get the flying mounts, the ones that are capable of flying through the air?
Let's narrow it down, do something a little more usable.
So we're going to create a list first that we're going to throw these mounts in, so is flying, just create an empty list, alright?
Now we're going to use the same code for mount in data mounts, we're going to drill down to the next level again, collected, well what do we want to do?
We're going to say if mount is flying, we're going to do something.
Now note that we don't actually have to check for the truthiness so to speak, we don't have to do if mount is flying equals equals True, because Python assumes True from default.
And it is in a string so we don't have to try and match it with a string called True.
We can just go if mount is flying, if it is, then we're just going to append it.
So is flying.append mount, alright?
And that's it.
So what this has stored now, it's actually stored this entire dictionary.
Each one of these dictionaries is now stored in the is flying.
What we can now do is talk to this thing, we can go Len is flying, we got 65 entries there.
Let's compare that with here.
We know our collected has 204.
I've collected 204 of these mounts in the game.
But we only had 65 stored in here, 65 of these dictionaries stored.
So we know this worked.
We know that it took just the flying mounts, the ones that are capable of flying.
So what we can do now, just to show you, we can go for i in is flying, print, we'll just keep pretty print just in case, pprint I, and you'll see we actually got all of that data, didn't we?
So we got the entire dictionary for each one of those flying mounts.
So again I'm sure you can guess, if we want just the data that we want, which is the name, we can go for I in is flying, pprint I, and then we use the tag again that we want to take, name, and there we go.
So our list has changed yet again, we no longer have things like that Scorpion, it goes straight onto the next flying mount that I've collected.
And that's it, so this is really really cool.
This is the best part about JSON.
It's got all the data there, it's just really hard to get to sometimes, so using this method you should be able to drill down through all of the sub dictionary or the nested dictionaries and find the data you want, and then you can

Okay, so that was JSON.
I hope you enjoyed it.
I know it seemed a bit basic, but the reality is that's most of the work with JSON that you'll end up doing, right, just passing the code, passing the output to find the tags that you need and pulling it out really.
So let's recap everything we did.
For the first bit of code, it was just pulling down and viewing the JSON files.
We just had a sort of overview of what it looked like, the format and what we were looking for and so on.
So we begin by putting JSON, okay.
Then we pull down the file using requests.
Then we used json.loads.
And this essentially decodes the JSON code that is, or the JSON output that's in the odd.text.
Okay so the request pulls down the JSON output and then json.loads decodes it, makes it readable for our code, okay.
Then we just handle the data similar to a dictionary, okay, and we saw that if we would adjust at the top level for the output that we had, if we had just run standard four loop iteration over the data, we got spammed with a lot of data, okay.
Now that was because we were just doing a standard print, yeah, so it didn't format it at all.
Now that's why we were importing it, the top there you can see from pprint import pprint and that is pretty print.
And that formats our JSON decoded output in a really nice way.
Pretty much the way that you would expect to see it if you had an in-browser JSON decoder or JSON viewer.
And this is a nice way of presenting it on our Python command line.
Then we had a look at the JSON nested dictionaries.
This is where things get a little more complicated and you can definitely see that with the data that we had.
So when we looked at our dataset, we saw that there was a mouse key sitting there, right.
So to iterate over that, we run four item in data mounts and then we p print the item.
And that got our next level of keys.
And specifically we then had a collected key which then underneath that had another dictionary and list below.
Okay, so we stared working our way down through the hierarchy of dictionaries, through the nested dictionaries.
Next from there we found we wanted to look for the specific names of the mounts and that's where we used the name key.
We specified that in the tag, okay.
And that p printed a really nice list of names for us.
Then we challenged ourselves to sit there and go okay out of all the mounts that we had collected, let's just print out the ones that are considered flying, okay.
And that's what this bit of code here does.
We have an empty list called IsFlying and then it goes through, checks to see if IsFlying is True, and remembering Python, you don't necessarily need to keep saying if something is true.
You can just say if something, and that is True in itself.
So if mount IsFlying, then we add the mount data, all of it, we add all of that mount data to the IsFlying list.
And then we can just iterate over that as we so pleased.
So this is the JSON data example that we've just gone over just as a little reminder for you.
It was quite in depth.
It just dug deep a little bit, okay.
You could see now everything if you've forgotten already, everything sort of makes sense.
So now it's your turn.
So what next?
Well for this one, there is actually something very specific I'd like you to try.
You're welcome to do whatever you want with JSON obviously, but I think a really good challenge for you at this point would be to go to our Code Challenge platform and just look at this code challenge on Code Challenger Number 16.
Query your favorite API, okay.
Now in this challenge we ask you to go out onto the internet to your favorite API and just do anything, okay.
Do any sort of a pull as long as you're querying the API.
But as we've discussed, APIs tend to return a lot of JSON data.
So a specific one you could try rather than going through all of these looking for one that might interest you is the OMDB API.
If you click this OMDB API link from our Challenge 16, you end up on this website here, OMDB Open Movie Database, okay.
And it actually returns JSON data and you can do some examples here and see what that will look like for you.
So if you want something to challenge yourself for day three, go through this here, OMDB, query the API, and see what you can do with the return data.
Just play with it, manipulate it, just like we did with our other code.
Maybe turn it into an app of some sort.
Just whatever you have time for and whatever you're willing to try.
And other than that, keep calm and

Hello, this is Michael Kennedy and I will be your guide for Day 10, 11 and 12.
And during these three days you're going to work with JSON APIs and in particular with a couple of search-based APIs.
But of course, what you'll learn here you'll be able to apply to pretty much any JSON API.
APIs are really important.
These are the way that you reach out and you add superpowers to your code, to your application.
You might write an application that has some data in the database, it's got some information that users have input, but maybe you want to add weather information, or integrate with Github or talk to Twitter.
All of these use APIs.
And so we're going to look at the foundation of APIs in this challenge, which is basically HTTP and JSON.
And we're going to do that from Python, of course.
So we're going to work with two services.
One service I'm going to demonstrate how we write code against it, and then I'll give you another service for your code challenge that you can play with, and if you don't like either of those services for your code challenge, feel free to just go find another one.
You'll probably have enough information to do something pretty interesting after this series of videos.
So let's go look at this movie db search service.

Over here at movie_service by talkpython.fm we have a service that has, I guess probably a couple thousand movies.
Notice down here this does not have all the movie data that you might want, it only has some of the movies, but it's more than enough for us to play with.
There's three things that this service will let you do.
It will let you basically search by keyword, so we come over here and call API search, and here we're searching for run.
And firefox likes to give you this pretty view here, let's go in and just look at the raw, so you can know what you're getting.
So here, the keyword is run, and we've got some hits.
And hits were for things like 'The Rundown', and 'Runaway Bride', and 'Bladerunner' and 'Chicken Run', and all these movies here.
So we'll be able to put whatever we want in the here, if we wanted away, boom.
'Flushed Away' is the first result.
It apparently has to do with sewers and rats and frogs.
Anyway, we can search for these movies using the service, and what we get back is this format here called JSON, which is the JavaScript Object Notation.
This is probably the most popular format for data exchange over these HTP services.
So it's great we can do it here, but we also want to do this in a slightly more structured way.
So we're going to look at two ways.
One using a tool specifically for accessing and working with APIs, cause it is pretty simple to go up here and type enter but what if our interaction required us to change the headers?
If our interaction required us to do an HTTP POST rather than a GET, which is what you normally do in browsers.
That gets tricky.
So we're going to see a tool that helps us explore the full spectrum of APIs, and then we're going to interact with it from Python.

To truly work with APIs, you need to be able to interact with the entire spectrum of the HTTP protocol, and browsers, while they technically do that, they don't let you as a consumer, as a user of the browser, really easily do that.
So we're going to look at this tool called Postman.
And I've already installed it, you can see it works on all the OSs, it's easy to install.
Let me come over here, and if we wanted to go to that same URL we had before, we could come over here and enter this, and hit Go.
And you see here's the same JSON that we got.
Here's the status code, what it means, okay versus other things, how long it took, how much data there was, what headers were sent back.
That's interesting.
We could even come over here and we could pass additional data, like user id is 72, or something like that.
Now that's not going to have any effect on this service, but it could, right?
We could even come over here and say, we're going to set the accept type, notice the auto-completion.
And accept application, let's just say JSON.
Right?
So be a little more explicit, this is the same thing we're getting.
We come over here and do a PUT and POST.
So as you interact with APIs, here and in other challenges, I encourage you to check out Postman.
It's free, and it's really nice, and you can even create an account, notice mine is nsync, and save these so you can sort of set up a structured way to interact with things, and go back and forth, save for testing.

We've seen how to poke at our API with just doing a GET request.
We've seen Postman lets us explore better but if we want to access it from code, we need another way.
Now, Python has a built in mechanism for accessing HTTP APIs but it's not the prettiest thing in the world.
So a guy named Kenneth Reitz created this thing called Requests.
Maybe you've heard of it.
It's definitely one of the most popular packages, period.
And to use it is really, really simple.
We're going to use it in just a moment.
If you want to get a URL, you just say GET.
Here you can even pass the auth, check the status code, check the encoding, check the text, and even convert the JSON back to Python dictionaries.
So this is a very popular library.
It's definitely the most-used for accessing HTTP services or Python directly.
So, unless for some reason, you are very adverse to having a dependency on an external package, this is the way, this is the way we're going to do it in Python.
So, next up, let's get started and actually write some code.

So it's time to get started writing our search app that's going to talk to the movie search API.
Now over here in our repository, I've got Day 10 to 12 selected and I've opened up the terminal here.
We want to open this in PyCharm.
You don't have to use PyCharm, but I'm going to open in in PyCharm.
I'm going to go through a couple of steps.
Whenever you depend upon external packages, it's a good idea to have a virtual environment.
In order to create one of those over here, let's call this app, let's make a folder really quick.
So call this Movie Search.
And go in here.
Then, I'm going to create a virtual environment here.
If we call it .env or venv, then PyCharm will automatically detect it and use it, so might as well call it that.
Now open this in PyCharm, can't seen anything because dot creates hidden files on Mac and Linux, but we're going to open this on Mac and Linux, you have to say file, open, directory and Mac, you can just drop it here.
Let's go ahead and tell PyCharm about about git, so everything I create is going to be put into the git repository for you.
And that'll be a little easier here, go ahead and also tell it to ignore this directory.
Maybe someday it'll do that on it's own.
Now I want to create two files, I don't like to cram everything into one Python file.
When I'm creating an application I want to partition it into pieces so let's go over here and create two things.
We're going to create a program and we're going to create an API.
And go ahead and tell PyCharm it can add stuff to git.
Okay so what we're going to do is we're going to write the code to access the service in the API file and we're going to work with it from the user interaction perspective and orchestrate it from over here.
So let's define a quick method over here, main.
And we'll just have pass.
Now we want to follow the convention for running this program if and only if it's the target of execution.
And in PyCharm, the way that, or, in Python, the way that you do that is you use this if dunder name equals dunder main, then that means it's supposed to run and I'll just call this method here.
This is a little bit set up and ready to go.
We're going to use something over in the API so we're going to create a method here that's going to take a string and it's going to return some search results.
So create.
And say find movie by title, let's say, because that's what we're actually searching for here.
And the keyword, and it's going to do some stuff and return some movies, that's going to be great.
How do we do that?
Well first we have to work with requests.
So we'll come over here and say import requests.
Now let me go ahead and run this, it's not going to work out so well, we're going to right click and create a run configuration in PyCharm, or you just type Python 3, you know, program.py.
That didn't actually import it so nothing actually happened.
But let's do, if we come over here and say import api as we're going to need to, now we try that again, boom, there is no requests.
So of course we have to install requests.
Now if you look in our terminal, notice we have the virtual environment activated here, so we could say, see what's in there.
And there's nothing really in there.
So one of the conventions people like to follow is when they have dependencies they'll have a file called requirements.txt.
And in there, they'll put library names like requests.
requests, like this.
And notice, PyCharm already knows, hey, you need to say pip install requests, or you can just click this and that'll do it for you.
If you wait for a second down here and then we do the pip list again, there should be more stuff including requests, our program should now run.
Ta-dah, okay.
So we have just the basic structure of this thing hanging together, right?
We've installed requests, we've created the program, and the api separation, we've got our requirements, and it's all kind of hanging together.
Next up, we actually have to write our method here, about how we find movies.

So we have the basic structure of our application written here, but now we want to actually go, when somebody calls this function, we want to go and download this result from our search service, so, recall over here in our movie search service, we put something like this.
We say api/search/{keyword}, and then the search will happen.
So let's go over here and first create the URL.
And the URL we're going to use f-strings.
This is a Python 3.6 feature, if you don't have Python 3.6 or above, then you're going to need to do the older style format.
So we'll say like this, and we want to replace that little part with the variable keyword, so in Python 3.6 with these f-strings, you can say, {keyword}.
Notice the autocomplete.
So, let's begin just by printing out URL, and just make sure that we're on the right path.
So we're going to come over here, and we're going to call result...
Spelling is hard.
Results equalsapi., notice there's our little thing, and let's just, for now, say this is going to be runner.
Going to search for runner.
So if we run this, that looks pretty good, we can click it and test.
Okay, looks like we got the maze runner and runner runner, that's a lot of running.
And so this is working.
Now instead of printing out this, let's actually use it.
So we'll say response equals requests, which we've already installed, get, and we'll pass the URL.
We could go ahead and just work with this.
We could say response.text, unless that failed, unless the, say, wireless is down or something.
So we need to be careful and say, I want to make sure there was no error.
There's a status code, you could check it.
But requests has this cool method called raise_for_status.
So if anything went wrong for whatever reason, you'll get an error, otherwise you can just keep going.
So now we have, and we can print out, what we got back here, and then again.
And notice, there's all the results coming back, well, at least all the ones the server would give us.
That's pretty cool.
Now, we actually, not to like...
We don't want to work with strings, we want to work with data.
So the next thing that we need to do is convert this text into Python dictionaries from the JSON source.

So we've seen that we've downloaded the data from our search service and we have created the right URL that works, we already tested that in our browser and printed out the text.
So the next thing we need to do is convert the text from JSON format into Python dictionaries.
Now we could use the json library in Python but request has a little goody for us here so we can say results equals response.json, that's all we need, it's converted to JSON so we could print out, it's converted to Python dictionary so we could print out the type of results and we could also print out the results themselves so if we do that, notice we get a dictionary and here we have the keyword and then we have the hits, which is a list of objects that have titles.
Let's try to print out, let's try to return those and then at the program level, print them out.
So, get rid of that and we'll just return results.get hits.
Right, we don't care about the extra data, we just want the results.
And then over here, we're going to get those results back and we can say, for are in results print, let's just print out the title.
Title is, let's, keep going with the f-strings, huh?
Say are .get, now notice we have to treat this as a dictionary, we'll prove this in a moment, like that.
Okay, let's run this and see what we get.
Man, look at that, title.
Blade Runner, Maze Runner, Kite Runner, something else, Logan's Run and so on.
Awesome and we could even do a little bit better, we could say, print, there are length of results, movies found, something like that.
There are six movies found, okay.
Excellent, you might want to do a little test movie, 1 movie, 2 movies, 0 movies, things like that but we're going to just keep it always plural here.
Alright so this is pretty good but notice this, I'm not loving this at all and the more we have to write out there, we could say something like that and IMDB score, let's say we're going to write that out so we'll say, are .get what, what do you put here?
I have no idea.
It's completely annoying that you get no help about this, we could go look this up but let's go and actually improve that in the next video.

Let's make two quick improvements before we wrap up this application.
First, we're always searching for runner.
How exciting is it to always run this program and just get the same results?
Let's actually make this a thing that the users can input.
So we'll begin by creating a variable.
We could just go and type a variable name, assign the value and put it over here, and use PyCharm to refactor.
Say I want a variable called keyword.
That's pretty cool.
Make sure it still runs, it does.
But let's actually go and say input.
We'll ask the user for input, and we'll give them a prompt.
We'll say keyword of title search for movie, or something like this.
So now when we run it down here I can search for away and we get The Runaways.
I can search for runner and we should get Bladerunner, and we can search for fight, Fight Club, whatever.
Whatever we want to search for, we can now do that.
This is already really awesome, but notice the API is, I'm going to call it crummy.
So we're going to fix that with one more thing.
We're going to use this thing called collections.namedtuple So down here I'll say I'm going to define this type that represents a movie.
It's going to be a collection.namedtuple, and the way it works is you say the name that it thinks its own name is, and then you say all the fields that go in there.
Now this turns out to be quite a pain, so let's go back over here and see what we get.
IMDB score, we have a rating, an we have all these things.
So this is the typical value here, it's going to look like this So in order for this to work well, we have to have all these values listed in series over here.
So, I'm just going to type this in, then I'll speed it up.
Phew, there I've typed them all in and I'll maybe do a few line breaks here just so it fits on the screen.
I just typed in the top level elements here.
So what we can do, is we can actually instead of just churning JSON results, we can actually go over them and return these types, and we'll see why that's a benefit in just a second.
So we'll say this, we'll say for our end results, you know, the hits, maybe H, I don't know.
We're going to need a list, going to say movies, that's a list.
I'm going to put this in here, so we'll say movies.append and we need to append one of these movies.
So we can create a new one and we have to type all of these values in here.
We have to say IMDB code equals R.get IMDB code.
Title equals R.get title, and so on.
Or, there's a way to unpack a dictionary, which is what this is, back into keyword arguments with the names being the keys just like that.
And, the way you do that is you say **, the dictionary name.
So it's as if you said IMDB code equals whatever value it has for IMDB code.
Title equals whatever value it has for title.
It's just a super short, simple way to do that.
So now if we return movies, up here is not going to work the same, but now we can just say R.title, and things like that.
HasScore, take that little bit away.
HasScore, let's say,r., what did we call it?
IMDB Score, that, now let's try that.
Now we are going to search for runner, stick with that.
Boom!
Look how cool that is.
Now, we're still not where we quite really wanted to be.
If I hit dot, we get not, so as in not helpful!
One final trick of what we're going to do, let's go over here and we can use what's called type hint in Python.
Python is a dynamic language, it has no syntactical typing, but you can give the editors hints, and I can say this is actually a string, so you say colon type name and the return value is a list which is actually defined in a typings module, so got to import that.
So we want typings.List.
Notice the, let's put at the top here.
And it's going to be a list of movies like this, okay.
So with a little bit of a hint, I can come over here and now I type r., look at that, director, duration, IMDB code, IMDB score.
Let's just add one more.
With code r.code, now that's the way we like to write.
All the help we can get.
Let's search one more.
Anything on computer, of course!
Hackers with code TT whatever HasScore 6.2.
Awesome, there you have it.
This is how we can consume the movie API.
When you've broken apart into our, our sort of user interaction bit of code here, our API access code here.
We use requests to actually get the data and convert it to JSON and we used a namedtuple to help package it up into something that makes more sense to our program, as well as adding a little type hints, so that the editor actually leverages that help we gave it.

Let's review the concepts in our search API.
We started out by consuming the MovieDb service, at movie_service.talkpython.fm.
We saw that we could get postmen, which is a really nice way to explore and interact with the API, the full spectrum of HTTP APIs.
And we use requests in Python to actually do the direct interaction in code.
When we zoom down to the code level, you can see that we start by importing requests.
We're going to need to use this library, so we got to import it, and remember we had to install it with pip as well, or make it part of the requirements and PyCharm helped us, but we could have also said pip install -r requirements.txt and achieve the same effect.
Then we're going to go and actually download the data.
Take the URL and say, request.get URL.
Like, this is a response, this is what we work with for the rest of the time.
We want to make sure everything worked, so we're going to check for success with raise for status, and then we want to take that string of JSON and turn it into a Python dictionary.
So we do that by calling .json.
Notice if the format is not actually JSON, this will crash.
But it was JSON in our example, so it completely worked.
After this it's plain Python, there's no more HTTP service involved, we just have a Python dictionary, and you work with it like you do regular data in Python

Now you've seen me build an application using requests to consume a JSON API.
It's your turn to consume a different API.
So drop over here in GitHub, and we're going to go through the various things and the read me here for what you do for this particular day.
We're going to start out by basically just making the skeleton, create a virtual environment, create a program .py and an api.py, and just make sure that you can import one from the other, and we're going to install Postman.
So make sure you install the Postman application.
Most of what today was was just watching the corresponding videos.
So get everything setup and get ready for the next day.

Second day of this section is going to actually be to work with an entirely different API, but another one around the podcast just to keep it simple and stable.
So what your going to do is we're going to interact with this search.talkpython.fm Now if you're over here and you're on Talk Python, and you try to search, you could say "I would like to search for let's say, a 100DaysOfCode".
Pull that up you can see it went back and found a couple of elements, an episode and a transcript, and it did this in a ridiculously fast period of time.
But there's actually an API underneath here there's JSON API, and you can write queries you can search yourself right?
it's open, and searchable for you and it's exactly like what you've been working with.
So let's go over here and play with this a little bit.
Click on this, and see the raw data, This is what you get back if you search for tests, so whatever you do is you put the little query string pieces there, so we'll put a100DaysOfCode, and it's going to come back Firefox tries to make that response pretty, but it's going to come back looking something like this.
Alright, so this is the API we're going to work with.
So the first thing you're donna do is explore the API imposement, and your goal for the day is to create a program that does the following: It gets a search word from the user, It calls the search service using that API we just looked at, it makes sure the response from the server was okay so request.
response.raise_for_status I believe, And then you want to return just a basic dictionary of the results, and use that to list out the titles.
So it should look something like this on Day 2.
Run your program, it has a little header if you want, ask the user for some search terms and it says "Boom.
These are the titles."

The last day is adding polish.
We're going to use a namedtuple.
Now here's an example from our movie one.
You want to create a corresponding one for a search result.
The other thing we're going to do, this is really, really simple and slick, is, if we look over here you'll see that there is a URL that you can get to for every result.
What you're going to do is you're going to write a little bit of code that asks the user which one of these would you like to view in your web browser.
Maybe give 'em a number, and they can put 3, and so then you're goal is to write a little bit of code that will open the web browser and a new window at that URL, and literally those two lines of code is all it takes across platform.

Okay, everyone welcome back.
This is web scraping with BeautifulSoup 4.
I'm Julian Sequeira, again, and just a disclaimer, this has nothing to do with dinner.
This has everything to do with web scraping.
If you're hungry, go get something to eat.
If not, crack on, because what we're going to do now is this is going to be a very quick module.
We're going to run through pulling down a webpage with requests, not in any detail because we've done that before.
Then we're going to parse that webpage information with BeautifulSoup 4.
You can do some really cool stuff, so I'm very excited to show you this one.
Just set up your environment and the next video, and then we'll get straight to some code.

This is the readme file for beautifulsoup4, for Day 46 to 48 on web stripping.
Now, day N the first day you're going to be working on this course, I would like you be watching the video on setting up the environment, getting a quick overview of beautifulsoup4.
This is if you have no familiarity on what it is and how it works, and then build your first Beautiful Soup 4 scrapper.
It's actually not too much work, but there is a bit of theory there, with the overview, and you should be able to get it up and running, and then give it a crack yourself, okay?
Watch the videos first, because if you're not familiar with it, it does help to watch it start to finish, okay?
Pull your first site, use the example site in the video, or if you really want to challenge yourself, grab another one.
Day 2, what I'd like you to do is watch this video on requests best practice, okay?
This is covering a little thing that people tend to do with requests that is actually the wrong way to do it, and we discuss the best practice for actually doing it, I won't give it away now.
Then, what I'd like you to do is follow along with this video, detailed Beautiful Soup 4 scrapping and searching.
This will actually go through how to do some targeted searching, so to speak, of the data that you pull down and scrape, okay?
It can be a bit tricky and a bit frustrating to find exactly what you want, but stick with it and you'll get there in the end.
And Day 3, as usual, it's your turn.
So, you've figured out how to scrape a website, you can pull the data that you want, so now I'd like you to actually do something with it, okay?
So store it database, display it in something like a Flask app or a GUI, automate it by emailing it, do whatever you can think of, right?
So, come up with something and do that.
If you can't think of anything, you could try this one.
I've added an extra option here for you to try, which is to find a site that looks complex.
Think of something that maybe has Flash, or whatever other animations on the website.
Pinpoint a data sample, so just something on the website you think that could be interesting, and then see if you can extract it using Beautiful Soup 4, okay?
So that's it, give anything like that a try.
Day 3 is your freestyle, free-for-all, do whatever you want, and just have a good play.
And, move on to the videos and get started.

Okay, we just have a little bit of setup to do for this one.
First things first, as always, let's create our virtual environment.
Same thing, venv.
Once that's installed, we need to install Beautiful Soup 4, of course, but we also need to pip install requests.
So let's do that quickly.
We'll just activate, the virtual environment here.
Okay, now we can do pip install requests, and once that's done, we can then do pip install bs4 You can probably type in beautifulsoup4, but bs4 is fine, and that's it.
We have that installed.
The last thing I'd like you to do is just create a file called scraper.py, and throw that into your project directory.
When you run it, it will look something like this, scraper.py, just here.
Okay, and that's it.
So once you've got all that set up, launch your file and let's move on to do some coding.

I just thought I'd give a quick overview for anyone who hasn't dealt with Beautiful Soup 4 before.
So if you haven't, feel free to keep watching, but if you have, skip on over because I'm just going to be repeating myself.
Now, Beautiful Soup 4 allows you to parse web pages.
Okay, we've all dealt with requests by now and we know that we're using requests, we can pull down the code behind their web page, right?
And we can then use Beautiful Soup 4 to parse that data.
All right, I'll tell you what I mean.
So, let's view the page source for our PyBites Code Challenges page.
And here you'll see all of your HTML.
Now, if I wanted specifically to get all of our code challenge names and just put them in a list to make in some sort of an email or whatever application I can think of, right.
Well, how am I going to do that?
If you go into the page source, you need to find, the first thing you need to do is you need to find that data in the code and here it is.
It's an unordered list with ID of article list, and a class of article list.
Okay, and then all of our different challenge headers are stored in list elements, okay?
Now, with that information in hand, we can then use Beautiful Soup 4 to search this page, remembering that requests will pretty much pull down this page looking like this, and Beautiful Soup 4 will parse that, and we can then tell it what to look for.
And now you can start thinking, imagining the cool things you can do with this.
So, we can skip all of this junk up here, all of this code that we don't care about, and drill straight down to the list that we do care about.
And you can use this on any site that you can think of.
You can search by all sorts of different criteria.
And we're going to show that in the next video.
So, get excited, because this is really, really fun stuff.

Time for some code.
We need to think about what we're going to do first.
The first thing we need to do is actually pull down our website, and what are we going to use for that?
We're going to use requests because we pip installed it, didn't we, that was a bit of a dead giveaway.
We're also going to import bs4.
That's it.
Let's specify the actual URL that we're going to be dealing with here, I'm just going to copy and paste it.
This is the URL of our Pybites projects page.
Looking here, we have out PyBites Code Challenges.
What we're going to do with this one is bring down all of these PyBites projects headers, so, our 100 days.
Our 100 Days Of Code, our 100 Days Of Django.
These different headers.
We're going to pull all of those down and we're just going to use that as a nice, simple example for this script.
Let's start off our code with the standard dum dum.
Now, what are we going to do?
The first thing, obviously, as I said, is to pull down the website, so let's create a function for that.
def pull_site, nice and creative.
We're going to use requests, so I'm going to do this really quickly just so that we can get a move on because we've dealt with requests before.
So, requests.get URL.
That will get the page and store it in the raw site page object.
So, raw site page .raise_for_status Now, this is just to make sure that it works.
If it doesn't work, we'll get an error.
And then, we're just going to return, raw site page.
Nice and easy.
Let's just assign that to something down here called site.
This will assign the raw site page to a variable called site.
Now, we'll record a video after this explaining why this is not a good idea, what we're doing.
But, for now, as just a nice little explainer, this will do.
Let's create another function.
This function we're going to call scrape.
It's going to be used against our site object.
We need to think ahead a little bit.
I'm going to think ahead by putting this list here.
If you think about our page, as we pull this data out of the page, these headers, we need to store them somewhere, don't we?
We must store them in a header list.
We create the empty list.
Now we get to the Beautiful Soup 4 stuff, and this is really, really easy, so don't worry if you don't wrap your head around it.
But it's only a couple of lines of code, which is why we all love Python, right?
We're going to create a soup object, and it's not a bowl of soup, it's just a normal object.
bs4.BeautifulSoup4, .BeautifulSoup, sorry.
We're going to take the text of the site.
So, site.text, we're going to take that.
That's going to be called against our Beautiful Soup 4.
We're going to use the Beautiful Soup 4 HTML parser in order to get our sort of HTML code nicely into this soup object.
Once we do that we have our soup object and HTML header list.
This is going to be a list of our HTML headers.
You can see what we're doing.
This is already really, really simple.
We've taken Beautiful Soup 4 and we've told it to get the text of the site using the HTML parser because site is going to be a HTML document, pretty much, right?
We're going to store that in soup.
We're creating an object here that is going to be...
I'll show you.
HTML header list equals soup.select.
What are we selecting?
The select option here for soup, it allows us to pull down exactly what we need.
We get to select something out of the HTML source.
Let's look at the HTML source again.
We'll go view page source.
We'll get to these headers.
The first header is called zero.PyBites apps.
We find that on the page, it's going to be in a nice little h3 class.
What's unique about it, and this is where you really have to get thinking and analyzing this page, the thing that's unique about all of our headers here, so, here's zero, here's number one down here, but they all have the project header CSS class.
Playing with bs4 does need some tinkering.
Occasionally, you'll find someone will have reused the same CSS class somewhere else in the same page, so when you select it you'll get more than just what you wanted.
But in this case, I know, because this is our site, we've only used project header against these headers that we want to see, these ones here.
We're going to select everything that has the project header class.
Let's copy that.
We'll go down here, this is what we're selecting.
We have to put the dot because it is a CSS class.
And let's see it.
All this has done is we've created the soup object of the site using the HTML parser, and then we've selected, within our soup object, everything with the CSS class project header.
We've stored those, we've stored everything that it finds into HTML header list.
Easy peasy.
Now all we need to do is iterate over this and store the information that we need into this header list.
We'll do that.
We'll go, for headers in HTML header_list we're going to go header_list.append, as we know how to do that.
headers.get text.
We're saying, just get the text.
Just to show you what that means.
Everything in here, in the class project header, we actually got the whole h3 tag.
That soup select pulled the whole tag, but all we wanted was the text.
That's what the get text option does, that's what this get text does right here.
It strips out the tags, the HTML code, and gets you just the plain string text that we asked for.
That's it.
We want to see these headers, so let's just quickly create another for loop here.
For headers in header_list print, ooo, what did I do there?
Print headers.
And that's it.
Save that, and what this will now allow us to do is print out the headers that we have stored in header list in this for loop here.
Let's have a look at that and see what it looks like.
Silly me, I've forgotten one thing, we actually have to call our scrape function.
So now we will write scrape site.
Simple.
Save that, and let's give it a crack.
I'll just move the screen to the right.
There's my command prompt.
Let's just run Python scraper.py.
Hit Enter.
And it worked, look at that.
There's that plain text that we asked for with the get text.
We got the first header, PyBites apps.
We got second header 100 Days Of Code, 100 Days Of Django, and so on, and so forth.
Now we have a list with just our headers.
This is really cool.
If you think about it, you could use this to create a newsletter.
You could use this to save your website's headers for who knows what, to print out in a list and stick on the wall as a trophy.
But this is the idea of Beautiful Soup 4.
You can get a website and you can just strip out all the tags and find the information you want, pull it out, and then, do things with it.
So, it's really, really cool.
The next video we're going to cover some more interesting...

Alright, one quick public service announcement regarding this code that we've just written.
This section here, the pulling of the site.
Not actually kosher to keep that in this script.
The reason for that is we don't want to submit a request to a website every time we want to scrape some data.
We might run a scraper like this at a different interval set to actually pulling the website.
The reason for that is, you think about it, not every site is going to update every few minutes.
Not every site is going to update every day.
So if you keep pinging that site with a request, you're going to very quickly spam them.
You might even get yourself blocked.
And you could use up their bandwidth limit.
There are certain websites that, you know, can only support a certain amount of hits per per minute, alright?
And if you keep doing that, you're going to make the website that you enjoy viewing so much pretty unhappy.
So the best practice here is to put all of this into a different script, run that on a cron job at a different interval or whatever other automated way you want to do that, and then using Beautiful Soup 4, point at the downloaded HTML file or at the page that you have pulled down from requests, alright.
Nice and easy.
It's actually much more pleasant for everyone to do it that way, and I would totally recommend doing it.
The only reason we've done it this way right now is just for demonstration purposes.
It's much easier.
But in production, definitely put your requests in a different script and use Beautiful Soup 4 to talk to a static file, not the actual URL, unless of course it's a one off thing.

Alrighty, now we want to actually do something interesting so let's go to our article page here on PyBites.
I want to do something like pull down every single article name that we have written.
Very, very daunting so the first thing we want to do is view the page source.
If you, just a quick tip, if you ever get stuck and you're not too sure what is what in this page here, you can click on inspect.
As you scroll down through inspect through the code the HTML code within the inspect, you'll actually be able to see what is which part of the page.
We need to lower this down here, lower this down here.
And here is our HTML code that runs the page.
As we hover down through these little tabs, these little arrows here, these drop downs, you'll see parts of the page get highlighted.
That's how you know which code is actioning which part of the page.
We know this is our main here.
We find the main tag and sure enough that's highlighted.
Now we can click on article here.
You can see that's highlighted the section in the middle that we want, right?
We can keep drilling down.
Here is the unordered list, ul, that is our list of article names.
Then here are all the list elements, the li.
Open them up and there is our href link and the actual name of the article.
That's what we want to pull down.
We can look at that quite simply here.
But in case this was a much more complex page such as news websites and game websites and whatever, you might want to use that inspect methodology.
Alright, now that we know we want to get the unordered list, let's do some playing in the Python shell.
I've started off by importing Beautiful Soup 4 and requests.
I've already done the requests start get of our articles page.
We've done the raise_for_status to make sure it worked.
Now let's create our soup object.
So soup equals bs4.BeautifulSoup, oops, don't need the capital O.
site.text.
So this is very similar to our other script.
And HTML parser.
Alright, and with that done what can we do?
We can search.
We don't have to use that select object.
That was very specific and that was very CSS oriented.
But now we're going to look at tags, alright?
Back on our webpage, how's this for tricky?
The individual links don't have specific CSS classes.
Uh-oh, so how are we going to, how are we going to find them?
Alright, we're going to have to do some searching right?
How about we search for the unordered list.
We could just do soup.tagname, isn't that cool?
Now you just have to specify the tag that you want to find.
You don't even have to put anything around it.
It doesn't need to be in brackets, nothing special.
Soup.ul.
Hang on a minute, what do we get?
Now look at this, we got an unordered list.
Ah but, we got this unordered list.
We got the actual menu bar on the left.
We didn't get this one.
Now why is that?
That is because soup.ul, or soup.tag only returns the very first tag that it finds that matches.
If we look at that source code again you'll find we actually have another unordered list on the page.
That is our menu here, okay.
That's not going to work using soup.ul is not going to work, because we're only pulling this one here.
What do we need to do to find the next one?
Well we need to do a little bit more digging.
We could try the soup.find_all options.
Let's see what that returns.
So soup.find_all.
Then we specify the name of the tag that we want.
We're going to specify ul.
Let's hit enter and see what happens.
Look at that, we've got everything.
We got all of the article names.
Let's scroll up quickly here.
But no we also got the very first unordered list as well.
We got that same menu bar.
How do we strip that out?
Well, I suppose we could go through and do regex and all sorts of crazy stuff.
But Beautiful Soup actually let's you drill down a little bit further.
What makes our unordered list here unique?
On our page, on PyBites, it's the only unordered list that lives within our main tag.
Look up here, here's main.
That's the opening of main and we saw that closing of main down on the bottom.
And look, it's the only unordered list.
What we can do is we can do soup.main.ul.
You can see how far we can actually drill down.
Isn't this really cool?
We run that and what do we get?
Let's go back up to where we ran the command and from soup.main.ul, we got the unordered list and we got all of the list elements with the atags, the hrefs, and the actual plain text in the middle.
There we go, we've already gotten exactly what we want.
But we actually don't need all of these tags around the side.
We don't even need the URL.
How are we going to get around that?
Well let's see what we can do.
We don't actually need the unordered list, do we?
We don't need this whole UL tag, we only need the list elements and we know from looking at the code that these are the only list elements within the main tag.
Main tag ends here, there's no more list elements.
What if we do that same, find_all, but just on list.
Let's go soup.manin.find_all, because we're only going to search within the main element.
There we go, look at that.
It's not formatted as nicely as the other one that we saw, but it is the information that we want.
You've got your list elements, you've got a URL, and they you've got your title, and then it closes off.
Let's give ourselves some white space to make this a little easy to read.
What can we do, we want to store all of that.
Let's store that in a list called all_li.
We're getting all of the li options.
We do soup.main.find_all li.
Now all of that is stored in all_li.
Now how cool is this?
The text in here in each one of these list elements is actually the stream, right?
You know that, we discussed it in the previous video.
What we can do, we can do for title in all_li I guess for items for each one of these items.
I shouldn't really use the word title.
Let's actually change that to be for item in all_li.
It's this we're going for each one of these just so you can follow along.
What do we want?
Well we want just the subject line.
We just want this string, we just want the text.
How do we specify that?
We go print, now this is going to be really easy and I'll tell you what, it's just crazy sometimes how simple they make it.
item.string, and that is it.
Can you believe it?
To strip out all of this, all the HTML and all we want is that, item.string.
You ready?
How cool is that?
We've just gone through I think it's like 100 objects, 100 articles that we've written.
We've just printed out all the names.
This would be so cool to store in a database and check back on it from time to time.
Email it out, keep it saved somewhere.
I love this sort of stuff.
There you go, we've look at quite a few things here.
We've looked at find_all.
We've looked at using just a tag name, and we've looked at using a sort of nested tag name.
So you can actually drill down through your HTML document which is super cool to help you find nested or children objects within your HTML.
That's Beautiful Soup 4.
That's pretty much it and it's absolutely wonderful.
There's obviously so much more to cover and that could be a course in itself.
The documentation is wonderful.
Definitely have a look through that if you get stuck.
But for most people this is pretty much the bread and butter of Beautiful Soup 4, so enjoy it, come up with cool stuff.
If you do make anything cool using bs4, let us know.
Send us an email or something like that.