Managing Python Dependencies Transcripts
Chapter: Setting Up Reproducible Environments & Application Deploys
Lecture: Separating Development and Production Dependencies
Login or
purchase this course
to watch this video and the rest of the course contents.
0:01
Let's discuss separating development and production dependencies for your programs. A common challenge with Python dependency management
0:11
in the real world is that your development and continuous integration environments
0:16
need additional dependencies that the production environment doesn't need. For example, your development environment might need
0:23
additional testing frameworks, debuggers or profiling tools that are not really necessary in production, and might actually slow things down.
0:31
The same would be true for you continuous integration environment. If you're running any kind of automated tests,
0:37
you probably want to include testing frameworks and all kinds of other supporting tools for that on your built server environment.
0:45
But again, with all of this, the goal is for the production environment to still run lean and mean; now the good news is that
0:53
there is actually a requirements files workflow that is well known and often used in the Python community that solves this problem.
1:00
The solution here is to create two different requirements files, one for development dependencies, and one for production dependencies,
1:10
typically, these would be called requirements-dev.txt for the development dependencies, and requirements.txt for the production dependencies.
1:21
There is a feature in pip that allows you to link up requirements files so you can make this workflow very convenient for developers.
1:28
They only need to remember to install the requirements-dev file and that will automatically pull in
1:34
and install all of the production dependencies as well, let's take a look at a real world example for this technique.
1:40
So I am looking at Love here which is a Python web application that was created by the fine folks at Yelp,
1:48
and you are going to see in a minute now that they are using the exact same pattern for managing their development in production dependencies.
1:56
So right now, I am in the root folder of this repository here and when I scroll down,
2:00
I'll find those two requirements files, so there is a requirements.txt and then there is also a requirements-dev.txt.
2:10
And this is exactly the split that I talked about earlier, so when you look at requirements.txt you can see that this includes
2:19
all of the production dependencies in their exact versions, and nicely enough, they actually put a little comment here to explain what is going on,
2:29
I think this is a really great use of the commenting feature that you can use in requirements files by the way.
2:34
So, these are just the production dependencies that would be necessary to run the application in a production environment,
2:39
now let's look at the other requirements files. So there is also this requirements-dev.txt and what this does,
2:48
it first of all pulls in the production requirements and then it installs a bunch of additional development dependencies,
2:57
so it pulls in a testing framework and the mark module, and some other modules that are useful during development
3:06
but not really on a production machine. And what's interesting here is that for some of the dependencies they actually pin them to specific version,
3:15
so every time you specify an exact version that is called pinning, and this is our version pinning and this is exactly what they do here.
3:22
But then for this other dependency called ipdb which is an enhanced version of the Python debugger,
3:29
they do not pin the dependency and this makes a lot of sense because the ipdb debugger would never actually run on an automated test server,
3:37
but it's more something that a developer would run on their local machine. So it might make sense to leave that dependency unpinned
3:44
so it will always upgrade to the latest version. Now, again you can see here that they put this little header explaining
3:50
what is going on in this file, and I think this is really helpful. Now these two requirements files cover the two usecases we discussed earlier.
3:57
So if a developer wants to set this up on a new machine, they would clone the repository and then run pip install -r requirements-dev.txt
4:06
and this brings in all of the development dependencies, but every time the application gets deployed onto a production system,
4:14
all that happens is a pip install -r requirements.txt so that only the production dependencies are brought in.
4:24
If you're building a web application this workflow is specifically supported by platforms like Heroku, so what Heroku does
4:31
when you deploy a web application to it, is it will look for a requirements.txt file
4:38
in the root of your source repository and try to install those dependencies, so if you follow the requirements.txt and requirements-dev.txt split,
4:48
it will only install the production dependencies and not the testing dependencies which is exactly what you want.
4:54
I like this workflow because it is simple but also highly flexible.