Managing Python Dependencies Transcripts
Chapter: Setting Up Reproducible Environments & Application Deploys
Lecture: Separating Development and Production Dependencies
0:01 Let's discuss separating development
0:04 and production dependencies for your programs.
0:07 A common challenge with Python dependency management
0:10 in the real world is that your development
0:13 and continuous integration environments
0:15 need additional dependencies that the production environment doesn't need.
0:19 For example, your development environment might need
0:22 additional testing frameworks, debuggers or profiling tools
0:25 that are not really necessary in production,
0:28 and might actually slow things down.
0:30 The same would be true for you continuous integration environment.
0:34 If you're running any kind of automated tests,
0:36 you probably want to include testing frameworks
0:39 and all kinds of other supporting tools for that on your built server environment.
0:44 But again, with all of this, the goal is for the production environment
0:47 to still run lean and mean; now the good news is that
0:52 there is actually a requirements files workflow that is well known
0:55 and often used in the Python community that solves this problem.
0:59 The solution here is to create two different requirements files,
1:04 one for development dependencies, and one for production dependencies,
1:09 typically, these would be called requirements-dev.txt
1:13 for the development dependencies,
1:17 and requirements.txt for the production dependencies.
1:20 There is a feature in pip that allows you to link up requirements files
1:23 so you can make this workflow very convenient for developers.
1:27 They only need to remember to install the requirements-dev file
1:31 and that will automatically pull in
1:33 and install all of the production dependencies as well,
1:36 let's take a look at a real world example for this technique.
1:39 So I am looking at Love here which is a Python web application
1:44 that was created by the fine folks at Yelp,
1:47 and you are going to see in a minute now that they are using
1:50 the exact same pattern for managing their development in production dependencies.
1:55 So right now, I am in the root folder of this repository here and when I scroll down,
1:59 I'll find those two requirements files, so there is a requirements.txt
2:05 and then there is also a requirements-dev.txt.
2:09 And this is exactly the split that I talked about earlier,
2:13 so when you look at requirements.txt you can see that this includes
2:18 all of the production dependencies in their exact versions,
2:21 and nicely enough, they actually put a little comment here
2:25 to explain what is going on,
2:28 I think this is a really great use of the commenting feature
2:30 that you can use in requirements files by the way.
2:33 So, these are just the production dependencies that would be necessary
2:36 to run the application in a production environment,
2:38 now let's look at the other requirements files.
2:41 So there is also this requirements-dev.txt and what this does,
2:47 it first of all pulls in the production requirements
2:52 and then it installs a bunch of additional development dependencies,
2:56 so it pulls in a testing framework and the mark module,
3:01 and some other modules that are useful during development
3:05 but not really on a production machine.
3:08 And what's interesting here is that for some of the dependencies
3:11 they actually pin them to specific version,
3:14 so every time you specify an exact version that is called pinning,
3:18 and this is our version pinning and this is exactly what they do here.
3:21 But then for this other dependency called ipdb
3:25 which is an enhanced version of the Python debugger,
3:28 they do not pin the dependency and this makes a lot of sense
3:31 because the ipdb debugger would never actually run on an automated test server,
3:36 but it's more something that a developer would run on their local machine.
3:39 So it might make sense to leave that dependency unpinned
3:43 so it will always upgrade to the latest version.
3:45 Now, again you can see here that they put this little header explaining
3:49 what is going on in this file, and I think this is really helpful.
3:52 Now these two requirements files cover the two usecases we discussed earlier.
3:56 So if a developer wants to set this up on a new machine,
3:59 they would clone the repository and then run pip install -r requirements-dev.txt
4:05 and this brings in all of the development dependencies,
4:09 but every time the application gets deployed onto a production system,
4:13 all that happens is a pip install -r requirements.txt
4:18 so that only the production dependencies are brought in.
4:23 If you're building a web application this workflow is specifically supported
4:27 by platforms like Heroku, so what Heroku does
4:30 when you deploy a web application to it,
4:34 is it will look for a requirements.txt file
4:37 in the root of your source repository and try to install those dependencies,
4:41 so if you follow the requirements.txt and requirements-dev.txt split,
4:47 it will only install the production dependencies
4:50 and not the testing dependencies which is exactly what you want.
4:53 I like this workflow because it is simple but also highly flexible.