Python for Entrepreneurs Transcripts
Chapter: Deploying to the cloud
Lecture: The initial configuration
0:01 Hopefully you took a look through the Ansible documentation, but I found that the best way to learn Ansible and how to do automated deployments
0:09 is just to dig in and start writing your first playbook. The first playbook is going to be one that we only run when we spin up a new server,
0:16 the base Ubuntu image that we selected on Digital Ocean is not harded against attackers who might want to try to lock into your box.
0:23 The main security mechanism that's in place from the start is just that you can only log in with an ssh key, but there are many more steps
0:29 that we need to take in order to protect this box, because it is live with a public IP address on the internet.
0:36 Shift back over into your development environment, and either go into the course demos or move into your own project
0:42 you can start writing the deployment playbooks for your project as we go along through this chapter. Or if you don't have your project set up just yet,
0:51 you can follow along through the course demo. Within the course demo's git repository, switch into chapter 15, deployment,
0:56 there will be another folder within it, blue yellow app deployment. This is the same project that Michael was working on the previous chapter,
1:05 where he set up email, we're now going to take this application and deploy it on our Digital Ocean server.
1:12 The first step is to create a deployment directory, that will store all the files Ansible needs to execute,
1:18 create that directory, and then switch into it. If you haven't already activated your virtual environment, go ahead and do that now.
1:23 You can see mine is activated here, I just have one called entre, and it's also using Python 3. The first thing we want to do, is install Ansible.
1:31 Now typically we would just say pip install Ansible, but since Python 3 support is being actively worked on, in the development branch,
1:38 I recommend installing the development branch rather than the current stable release. As soon as Ansible cuts the 2.3 version
1:46 you can then install Ansible via Ansible 2.3. The way that we install from git, is we instead use the git protocol
1:54 pointed to the repository that we want to install, in our case that's going to be github.com/ansible/ansible.
2:02 Now this is going to clone the git repository, and install it into our virtual environment, in my case, I've already installed Ansible,
2:14 wait for it to finish in your own environment, and then we can proceed. Now that Ansible is installed, when you do pip freeze
2:21 it'll show us that Ansible 2.3.0 is installed, that's the minimum version that we're going to want.
2:27 With Ansible installed, we can now start to write our files. The first file that we need is a hosts file,
2:33 this is where we tell Ansible where our servers are located; open up hosts, file name hosts, under the deploy a directory in your editor.
2:41 In my case, I'm going to use Vim, and we're going to start this file off by naming init config, which is going to correspond
2:52 to the playbook that we're running. Next, we need to specify the IP address of our server. Now, if you don't remember what that is, you can shift over
3:01 into our droplet, click copy and just paste it in. If we had many servers we wanted to deploy too, they would all be listed here
3:10 but in our case we're only going to deploy to a single server. Next up, we need to do one more thing, which is we need to specify
3:16 the Ansible Python interpreter that we want to use when we connect to a remote machine.
3:21 Ansible is actually going to be interacting with the Python environment on each remote machine, so our Digital Ocean droplet
3:27 already has Python installed by default, but it's Python 3 in Ubuntu 16.04, that's the default installation and we want to explicitly specify that,
3:38 otherwise we'll encounter some errors down the road when Ansible is trying to work without remote environment. So type in ansible_Python_interpreter
3:47 and specify under user bin Python 3, save that. Alright, that's all we need to do right now for a host file we'll add to it later,
3:55 when we write our second playbook, this is all we need at the moment. Quit out of your editor, and that's our first file, hosts.
4:02 We're going to write a second file, and that file is init_config.yml. Open that up, and I typically start my files with a comment
4:11 very high level of what this playbook intends to accomplish. So in our case this is just going to lock down the server,
4:20 and set up a non-root user. So what's this non-root user? We're going to create a separate user named deployer
4:29 and we are only going to log into the server with deployer we're actually going to disable the root user as a log in account
4:34 the reason behind this is because many malicious actors will just try to connect to 04:36 servers as root, they won't try any other accounts, and even if they do
4:41 they need to know the username which really could be anything that we want it to be;
4:46 it's just one of the security precautions that we're taking as we set up our server.
4:49 Now we're going to specify which hosts we want to apply this configuration to, and we're just going to give this a name, here we'll just say
4:56 apply initial configuration to server, next up, we're going to specify the hosts,
5:02 now the hosts is going to apply based on what we've set up in our hosts file. And we can't say all here, and that will apply this configuration
5:09 to every single server that we've listed, but in our case, we're going to say init config. For this playbook our user is going to be root,
5:17 because we're going to log in as root as we lock down our server and later on we'll log in as a deployer user,
5:25 and the roles that we'll apply are just going to be init config. Now, the role is a directory that we're going to create
5:33 and that will contain other yaml files with tasks within them that are going to run when we execute this with the Ansible playbook command,
5:41 that's all we need for now in the initial config yaml file. We'll save, quit that, now we're going to create two subdirectories;
5:48 the first one is going to be group vars, this is the variables that we'll use, that are customized for our deployment,
5:56 and the second one is roles, so create those two directories, move into roles, and create a subdirectory init_config,
6:03 move into init_config, and then create a directory named tasks. This is the directory structure than Ansible is going to use
6:13 in order to kick off and execute the playbook. Within tasks, create a new file, main.yml
6:20 this is where we're going to write the playbook with specific tasks each step that we need to lock it on our server,
6:26 again, I always start with a brief comment, now let's write our first task. Again name is just a description of what we're going to do,
6:33 this first one is going to update and upgrade packages, and install a package named fail2ban.
6:41 Sometimes when I'm writing playbooks, I will actually just write out all the tasks
6:44 just to start, so I know what steps I need to take to flesh out the whole playbook, and that's what we'll do here;
6:50 so the first step is, we're going to install fail2ban, which is going to prevent anyone from continually trying to access the server
6:57 it will create a delay between when they try to log into the server and when they are allowed for their next attempt.
7:04 We can also use it to blacklist certain IP addresses, if they're constantly trying to attack our server.
7:09 Now the second step, we're going to create a non-root group so any user in this group is going to get what is called super user privileges
7:16 which allows them to make a system wide changes that are called privileged changes, which is escalated privilege required,
7:26 execute certain applications or access certain resources on a Linux system. After we create our non-root group, were going to create a non-root user,
7:35 this user will be added to that non-root group, we need to add an authorized key, which is the equivalent of what we did
7:41 when we specified our ssh key on Digital Ocean, after we have an authorized key, we will add the non-root group to super user privileges.
7:52 That will give our non-root user the ability to make all the changes that we need on the system, like restarting services,
7:59 installing new packages, those sorts of things. Next, we're going to disable the root ssh logins. That way, no one will be able to ssh in
8:14 by connecting to the root user, once we start ssh. One more step, we're going to make sure that no one can log in via password on ssh,
8:23 we only want to be able to log in via our private key, once we've made those two changes, the last step is to restart the ssh server.
8:31 So that's the high level of what our initial configuration is going to do on our server; now we need to fill in each one of these tasks.
8:40 For this task, we're going to use the apt module, the apt module takes a few parameters,
8:46 first is the name and this is going to be what package we want to install; in our case, fail2ban, we want it to be present,
8:54 so we want it to be installed on the system, and we want to make sure we update the system cash
9:01 so we don't get an old version of fail2ban installed on our system. And we'll specify yes to upgrade packages that are out of date.
9:11 Ok, so how do we know what the apt module does? This is a good time to go take a look at the modules index for Ansible.
9:17 Typically, if you just do a search, the first result that's going to pop up is the Ansible documentation, Ansible and then the package name,
9:23 and then the module name, so this documentation tells us how this module works, as well as any requirements,
9:29 so we're going to be running this with Python 3, and aptitude is installed by default on Ubuntu, that's the package manager for Ubuntu.
9:38 And these are the parameters that can be specified and in some cases we don't need to specify them,
9:43 so for example if we see that something is not required, it will either have a default, or it just won't apply when we run this task.
9:51 So there are a lot more advanced options that we can set here we've only used the basic ones because we're just installing a simple package;
9:57 this is why I keep the Ansible module documentation open because every single time I write a task I want to see
10:04 if there's some sort of additional option that I should be using as a part of writing that task.
10:09 Switch back over to our playbook, let's write our next task. We're going to use the group module in order to create a group
10:14 and the name of the group is going to be the value set by a variable. We haven't set any variables yet,
10:21 but this is what our group vars directory is going to contain, we're going to write a variable separate from our playbook
10:26 within the group vars directory, and one of those variables is going to be a deploy group, the name of the deploy group.
10:32 Now, this should look familiar, state present, because we did the same thing with the apt module
10:37 so what we're saying with this module is create a group named whatever our deployed group variable value is.
10:42 Now, if that variable is not set, Ansible will throw an error. In most cases, I do not set default values, because I actually want
10:48 Ansible to tell me when it's encountered an error. If you're writing a playbook in the future, what you can also do is
10:53 you can specify a default value with default, so in this case, we could say deployers,
11:01 and we need to make sure that we have single quote within the double quotes, and that we use deployers as the value, if deploy group is not set.
11:12 So it's just a handy tip for when you're working with Ansible, if you do want to set default values. In our case, we don't.
11:18 Next what we want to write is the user module, the name of the user is going to be set as deploy user,
11:25 the group will be the deploy group, we'll use the value of that variable
11:30 we'll set a default shell, I always like it to be bash rather than the default just ssh shell, we'll say we want this user to exist.
11:39 For some reason, we were writing a playbook where we wanted a user to be deleted or definitely not exist, we could say absent,
11:45 we need this user, so that's why we would say present. Next up, we use the authorized key module
11:51 and now we're going to set an authorized key for a user, the user is going to be the one that we have specified
11:56 as a value to our deploy user variable, state present again and note that in the previous task, particular present within quotes
12:04 now that shouldn't matter, but let's remove them here Ansible can get a little tricky with the quotes
12:11 but a general rule of thumb is that if you're using a variable you're going to want to surround that with quotes
12:16 because your variable may have spaces or special characters in the value, if we know because we're hardcoding something like present or absent
12:22 that it won't have special characters or spaces we don't need the quotes, so I'll typically leave them out.
12:28 However, spaces and special characters can mess up Ansible's parser which is parsing the yaml, so if you want to be on the safe side
12:35 you can use quotes around everything in your playbook and the only one I don't seem to have trouble with with spaces is the name task.
12:42 So again, typically I won't use quotes around the value to a name. Alright, let's finish up the authorized key, we need one more bit here,
12:50 we need to specify where the key is located on our local system, so that it can be uploaded to the remote machine,
12:56 so we'll use the key argument and then we're going to look for a file we're going to do a lookup for a file, we're going to use another variable
13:03 we'll set ssh dir and we'll add the name of our key which in our case was do deploy, and we particularly want the public key.
13:11 Now real quick, we could leave this and it's probably better to leave this as a variable, so there's multiple ways that we can write our Ansible tasks
13:20 so maybe the whole idea is we want these to be maintainable over time and variables help us to keep our playbooks maintainable.
13:27 So instead of hardcoding do deploy as our name, let's call it ssh key and we'll add the .pub at the end; we'll say ssh key name
13:35 and then we'll make sure that we create a variable for that later. Alright, we're getting pretty close to being able to run this,
13:41 let's fill out the remaining four tasks. The next two tasks we're going to modify the ssh daemon configuration,
13:48 we're going to use another module named lineinfile which allows us to replace certain lines in files
13:55 the destination of this is going to be on our remote machine, and actually, in this case we're going to be modifying
14:01 the Super User, the sudoers file, we need to modify it so that anyone in our deploy group gets Super User privileges when they use the sudo command.
14:11 The state should be present, we use a regular expression to define that we want the deploy group in this file
14:20 so what we're saying here is, if the value of our deploy group is in the file you can skip this step or Ansible can skip this step
14:29 but if it's not located in the file, we want to add a line, and that line is the value of the deploy group, and then some text that specifies
14:38 that the deploy group is getting all privileges. Finally, we'll validate the file, to make sure we didn't mess anything up.
14:54 Next, we're going to use the replace module the file here, it's going to be our ssh server configuration, we'll again use the regular expression
15:08 and if the file says permit root login, then we want to change that
15:12 and replace it with permit root login, no back up- we don't need to back up the file. Let's copy this, next disable ssh logins by password,
15:23 password authentication- yes, it should be replaced with password authentication- no.
15:35 Last task, we just want to restart the ssh service, so we'll use the service module the name of the service is ssh, and state should be restarted.
15:47 Cool, that's our playbook, save it and now we just need one more file to declare our variables, and we'll be able to start executing this.
15:55 So go into the group vars file, and we'll just create a file named all. So this is just going to contain all of our variables for our deployment.
16:07 And if your deployment gets really complicated, you can create many variable files I typically don't have so many variables for single applications
16:13 that I just store them all in one file for convenience. Alright, so there are four variables that we defined in that init_config playbook.
16:20 The first one was deploy group, I'll just say deployers, the second one is deploy user, let's call this deployer;
16:29 The third one is the directory that stores our ssh keys, I just stored the do deploy under my home directory, and then the ssh keyname do deploy.
16:46 That should do it, save the file, and let's see if we can execute this. In order to execute our playbook,
16:55 we're going to use the Ansible playbook command we have to specify a few arguments here, and rather than just typing this over and over again
17:05 what I typically do is I just store them in a shell file, so this is going to be a bash shell script,
17:11 and we are going to use the Ansible playbook command now -vvvv says give me very verbose output sometimes you're going to want that, sometimes not,
17:21 especially when I'm writing a playbook, executing for the first few times I want to see all of the output, just in case I run into any errors
17:27 I want to know immediately what those errors are, so I can fix them. We're going to use the init_config yaml file that we created,
17:34 we're going to specify a private key that is stored under our home directory, do deploy
17:40 these will all be root, they were connecting their own machine with and we're going to use our hosts file,
17:46 save that, let me try that, in order to execute the shell script you need to change its permissions
17:51 so 0700, on init.sh, now if we look at this init.sh file, it has the x for executable, let's kick it off,
18:02 oh no, of course, the first thing we run into is one error so let's modify this and what actually the problem is
18:14 is that it's a private -key, alright, one more time okay so this is typically what you're going to find
18:25 when you work with Ansible and you are just running it for the very first time so let's work through a few of the issues that I ran into.
18:32 This is the verbose output, and this is a good opportunity to just kind of understand the errors we may run into, so what's the problem here?
18:39 What it's telling us is that it couldn't find this public key, now at first this is a little bit confusing, this is definitely our public key,
18:47 but what I realize just by looking at it is this appears incorrect, so we don't want to specify the relative path,
18:55 which we did with a period before the forward slash we just want to specify the absolute path, so let's go ahead and modify this init.sh again
19:04 and in our case it's going to be just home/matt/do_deploy and we'll save this, all right, one more time- awesome, so now it was able to find the file
19:21 but again, we ran into an error, so let's take a look at what this error message is telling us.
19:26 Well, it looks like there's a problem with one of our tasks, now when you're working with Ansible, and you're writing a new playbook
19:32 you can't expect everything to go right, just from the beginning; that's why I think it's really important just to dig into
19:37 what are the errors that are occurring, and not be afraid of them, because the great part is is that we're actually getting some midi information here,
19:43 that we may have used two arguments based on what we're seeing here and they're mutually exclusive, you can't put both of them in there at once
19:52 in our case we use both package and upgrade at the same time, now that may not be necessary, so let's go into our roles,
19:59 init_config, tasks, modify_main.yml. So the issue here is we are not actually going to use this upgrade
20:07 we just want to make sure that the fail2ban package is present upgrade is used when we actually want to upgrade existing packages that are there.
20:14 So we'll remove that, save it and now try to run it again. A whole bunch of blue output, that's good so far,
20:27 it looks like we may have gotten past the point that we had earlier and let's let Ansible do its thing.
20:37 Alright, so we got even further, it doesn't quite look like everything worked out all right, one more error that we ran into,
20:44 this one looks fairly easy and straightforward, it looks like the period is missing from deploy.pub
20:50 so it wasn't able to use, to play pub as an authorized key, so this typically happens when you're working with variables
20:58 so if we take a look at what's under our variables file, we're going to see that the ssh key name is do deploy
21:05 but there's no period after it, so if we go under our tasks, and take a look at what we have here, it looks like we use the name
21:15 and then we appended pub but no period after that, so we can save that, and once again, we'll kick it off.
21:21 And this is a fairly standard workflow for how you're going to work with Ansible,
21:25 and once you got the scripts written, you can run them over and over again, it's going to automate your whole deployment
21:31 but there is a little bit of work and just getting it automated to begin with.
21:34 Let's kick this off again, okay, amazing, so it looks like everything worked just fine.
21:39 Now, if it actually did work we should be able to connect to the remote machine as a check with this new deployer user, let's try that right now.
21:48 So we're going to ssh in, and we're going to specify the do deploy key and this time, we're going to use deployer user name as opposed to root,
21:58 and we'll copy and paste our IP address, and let's give this a try- yes, we are online and now our server is more locked down
22:12 because no one can log in with the root user name, and let's just test that out real quick;
22:17 we know we could do it earlier, because we were able to log in but now let's try to log in with the root user.
22:22 No, unable to do so, and it's just going to give a blanked denial which is permission denied due to a public key.
22:30 Really that's our ssh configuration, that is preventing us from logging in via root. With that, we've written our first Ansible playbook,
22:39 this is a playbook that you're going to want to run whenever you spin up a new Digital Ocean instance,
22:44 it's going to lock down create a new user for you, that can handle the rest of the deployment, and now are ready to install our web application
22:52 and make sure everything is up and running.