Python for Entrepreneurs Transcripts
Chapter: Deploying to the cloud
Lecture: The initial configuration
Login or
purchase this course
to watch this video and the rest of the course contents.
0:01
Hopefully you took a look through the Ansible documentation, but I found that the best way to learn Ansible and how to do automated deployments
0:09
is just to dig in and start writing your first playbook. The first playbook is going to be one that we only run when we spin up a new server,
0:16
the base Ubuntu image that we selected on Digital Ocean is not harded against attackers who might want to try to lock into your box.
0:23
The main security mechanism that's in place from the start is just that you can only log in with an ssh key, but there are many more steps
0:29
that we need to take in order to protect this box, because it is live with a public IP address on the internet.
0:36
Shift back over into your development environment, and either go into the course demos or move into your own project
0:42
you can start writing the deployment playbooks for your project as we go along through this chapter. Or if you don't have your project set up just yet,
0:51
you can follow along through the course demo. Within the course demo's git repository, switch into chapter 15, deployment,
0:56
there will be another folder within it, blue yellow app deployment. This is the same project that Michael was working on the previous chapter,
1:05
where he set up email, we're now going to take this application and deploy it on our Digital Ocean server.
1:12
The first step is to create a deployment directory, that will store all the files Ansible needs to execute,
1:18
create that directory, and then switch into it. If you haven't already activated your virtual environment, go ahead and do that now.
1:23
You can see mine is activated here, I just have one called entre, and it's also using Python 3. The first thing we want to do, is install Ansible.
1:31
Now typically we would just say pip install Ansible, but since Python 3 support is being actively worked on, in the development branch,
1:38
I recommend installing the development branch rather than the current stable release. As soon as Ansible cuts the 2.3 version
1:46
you can then install Ansible via Ansible 2.3. The way that we install from git, is we instead use the git protocol
1:54
pointed to the repository that we want to install, in our case that's going to be github.com/ansible/ansible.
2:02
Now this is going to clone the git repository, and install it into our virtual environment, in my case, I've already installed Ansible,
2:14
wait for it to finish in your own environment, and then we can proceed. Now that Ansible is installed, when you do pip freeze
2:21
it'll show us that Ansible 2.3.0 is installed, that's the minimum version that we're going to want.
2:27
With Ansible installed, we can now start to write our files. The first file that we need is a hosts file,
2:33
this is where we tell Ansible where our servers are located; open up hosts, file name hosts, under the deploy a directory in your editor.
2:41
In my case, I'm going to use Vim, and we're going to start this file off by naming init config, which is going to correspond
2:52
to the playbook that we're running. Next, we need to specify the IP address of our server. Now, if you don't remember what that is, you can shift over
3:01
into our droplet, click copy and just paste it in. If we had many servers we wanted to deploy too, they would all be listed here
3:10
but in our case we're only going to deploy to a single server. Next up, we need to do one more thing, which is we need to specify
3:16
the Ansible Python interpreter that we want to use when we connect to a remote machine.
3:21
Ansible is actually going to be interacting with the Python environment on each remote machine, so our Digital Ocean droplet
3:27
already has Python installed by default, but it's Python 3 in Ubuntu 16.04, that's the default installation and we want to explicitly specify that,
3:38
otherwise we'll encounter some errors down the road when Ansible is trying to work without remote environment. So type in ansible_Python_interpreter
3:47
and specify under user bin Python 3, save that. Alright, that's all we need to do right now for a host file we'll add to it later,
3:55
when we write our second playbook, this is all we need at the moment. Quit out of your editor, and that's our first file, hosts.
4:02
We're going to write a second file, and that file is init_config.yml. Open that up, and I typically start my files with a comment
4:11
very high level of what this playbook intends to accomplish. So in our case this is just going to lock down the server,
4:20
and set up a non-root user. So what's this non-root user? We're going to create a separate user named deployer
4:29
and we are only going to log into the server with deployer we're actually going to disable the root user as a log in account
4:34
the reason behind this is because many malicious actors will just try to connect to 04:36 servers as root, they won't try any other accounts, and even if they do
4:41
they need to know the username which really could be anything that we want it to be;
4:46
it's just one of the security precautions that we're taking as we set up our server.
4:49
Now we're going to specify which hosts we want to apply this configuration to, and we're just going to give this a name, here we'll just say
4:56
apply initial configuration to server, next up, we're going to specify the hosts,
5:02
now the hosts is going to apply based on what we've set up in our hosts file. And we can't say all here, and that will apply this configuration
5:09
to every single server that we've listed, but in our case, we're going to say init config. For this playbook our user is going to be root,
5:17
because we're going to log in as root as we lock down our server and later on we'll log in as a deployer user,
5:25
and the roles that we'll apply are just going to be init config. Now, the role is a directory that we're going to create
5:33
and that will contain other yaml files with tasks within them that are going to run when we execute this with the Ansible playbook command,
5:41
that's all we need for now in the initial config yaml file. We'll save, quit that, now we're going to create two subdirectories;
5:48
the first one is going to be group vars, this is the variables that we'll use, that are customized for our deployment,
5:56
and the second one is roles, so create those two directories, move into roles, and create a subdirectory init_config,
6:03
move into init_config, and then create a directory named tasks. This is the directory structure than Ansible is going to use
6:13
in order to kick off and execute the playbook. Within tasks, create a new file, main.yml
6:20
this is where we're going to write the playbook with specific tasks each step that we need to lock it on our server,
6:26
again, I always start with a brief comment, now let's write our first task. Again name is just a description of what we're going to do,
6:33
this first one is going to update and upgrade packages, and install a package named fail2ban.
6:41
Sometimes when I'm writing playbooks, I will actually just write out all the tasks
6:44
just to start, so I know what steps I need to take to flesh out the whole playbook, and that's what we'll do here;
6:50
so the first step is, we're going to install fail2ban, which is going to prevent anyone from continually trying to access the server
6:57
it will create a delay between when they try to log into the server and when they are allowed for their next attempt.
7:04
We can also use it to blacklist certain IP addresses, if they're constantly trying to attack our server.
7:09
Now the second step, we're going to create a non-root group so any user in this group is going to get what is called super user privileges
7:16
which allows them to make a system wide changes that are called privileged changes, which is escalated privilege required,
7:26
execute certain applications or access certain resources on a Linux system. After we create our non-root group, were going to create a non-root user,
7:35
this user will be added to that non-root group, we need to add an authorized key, which is the equivalent of what we did
7:41
when we specified our ssh key on Digital Ocean, after we have an authorized key, we will add the non-root group to super user privileges.
7:52
That will give our non-root user the ability to make all the changes that we need on the system, like restarting services,
7:59
installing new packages, those sorts of things. Next, we're going to disable the root ssh logins. That way, no one will be able to ssh in
8:14
by connecting to the root user, once we start ssh. One more step, we're going to make sure that no one can log in via password on ssh,
8:23
we only want to be able to log in via our private key, once we've made those two changes, the last step is to restart the ssh server.
8:31
So that's the high level of what our initial configuration is going to do on our server; now we need to fill in each one of these tasks.
8:40
For this task, we're going to use the apt module, the apt module takes a few parameters,
8:46
first is the name and this is going to be what package we want to install; in our case, fail2ban, we want it to be present,
8:54
so we want it to be installed on the system, and we want to make sure we update the system cash
9:01
so we don't get an old version of fail2ban installed on our system. And we'll specify yes to upgrade packages that are out of date.
9:11
Ok, so how do we know what the apt module does? This is a good time to go take a look at the modules index for Ansible.
9:17
Typically, if you just do a search, the first result that's going to pop up is the Ansible documentation, Ansible and then the package name,
9:23
and then the module name, so this documentation tells us how this module works, as well as any requirements,
9:29
so we're going to be running this with Python 3, and aptitude is installed by default on Ubuntu, that's the package manager for Ubuntu.
9:38
And these are the parameters that can be specified and in some cases we don't need to specify them,
9:43
so for example if we see that something is not required, it will either have a default, or it just won't apply when we run this task.
9:51
So there are a lot more advanced options that we can set here we've only used the basic ones because we're just installing a simple package;
9:57
this is why I keep the Ansible module documentation open because every single time I write a task I want to see
10:04
if there's some sort of additional option that I should be using as a part of writing that task.
10:09
Switch back over to our playbook, let's write our next task. We're going to use the group module in order to create a group
10:14
and the name of the group is going to be the value set by a variable. We haven't set any variables yet,
10:21
but this is what our group vars directory is going to contain, we're going to write a variable separate from our playbook
10:26
within the group vars directory, and one of those variables is going to be a deploy group, the name of the deploy group.
10:32
Now, this should look familiar, state present, because we did the same thing with the apt module
10:37
so what we're saying with this module is create a group named whatever our deployed group variable value is.
10:42
Now, if that variable is not set, Ansible will throw an error. In most cases, I do not set default values, because I actually want
10:48
Ansible to tell me when it's encountered an error. If you're writing a playbook in the future, what you can also do is
10:53
you can specify a default value with default, so in this case, we could say deployers,
11:01
and we need to make sure that we have single quote within the double quotes, and that we use deployers as the value, if deploy group is not set.
11:12
So it's just a handy tip for when you're working with Ansible, if you do want to set default values. In our case, we don't.
11:18
Next what we want to write is the user module, the name of the user is going to be set as deploy user,
11:25
the group will be the deploy group, we'll use the value of that variable
11:30
we'll set a default shell, I always like it to be bash rather than the default just ssh shell, we'll say we want this user to exist.
11:39
For some reason, we were writing a playbook where we wanted a user to be deleted or definitely not exist, we could say absent,
11:45
we need this user, so that's why we would say present. Next up, we use the authorized key module
11:51
and now we're going to set an authorized key for a user, the user is going to be the one that we have specified
11:56
as a value to our deploy user variable, state present again and note that in the previous task, particular present within quotes
12:04
now that shouldn't matter, but let's remove them here Ansible can get a little tricky with the quotes
12:11
but a general rule of thumb is that if you're using a variable you're going to want to surround that with quotes
12:16
because your variable may have spaces or special characters in the value, if we know because we're hardcoding something like present or absent
12:22
that it won't have special characters or spaces we don't need the quotes, so I'll typically leave them out.
12:28
However, spaces and special characters can mess up Ansible's parser which is parsing the yaml, so if you want to be on the safe side
12:35
you can use quotes around everything in your playbook and the only one I don't seem to have trouble with with spaces is the name task.
12:42
So again, typically I won't use quotes around the value to a name. Alright, let's finish up the authorized key, we need one more bit here,
12:50
we need to specify where the key is located on our local system, so that it can be uploaded to the remote machine,
12:56
so we'll use the key argument and then we're going to look for a file we're going to do a lookup for a file, we're going to use another variable
13:03
we'll set ssh dir and we'll add the name of our key which in our case was do deploy, and we particularly want the public key.
13:11
Now real quick, we could leave this and it's probably better to leave this as a variable, so there's multiple ways that we can write our Ansible tasks
13:20
so maybe the whole idea is we want these to be maintainable over time and variables help us to keep our playbooks maintainable.
13:27
So instead of hardcoding do deploy as our name, let's call it ssh key and we'll add the .pub at the end; we'll say ssh key name
13:35
and then we'll make sure that we create a variable for that later. Alright, we're getting pretty close to being able to run this,
13:41
let's fill out the remaining four tasks. The next two tasks we're going to modify the ssh daemon configuration,
13:48
we're going to use another module named lineinfile which allows us to replace certain lines in files
13:55
the destination of this is going to be on our remote machine, and actually, in this case we're going to be modifying
14:01
the Super User, the sudoers file, we need to modify it so that anyone in our deploy group gets Super User privileges when they use the sudo command.
14:11
The state should be present, we use a regular expression to define that we want the deploy group in this file
14:20
so what we're saying here is, if the value of our deploy group is in the file you can skip this step or Ansible can skip this step
14:29
but if it's not located in the file, we want to add a line, and that line is the value of the deploy group, and then some text that specifies
14:38
that the deploy group is getting all privileges. Finally, we'll validate the file, to make sure we didn't mess anything up.
14:54
Next, we're going to use the replace module the file here, it's going to be our ssh server configuration, we'll again use the regular expression
15:08
and if the file says permit root login, then we want to change that
15:12
and replace it with permit root login, no back up- we don't need to back up the file. Let's copy this, next disable ssh logins by password,
15:23
password authentication- yes, it should be replaced with password authentication- no.
15:35
Last task, we just want to restart the ssh service, so we'll use the service module the name of the service is ssh, and state should be restarted.
15:47
Cool, that's our playbook, save it and now we just need one more file to declare our variables, and we'll be able to start executing this.
15:55
So go into the group vars file, and we'll just create a file named all. So this is just going to contain all of our variables for our deployment.
16:07
And if your deployment gets really complicated, you can create many variable files I typically don't have so many variables for single applications
16:13
that I just store them all in one file for convenience. Alright, so there are four variables that we defined in that init_config playbook.
16:20
The first one was deploy group, I'll just say deployers, the second one is deploy user, let's call this deployer;
16:29
The third one is the directory that stores our ssh keys, I just stored the do deploy under my home directory, and then the ssh keyname do deploy.
16:46
That should do it, save the file, and let's see if we can execute this. In order to execute our playbook,
16:55
we're going to use the Ansible playbook command we have to specify a few arguments here, and rather than just typing this over and over again
17:05
what I typically do is I just store them in a shell file, so this is going to be a bash shell script,
17:11
and we are going to use the Ansible playbook command now -vvvv says give me very verbose output sometimes you're going to want that, sometimes not,
17:21
especially when I'm writing a playbook, executing for the first few times I want to see all of the output, just in case I run into any errors
17:27
I want to know immediately what those errors are, so I can fix them. We're going to use the init_config yaml file that we created,
17:34
we're going to specify a private key that is stored under our home directory, do deploy
17:40
these will all be root, they were connecting their own machine with and we're going to use our hosts file,
17:46
save that, let me try that, in order to execute the shell script you need to change its permissions
17:51
so 0700, on init.sh, now if we look at this init.sh file, it has the x for executable, let's kick it off,
18:02
oh no, of course, the first thing we run into is one error so let's modify this and what actually the problem is
18:14
is that it's a private -key, alright, one more time okay so this is typically what you're going to find
18:25
when you work with Ansible and you are just running it for the very first time so let's work through a few of the issues that I ran into.
18:32
This is the verbose output, and this is a good opportunity to just kind of understand the errors we may run into, so what's the problem here?
18:39
What it's telling us is that it couldn't find this public key, now at first this is a little bit confusing, this is definitely our public key,
18:47
but what I realize just by looking at it is this appears incorrect, so we don't want to specify the relative path,
18:55
which we did with a period before the forward slash we just want to specify the absolute path, so let's go ahead and modify this init.sh again
19:04
and in our case it's going to be just home/matt/do_deploy and we'll save this, all right, one more time- awesome, so now it was able to find the file
19:21
but again, we ran into an error, so let's take a look at what this error message is telling us.
19:26
Well, it looks like there's a problem with one of our tasks, now when you're working with Ansible, and you're writing a new playbook
19:32
you can't expect everything to go right, just from the beginning; that's why I think it's really important just to dig into
19:37
what are the errors that are occurring, and not be afraid of them, because the great part is is that we're actually getting some midi information here,
19:43
that we may have used two arguments based on what we're seeing here and they're mutually exclusive, you can't put both of them in there at once
19:52
in our case we use both package and upgrade at the same time, now that may not be necessary, so let's go into our roles,
19:59
init_config, tasks, modify_main.yml. So the issue here is we are not actually going to use this upgrade
20:07
we just want to make sure that the fail2ban package is present upgrade is used when we actually want to upgrade existing packages that are there.
20:14
So we'll remove that, save it and now try to run it again. A whole bunch of blue output, that's good so far,
20:27
it looks like we may have gotten past the point that we had earlier and let's let Ansible do its thing.
20:37
Alright, so we got even further, it doesn't quite look like everything worked out all right, one more error that we ran into,
20:44
this one looks fairly easy and straightforward, it looks like the period is missing from deploy.pub
20:50
so it wasn't able to use, to play pub as an authorized key, so this typically happens when you're working with variables
20:58
so if we take a look at what's under our variables file, we're going to see that the ssh key name is do deploy
21:05
but there's no period after it, so if we go under our tasks, and take a look at what we have here, it looks like we use the name
21:15
and then we appended pub but no period after that, so we can save that, and once again, we'll kick it off.
21:21
And this is a fairly standard workflow for how you're going to work with Ansible,
21:25
and once you got the scripts written, you can run them over and over again, it's going to automate your whole deployment
21:31
but there is a little bit of work and just getting it automated to begin with.
21:34
Let's kick this off again, okay, amazing, so it looks like everything worked just fine.
21:39
Now, if it actually did work we should be able to connect to the remote machine as a check with this new deployer user, let's try that right now.
21:48
So we're going to ssh in, and we're going to specify the do deploy key and this time, we're going to use deployer user name as opposed to root,
21:58
and we'll copy and paste our IP address, and let's give this a try- yes, we are online and now our server is more locked down
22:12
because no one can log in with the root user name, and let's just test that out real quick;
22:17
we know we could do it earlier, because we were able to log in but now let's try to log in with the root user.
22:22
No, unable to do so, and it's just going to give a blanked denial which is permission denied due to a public key.
22:30
Really that's our ssh configuration, that is preventing us from logging in via root. With that, we've written our first Ansible playbook,
22:39
this is a playbook that you're going to want to run whenever you spin up a new Digital Ocean instance,
22:44
it's going to lock down create a new user for you, that can handle the rest of the deployment, and now are ready to install our web application
22:52
and make sure everything is up and running.