In this article written by Gourav Shah, author of the book Ansible Playbook Essentials, we will learn about the following topics:
- The anatomy of a playbook
- What plays are and how to write a Hosts inventory and search patterns
- Ansible modules and the batteries-included approach
- Orchestrating infrastructure with Ansible
Ansible as an orchestrator
(For more resources related to this topic, see here.)
Getting introduced to Ansible
Ansible is a simple, flexible, and extremely powerful tool that gives you the ability to automate common infrastructure tasks, run ad hoc commands, and deploy multitier applications spanning multiple machines. Even though you can use Ansible to launch commands on a number of hosts in parallel, the real power lies in managing those using playbooks.
As systems engineer, infrastructure that we typically need to automate contains complex multitier applications. Each of which represents a class of servers, for example, load balancers, web servers, database servers, caching applications, and middleware queues. Since many of these applications have to work in tandem to provide a service, there is topology involved as well. For example, a load balancer would connect to web servers, which in turn read/write to a database and connect to the caching server to fetch in-memory objects. Most of the time, when we launch such application stacks, we need to configure these components in a very specific order.
Here is an example of a very common three-tier web application running a load balancer, a web server, and a database backend:
Ansible lets you translate this diagram into a blueprint, which defines your infrastructure policies. The format used to specify such policies is what playbooks are.
Example policies and the sequence in which those are to be applied is shown in the following steps:
- Install, configure, and start the MySQL service on the database servers.
- Install and configure the web servers that run Nginx with PHP bindings.
- Deploy a WordPress application on the web servers and add respective configurations to Nginx.
- Start the Nginx service on all web servers after deploying WordPress. Finally, install, configure, and start the haproxy service on the load balancer hosts. Update haproxy configurations with the hostnames of all the web servers created earlier.
The following is a sample playbook that translates the infrastructure blueprint into policies enforceable by Ansible:
Plays
A playbook consists of one or more plays, which map groups of hosts to well-defined tasks. The preceding example contains three plays, each to configure one layer in the multitiered web application. Plays also define the order in which tasks are configured. This allows us to orchestrate multitier deployments. For example, configure the load balancers only after starting the web servers, or perform two-phase deployment where the first phase only adds this configurations and the second phase starts the services in the desired order.
YAML – the playbook language
As you may have already noticed, the playbook that we wrote previously resembles more of a text configuration than a code snippet. This is because the creators of Ansible chose to use a simple, human-readable, and familiar YAML format to blueprint the infrastructure. This adds to Ansible’s appeal, as users of this tool need not learn any special programming language to get started with. Ansible code is self-explanatory and self-documenting in nature. A quick crash course on YAML should suffice to understand the basic syntax. Here is what you need to know about YAML to get started with your first playbook:
- The first line of a playbook should begin with “— ” (three hyphens) which indicates the beginning of the YAML document.
- Lists in YAML are represented with a hyphen followed by a white space. A playbook contains a list of plays; they are represented with “- “. Each play is an associative array, a dictionary, or a map in terms of key-value pairs.
- Indentations are important. All members of a list should be at the same indentation level.
- Each play can contain key-value pairs separated by “:” to denote hosts, variables, roles, tasks, and so on.
Our first playbook
Equipped with the basic rules explained previously and assuming readers have done a quick dive into YAML fundamentals, we will now begin writing our first playbook. Our problem statement includes the following:
- Create a devops user on all hosts. This user should be part of the devops group.
- Install the “htop” utility. Htop is an improved version of top—an interactive system process monitor.
- Add the Nginx repository to the web servers and start it as a service.
Now, we will create our first playbook and save it as simple_playbook.yml containing the following code:
--- - hosts: all remote_user: vagrant sudo: yes tasks: - group: name: devops state: present - name: create devops user with admin privileges user: name: devops comment: "Devops User" uid: 2001 group: devops - name: install htop package action: apt name=htop state=present update_cache=yes - hosts: www user: vagrant sudo: yes tasks: - name: add official nginx repository apt_repository: repo: 'deb http://nginx.org/packages/ubuntu/ lucid nginx' - name: install nginx web server and ensure its at the latest version apt: name: nginx state: latest - name: start nginx service service: name: nginx state: started
Our playbook contains two plays. Each play consists of the following two important parts:
- What to configure: We need to configure a host or group of hosts to run the play against. Also, we need to include useful connection information, such as which user to connect as, whether to use sudo command, and so on.
- What to run: This includes the specification of tasks to be run, including which system components to modify and which state they should be in, for example, installed, started, or latest. This could be represented with tasks and later on, by roles.
Let’s now look at each of these briefly.
Creating a host inventory
Before we even start writing our playbook with Ansible, we need to define an inventory of all hosts that need to be configured, and make it available for Ansible to use. Later, we will start running plays against a selection of hosts from this inventory. If you have an existing inventory, such as cobbler, LDAP, a CMDB software, or wish to pull it from a cloud provider, such as ec2, it can be pulled from Ansible using the concept of a dynamic inventory.
For text-based local inventory, the default location is /etc/ansible/hosts. For our learning environment, however, we will create a custom inventory file customhosts in our working directory, the contents of which are shown as follows. You are free to create your own inventory file:
#customhosts #inventory configs for my cluster [db] 192.168.61.11 ansible_ssh_user=vagrant [www] www-01.example.com ansible_ssh_user=ubuntu www-02 ansible_ssh_user=ubuntu [lb] lb0.example.com
Now, when our playbook maps a play to the group, the www (hosts: www), hosts in that group will be configured. The all keywords will match to all hosts from the inventory.
The following are the guidelines to for creating inventory files:
- Inventory files follow INI style configurations, which essentially include configuration blocks that start with host group/class names included in “[ ]“. This allows the selective execution on classes of systems, for example, [namenodes].
- A single host can be part of multiple groups. In such cases, host variables from both the groups will get merged, and the precedence rules apply. We will discuss variables and precedence in detail later.
- Each group contains a list of hosts and connection details, such as the SSH user to connect as, the SSH port number if non-default, SSH credentials/keys, sudo credentials, and so on. Hostnames can also contain globs, ranges, and more, to make it easy to include multiple hosts of the same type, which follow some naming patterns.
After creating an inventory of the hosts, it’s a good idea to validate connectivity using Ansible’s ping module (for example, ansible -m ping all).
Patterns
In the preceding playbook, the following lines decide which hosts to select to run a specific play:
- hosts: all - hosts: www
The first code will match all hosts, and the second code will match hosts which are part of the www group.
Patterns can be any of the following or their combinations:
|
Pattern Types |
Examples |
|
Group name |
namenodes |
|
Match all |
all or * |
|
Range |
namenode[0:100] |
|
Hostnames/hostname globs |
*.example.com, host01.example.com |
|
Exclusions |
namenodes:!secondaynamenodes |
|
Intersection |
namenodes:&zookeeper |
|
Regular expressions |
~(nn|zk).*.example.org |
Tasks
Plays map hosts to tasks. Tasks are a sequence of actions performed against a group of hosts that match the pattern specified in a play. Each play typically contains multiple tasks that are run serially on each machine that matches the pattern. For example, take a look at the following code snippet:
- group: name:devops state: present - name: create devops user with admin privileges user: name: devops comment: "Devops User" uid: 2001 group: devops
In the preceding example, we have two tasks. The first one is to create a group, and second is to create a user and add it to the group created earlier. If you notice, there is an additional line in the second task, which starts with name:. While writing tasks, it’s good to provide a name with a human-readable description of what this task is going to achieve. If not, the action string will be printed instead.
Each action in a task list can be declared by specifying the following:
- The name of the module
- Optionally, the state of the system component being managed
- The optional parameters
With newer versions of Ansible (0.8 onwards), writing an action keyword is now optional. We can directly provide the name of the module instead. So, both of these lines will have a similar action, that is,. installing a package with the apt module:
action: apt name=htop state=present update_cache=yes apt: name=nginx state=latest
Ansible stands out from other configuration management tools, with its batteries-included included approach. These batteries are “modules.” It’s important to understand what modules are before we proceed.
Modules
Modules are the encapsulated procedures that are responsible for managing specific system components on specific platforms.
Consider the following example:
- The apt module for Debian and the yum module for RedHat helps manage system packages
- The user module is responsible for adding, removing, or modifying users on the system
- The service module will start/stop system services
Modules abstract the actual implementation from users. They expose a declarative syntax that accepts a list of the parameters and states of the system components being managed. All this can be declared using the human-readable YAML syntax, using key-value pairs.
In terms of functionality, modules resemble providers for those of you who are familiar with Chef/Puppet software. Instead of writing procedures to create a user, with Ansible we declare which state our component should be in, that is, which user to create, its state, and its characteristics, such as UID, group, shell, and so on. The actual procedures are inherently known to Ansible via modules, and are executed in the background.
The Command and Shell modules are special ones. They neither take key-value pairs as parameters, nor are idempotent.
Ansible comes preinstalled with a library of modules, which ranges from the ones which manage basic system resources to more sophisticated ones that send notifications, perform cloud integrations, and so on. If you want to provision an ec2 instance, create a database on the remote PostgreSQL server, and get notifications on IRC, then Ansible has a module for it. Isn’t this amazing?
No need to worry about finding an external plugin, or struggle to integrate with cloud providers, and so on. To find a list of modules available, you can refer to the Ansible documentation at http://docs.ansible.com/list_of_all_modules.html.
Ansible is extendible too. If you do not find a module that does the job for you, it’s easy to write one, and it doesn’t have to be in Python. A module can be written for Ansible in the language of your choice. This is discussed in detail at http://docs.ansible.com/developing_modules.html.
The modules and idempotence
Idempotence is an important characteristic of a module. It is something which can be applied on your system multiple times, and will return deterministic results. It has built-in intelligence. For instance, we have a task that uses the apt module to install Nginx and ensure that it’s up to date. Here is what happens if you run it multiple times:
- Every time idempotance is run multiple times, the apt module will compare what has been declared in the playbook versus the current state of that package on the system. The first time it runs, Ansible will determine that Nginx is not installed, and will go ahead with the installation.
- For every consequent run, it will skip the installation part, unless there is a new version of the package available in the upstream repositories.
This allows executing the same task multiple times without resulting in the error state. Most of the Ansible modules are idempotent, except for the command and shell modules. Users will have to make these modules idempotent.
Running the playbook
Ansible comes with the ansible-playbook command to launch a playbook with. Let’s now run the plays we created:
$ ansible-playbook simple_playbook.yml -i customhosts
Here is what happens when you run the preceding command:
- The ansible-playbook parameter is the command that takes the playbook as an argument (simple_playbook.yml) and runs the plays against the hosts
- The simple_playbook parameter contains the two plays that we created: one for common tasks, and the other for installing Nginx
- The customhosts parameter is our host’s inventory, which lets Ansible know which hosts, or groups of hosts, to call plays against
Launching the preceding command will start calling plays, orchestrating in the sequence that we described in the playbook. Here is the output of the preceding command:
Let’s now analyze what happened:
- Ansible reads the playbooks specified as an argument to the ansible-playbook command and starts executing plays in the serial order.
- The first play that we declared, runs against the “all” hosts. The all keyword is a special pattern that will match all hosts (similar to *). So, the tasks in the first play will be executed on all hosts in the inventory we passed as an argument.
- Before running any of the tasks, Ansible will gather information about the systems that it is going to configure. This information is collected in the form of facts.
- The first play includes the creation of the devops group and user, and installation of the htop package. Since we have three hosts in our inventory, we see one line per host being printed, which indicates whether there was a change in the state of the entity being managed. If the state was not changed, “ok” will be printed.
- Ansible then moves to the next play. This is executed only on one host, as we have specifed “hosts:www” in our play, and our inventory contains a single host in the group “www“.
- During the second play, the Nginx repository is added, the package is installed, and the service is started.
- Finally, Ansible prints the summary of the playbook run in the “PLAY RECAP” section. It indicates how many modifications were made, if any of the hosts were unreachable, or execution failed on any of the systems.
What if a host is unresponsive, or fails to run tasks? Ansible has built-in intelligence, which will identify such issues and take the failed host out of rotation. It will not affect the execution on other hosts.
Orchestrating Infrastructure with Ansible
Orchestration can mean different things at different times when used in different scenarios. The following are some of the orchestration scenarios described:
- Running ad hoc commands in parallel on a group of hosts, for example, using a for loop to walk over a group of web servers to restart the Apache service. This is the crudest form of orchestration.
- Invoking an orchestration engine to launch another configuration management tool to enforce correct ordering.
- Configuring a multitier application infrastructure in a certain order with the ability to have fine-grained control over each step, and the flexibility to move back and forth while configuring multiple components. For example, installing the database, setting up the web server, coming back to the database, creating a schema, going to web servers to start services, and more.
Most real-world scenarios are similar to the last scenario, which involve a multitier application stacks and more than one environment, where it’s important to bring up and update nodes in a certain order, and in a coordinated way. It’s also useful to actually test that the application is up and running before moving on to the next. The workflow to set up the stack for the first time versus pushing updates can be different. There can be times when you would not want to update all the servers at once, but do them in batches so that downtime is avoided.
Ansible as an orchestrator
When it comes to orchestration of any sort, Ansible really shines over other tools. Of course, as the creators of Ansible would say, it’s more than a configuration management tool, which is true. Ansible can find a place for itself in any of the orchestration scenarios discussed earlier. It was designed to manage complex multitier deployments. Even if you have your infrastructure being automated with other configuration management tools, you can consider Ansible to orchestrate those.
Let’s discuss the specific features that Ansible ships with, which are useful for orchestration.
Multiple playbooks and ordering
Unlike most other configuration management systems, Ansible supports running different playbooks at different times to configure or manage the same infrastructure. You can create one playbook to set up the application stack for the first time, and another to push updates over time in a certain manner. Another property of the playbook is that it can contain more than one play, which allows the separation of groups of hosts for each tier in the application stack, and configures them at the same time.
Pre-tasks and post-tasks
We have used pre-tasks and post-tasks earlier, which are very relevant while orchestrating, as these allow us to execute a task or run validations before and after running a play. Let’s use the example of updating web servers that are registered with the load balancer. Using pre-tasks, a web server can be taken out of a load balancer, then the role is applied to the web servers to push updates, followed by post-tasks which register the web server back to the load balancer. Moreover, if these servers are being monitored by Nagios, alerts can be disabled during the update process and automatically enabled again using pre-tasks and post-tasks. This can avoid the noise that the monitoring tool may generate in the form of alerts.
Delegation
If you would like tasks to be selectively run on a certain class of hosts, especially the ones outside the current play, the delegation feature of Ansible can come in handy. This is relevant to the scenarios discussed previously and is commonly used with pre-tasks and post-tasks. For example, before updating a web server, it needs to be deregistered from the load balancer. Now, this task should be run on the load balancer, which is not part of the play. This dilemma can be solved by using the delegation feature. With pre-tasks, a script can be launched on the load balancer using the delegate_to keyword, which does the deregistering part as follows:
- name: deregister web server from lb
shell: < script to run on lb host >
delegate_to: lbIf there areis more than one load balancers, anan inventory group can be iterated over as, follows:
- name: deregister web server from lb
shell: < script to run on lb host >
delegate_to: "{{ item }}"
with_items: groups.lb
Rolling updates
This is also called batch updates or zero-downtime updates. Let’s assume that we have 100 web servers that need to be updated. If we define these in an inventory and launch a playbook against them, Ansible will start updating all the hosts in parallel. This can also cause downtime. To avoid complete downtime and have a seamless update, it would make sense to update them in batches, for example, 20 at a time. While running a playbook, batch size can be mentioned by using the serial keyword in the play. Let’s take a look at the following code snippet:
- hosts: www remote_user: vagrant sudo: yes serial: 20
Tests
While orchestrating, it’s not only essential to configure the applications in order, but also to ensure that they are actually started, and functioning as expected. Ansible modules, such as wait_for and uri, help you build that testing into the playbooks, for example:
- name: wait for mysql to be up
wait_for: host=db.example.org port=3106 state=started
- name: check if a uri returns content
uri: url=http://{{ inventory_hostname }}/api
register: apicheck
The wait_for module can be additionally used to test the existence of a file. It’s also useful when you would like to wait until a service is available before proceeding.
Tags
Ansible plays map roles to specific hosts. While the plays are run, the entire logic that is called from the main task is executed. While orchestrating, we may need to just run a part of the tasks based on the phases that we want to bring the infrastructure in. One example is a zookeeper cluster, where it’s important to bring up all the nodes in the cluster at the same time, or in a gap of a few seconds. Ansible can orchestrate this easily with a two-phase execution. In the first phase, you can install and configure the application on all nodes, but not start it. The second phase involves starting the application on all nodes almost simultaneously. This can be achieved by tagging individual tasks, for example, configure, install, service, and more.
For example, let’s take a look at the following screenshot:
While running a playbook, all tasks with a specific tag can be called using –-tags as follows:
$ Ansible-playbook -i customhosts site.yml –-tags install
Tags can not only be applied to tasks, but also to the roles, as follows:
{ role: nginx, when: Ansible_os_family == 'Debian', tags: 'www' }
If a specific task needs to be executed always, even if filtered with a tag, use a special tag called always. This will make the task execute unless an overriding option, such as –skip-tags always is used.
Patterns and limits
Limits can be used to run tasks on a subset of hosts, which are filtered by patterns. For example, the following code would run tasks only on hosts that are part of the db group:
$ Ansible-playbook -i customhosts site.yml --limit db
Patterns usually contain a group of hosts to include or exclude. A combination of more than one pattern can be specified as follows:
$ Ansible-playbook -i customhosts site.yml --limit db,lb
Having a colon as separator can be used to filter hosts further. The following command would run tasks on all hosts except for the ones that belong to the groups www and db:
$ Ansible-playbook -i customhosts site.yml --limit 'all:!www:!db'
Note that this usually needs to be enclosed in quotes. In this pattern, we used the all group, which matches all hosts in the inventory, and can be replaced with *. That was followed by ! to exclude hosts in the db group. The output of this command is as follows, which shows that plays by the name db and www were skipped as no hosts matched due to the filter we used previously:
Let’s now see these orchestration features in action. We will begin by tagging the role and do the multiphase execution followed by writing a new playbook to manage updates to the WordPress application.
Review questions
Do you think you’ve understood the article well enough? Try answering the following questions to test your understanding:
- What is idempotence when it comes to modules?
- What is the host’s inventory and why is it required?
- Playbooks map ___ to ___ (fill in the blanks)
- What types of patterns can you use while selecting a list of hosts to run plays against?
- Where is the actual procedure to execute an action on a specific platform defined?
- Why is it said that Ansible comes with batteries included?
Summary
In this article, you learned about what Ansible playbooks are, what components those are made up of, and how to blueprint your infrastructure with it. We also did a primer on YAML—the language used to create plays. You learned about how plays map tasks to hosts, how to create a host inventory, how to filter hosts with patterns, and how to use modules to perform actions on our systems. We then created a simple playbook as a proof of concept. We also learned about orchestration, using Ansible as an orchestrator and different tasks that we can perform using Ansible as an orchestrator.
Resources for Article:
Further resources on this subject:
- Advanced Playbooks [article]
- Ansible – An Introduction [article]
- Getting Started with Ansible [article]
