Deploying Elasticsearch with Ansible

With the recent release of Elasticsearch 2.0.0 a completely new Ansible role has been released as well. This post will show you how to install and configure multiple Elasticsearch nodes using this new role.

As Consulting Engineers at Elastic we regularly deploy large Elasticsearch clusters across multiple machines. These clusters vary in topology and configuration and are often deployed on heterogeneous hardware in complex architectures. Repeating the same task of software deployment and configuration, on potentially hundreds of machines per customer, is a great opportunity for automation. In the interests of both efficiency and accuracy — as well as our sanity! — we regularly review and seek new tools to simplify this process. By minimizing the time spent on actual software deployment, we maximize the time for more interesting problems that our customers regularly pose!

We continue to support a Puppet module which was recently updated to support Elasticsearch 2.x. Whilst an excellent configuration and orchestration management tool, Puppet can present a steep learning curve and significant investment for some organizations. For deploying new self-contained clusters — with no history of Puppet within the organization — Ansible's push architecture can represent a simpler alternative. Ansible is a Python-based automation tool that wraps SSH, allowing complex idempotent commands to be sequenced. It can configure systems, deploy software, and orchestrate more advanced IT tasks such as continuous deployments or zero-downtime rolling updates. With no requirement for an agent, or external dependencies other than SSH access and Python 2 on the target host, it represents an ideal solution for deploying an Elasticsearch cluster on a set of newly provisioned servers. Whilst Puppet remains a popular configuration management tool for Linux systems, Ansible has obtained equivalent interest in the last few years. In response to both our own requirements and a growing community interest, we are now officially releasing an Ansible role to simplify the deployment of Elasticsearch.

The role's structure directly maps to its key capabilities: installation of prerequisites (e.g. Java), installation of Elasticsearch based on the target platform, Elasticsearch configuration, management of plugins and finally service management. The assignment of the role to a host results in the installation of an Elasticsearch instance. For hosts where multiple instances are required, simply assign the role multiple times — as illustrated in the below example. The configuration of Elasticsearch is supported through a map, which is in turn serialized to yaml, thus requiring no changes when new parameters are added.

To help with illustrating the capabilities of the role we've assembled a quick example using an Ubuntu-based docker image. The user will require:

• A desktop/laptop of reasonable specification with an an installation of Docker > v1.9.0 or Docker-machine using Windows/OS X. This will act as the “jump” machine from which all software will be deployed.
• Ansible version 1.9.4 or later installed on a Linux/OS X-based machine from which the playbook will be executed.

The following assumes Ansible has been installed in /opt/ansible on your desktop. Where not specified, the user should assume all actions should be performed from this machine.

1. Navigate to the root installation folder to Ansible and create a playbook directory with your user permissions.

sudo mkdir -p /opt/ansible/playbooks/
sudo chown -R <User> /opt/ansible/playbooks

2. Change to the playbooks directory and clone the provided playbook examples.

cd /opt/ansible/playbooks && git clone https://github.com/elastic/ansible-elasticsearch-example.git

3. Clone the Elasticsearch role into a roles subdirectory beneath the example.

cd /opt/ansible/playbooks/ansible-elasticsearch-example/git 
clone https://github.com/elastic/ansible-elasticsearch.git roles/elasticsearch

The file playbook.yml declares a very simple playbook with one play. This play applies the Elasticsearch role twice to a server belonging to the “nodes” group. The first application of the role will install a dedicated master node, whilst the second installs a data node. Users can easily modify this file to contain either more plays, more roles or a combination of both.

4. Install the docker image gingerwizard/ansible-test. This image is provided for the purposes of this blog and is not an official image.

dockerpull gingerwizard/ansible-test

5. Run a docker container for the image.

docker run -d -P --name ansible-test -h ansible-test gingerwizard/ansible-test

6. Note the IP address, SSH port and Elasticsearch http ports for the image “gingerwizard/ansible-test“. Both 9200 (master node) and 9201 (data node) are exposed. Note the container id in the response of the first command, for use in the following inspect command.

docker ps -a
docker inspect <container_id> | grep "IPAddress"

7. Prior to running the playbook the user is required to make some modifications to the provided inventory file. This file declares the details of the docker image on which Elasticsearch will be installed. Edit the “hosts” file with the appropriate IP and SSH port.

[nodes]
ansible-test ansible_ssh_host=<IP Address> ansible_ssh_port=<SSH Port>

Although the role installs Java where required, the provided docker image includes this to keep the deployment time minimal.

8. Deploy the Elasticsearch to the container!

ansible-playbook playbook.yml -u root -i hosts --ask-pass

The user will be prompted for the root password -'Ansible!'

The play should complete with:

PLAY RECAP *********************************************************************
ansible-test               : ok=69   changed=25   unreachable=0    failed=0

9. Once complete you can test Elasticsearch has started by querying the ip address and http ports (noted in step 8.) of both the master and data node.

curl http://<IPAddress>:<Port>/

The playbook illustrates some of the available configuration parameters available to the user. For further details and features refer to the provided readme.

Over the coming weeks we expect to release Ansible roles for both the Beats products and Logstash, whilst also continuing to add features to the Elasticsearch role, with the aim of being able to deploy the complete Elastic stack with a single command. Per usual, we welcome community contributions and feedback!