“Hot-Warm” architecture

The current blog applies to Elasticsearch versions 1.x and 2.x. If you are looking for “Hot-Warm” Architecture in Elasticsearch 5.x click here.

When using Elasticsearch for larger time-data analytics use cases, we recommend using time-based indices and a tiered architecture with 3 different types of nodes (Master, Hot-Node and Warm-Node), which we refer to as the "Hot-Warm" architecture.

Each node has their own characteristics, which are described below.

Master nodes

We recommend running 3 dedicated master nodes per cluster having dedicated master nodes which run in their own JVM increases stability and resilience as they are not affected by garbage collection that can affect other types of nodes. These nodes do not handle requests and do not hold any data, and therefore only require less resources (such as CPU, RAM and Disk)

Hot data nodes

Hot data nodes are designed to perform all indexing within the cluster and will also hold the most recent daily indices that generally tend to be queried most frequently. As indexing is a very IO intensive operation, these servers need to be powerful and backed by attached SSD storage.

Warm data nodes

Warm data nodes are designed to handle a large amount of read-only indices that are not queried frequently. As these indices are read-only, warm nodes tend to utilise very large attached spinning disks instead of SSDs.

Elasticsearch needs to know which servers contain the hot nodes and which server contain the warm nodes. This can be achieved by assigning arbitrary tags to each server.

For instance, you could tag the node with node.box_type: hot in elasticsearch.yml, or you could start a node using ./bin/elasticsearch --node.box_type hot.

And the nodes on the warm zone are "tagged" with node.box_type: warm in elasticsearch.yml or you could start a node using ./bin/elasticsearch --node.box_type warm

The box_type parameter is completely arbitrary and you could name it whatever you like. These arbitrary values will be used to tell Elasticsearch where to allocate an index.

We can ensure that today’s index is on our SSD boxes by creating it with the following settings:

PUT /logs_2015-08-31
{
  "settings": {
    "index.routing.allocation.require.box_type" : "hot"
  }
}

After few days if the index no longer needs to be on our strongest boxes, we can move it to the nodes tagged as warm by updating its index settings:

PUT /logs_2014-08-31/_settings
{
  "index.routing.allocation.require.box_type" : "warm"
}

Now how can we achieve that using Logstash :

We need update the index templates to include allocation filtering "index.routing.allocation.require.box_type" : "hot" so that any new indices are created on the nodes that are tagged with hot.

Example:

{
  "template": "logstash-*",
  "settings": {
    "index.refresh_interval": "5s",
    "index.routing.allocation.require.box_type": "hot"
  },
  "mappings": {
    "_default_": {
      "_all": {
        "enabled": true,
        "omit_norms": true
      },
      "dynamic_templates": [
        {
          "message_field": {
            "match": "message",
            "match_mapping_type": "string",
            "mapping": {
              "type": "string",
              "index": "analyzed",
              "omit_norms": true
            }
          }
        },
        {
          "string_fields": {
            "match": "*",
            "match_mapping_type": "string",
            "mapping": {
              "type": "string",
              "index": "analyzed",
              "omit_norms": true,
              "fields": {
                "raw": {
                  "type": "string",
                  "index": "not_analyzed",
                  "ignore_above": 256
                }
              }
            }
          }
        }
      ],
      "properties": {
        "@version": {
          "type": "string",
          "index": "not_analyzed"
        },
        "geoip": {
          "type": "object",
          "dynamic": true,
          "properties": {
            "location": {
              "type": "geo_point"
            }
          }
        }
      }
    }
  }
}

When indexes are no longer being written to and not being searched frequently they no longer need to be on the hot nodes. We can move them to the nodes tagged as warm by updating their index settings: "index.routing.allocation.require.box_type" : "warm".

Elasticsearch will automatically migrate the indices over to the warm nodes. Finally we recommend running an optimize on the migrated indices. It would be a bad idea to optimize the index while it was still allocated to the strong boxes, as the optimization process could swamp the I/O on those nodes and impact the indexing of today’s logs. But the medium boxes aren’t doing very much, so we are safe to optimize.

Now that we have seen how to change the allocation of an index, let's look at how to use Curator to automate the process.

In the example below we are using Curator 3.0 to move the indexes from hot nodes to warm nodes after 3 days.

/usr/bin/curator --logfile /var/log/curator.log --loglevel INFO --logformat default --master-only --host localhost --port 9200 allocation --rule box_type=warm indices --time-unit days --older-than 3 --timestring '%Y.%m.%d'

Finally we can use Curator to optimize the index, make sure you wait enough for the reallocation to finish before running optimize.

/usr/bin/curator --host localhost --port 9200 optimize indices --older-than 3  --time-unit days  --timestring '%Y.%m.%d'