Puppet State of the Union: May 2017

The Elastic Stack is a far-reaching ecosystem with many interconnected parts, and the official Puppet Modules are no exception when it comes to the configuration management side of the house. Today, we want to highlight some developments taking place in the Elastic Puppet ecosystem to help spread awareness of new features and upcoming changes.

Modules Roadmap

First, some housekeeping items that pertain to all of our official modules.

Puppet 3

The most prominent upcoming change is the deprecation of Puppet 3 (if you’re skimming this post, don’t panic! You still have time). As noted in the upstream Puppet Labs page, version 3 met its end in December 2016, which means that it is not only deprecated, but end-of-life, and past the cutoff date for any future updates. As systems operators ourselves, we recognize that keeping stride with software versions is one of the challenges that comes with the discipline, particularly when it comes to large enterprises.

With that in mind, we’ll be keeping version 3.8 of Puppet in our list of supported versions for a period to ensure everyone has a window of time to work in, for those cases where upgrading is a challenge. Nevertheless, the Puppet ecosystem is moving on from version 3, and we can only hold on to old versions for so long. It’s for this reason that we will be removing support for Puppet 3 with the 6.x release of the Elastic Stack, which grants many months of leeway for those operations engineers with a Puppet upgrade still on their plate. Version 6 of the Elastic Stack is likely to arrive in late 2017, and will be prefaced with additional announcements as its release draws closer.

New Forge Namespace

What open source project doesn’t like changing names every so often? Our modules on the Puppet Forge now properly live under the “elastic” namespace where all future module updates and release will land. While any references to old modules under the elasticsearch namespace will continue to work, the modules there are no longer updated. We suggest updating any automation or tooling that references released modules from the Forge to this new namespace (for example, to elastic/elasticsearch).

Official Puppet Kibana Module

One of the most common requests we hear from Puppet users is for other parts of our stack to ship official modules. At Elastic{ON} 2017 we released a beta version of an official Puppet Kibana module, which is now formally released at version 5.0 (major version synchronized with the rest of the Elastic Stack). The module supports everything you would expect from our Puppet modules, including Kibana plugin resources to make managing Kibana through Puppet a much easier experience.

As always, we’re eager to hear from you regarding any bugs you've found or features you’d like to see and would encourage any bug reports or feature requests to be filed in the puppet-kibana GitHub repository.

Elasticsearch: Tasty New Features

The official Elasticsearch module hasn’t stood still either. In addition to the steady march of bugfixes, the official Elasticsearch module supports a couple of useful new resource types: indices and pipelines.

While the module already supported managing index templates (useful for auto-created indices from tools like Logstash or Beats), the elasticsearch::index type can help manage mappings and settings for existing indices in an Elasticsearch cluster. For example, to manage an Elasticsearch index called “logs” to retain only 1 replica per shard:

elasticsearch::index { 'logs':
  settings => {
    'index' => {
      'number_of_replicas' => 1
    }
  }
}

Likewise, pipelines are now supported as well. The ability to codify processing pipelines in Puppet can help centralize your workflow for processing logs and documents in Elasticsearch (see our other blog posts about ingest node pipelines for more introductory content). For example, to create a pipeline that will automatically parse dates, the following Puppet resource could be used:

elasticsearch::pipeline { ‘parsedate’:
  content => {
    'description' => 'Parse the date field into @timestamp',
    'processors' => [{
      'date' => {
        'field' => 'date',
        'formats' => ['dd/MM/yyyy hh:mm:ss'],
        'timezone' => 'US/Pacific'
      }
    }]
  }
}

Logstash: All Aboard for Version 5

Version 5 of Logstash brought huge improvements to the handling of the Logstash system service, adding full support for systemd, among other things. In the past, the Puppet module was responsible for much more of the work around service handling, which was a common source of problems as the module became progressively more complex in order to handle various edge-cases across distributions. With Logstash itself now handling the service, the Puppet module is much simpler and more stable when it comes to service management.

It's important to note, however, that the changes in service management were so comprehensive that it became impractical for the module to support Logstash 2.x and Logstash 5.x at the same time. Thus, module version 5 only supports Logstash 5. Of course, the legacy module elasticsearch/logstash is still available if you need to run Logstash 2.

It's now possible to fully manage Logstash settings, startup options, and JVM options with the module. Like this:

class { 'logstash':
  settings => {
    'pipeline' => {
      'batch' => {
        'size' => 25,
        'delay => 5,
      }
    }
  }
}

Since the settings are all defined as standard arrays and hashes, they work well with Hiera too.

Plugin support is now comprehensive, with support for remote locations like puppet:// and http:// URLs. The latter is essential for people choosing to install the new X-Pack for Logstash.

Finally, a lot of progress on the module is something you probably won't see, but hopefully you get to see the results. All new features, including those above, are now developed under testing from a re-written test suite. The tests are there to ensure everything in the module works reliably, across every supported operating system and Puppet version.