How to write a Logstash filter plugin

edit

How to write a Logstash filter plugin

edit

To develop a new output for Logstash, you build a self-contained Ruby gem whose source code lives in its own GitHub repository. The Ruby gem can then be hosted and shared on RubyGems.org. You can use the example output implementation as a starting point. (If you’re unfamiliar with Ruby, you can find an excellent quickstart guide at https://www.ruby-lang.org/en/documentation/quickstart/.)

As of Logstash 1.5, all plugins are self-contained Ruby gems. This change makes it possible to develop and release plugins separately. In previous versions, plugins were part of the core Logstash distribution.

Get started

edit

Let’s step through creating an output plugin using the example output plugin.

Create a GitHub repo for your new plugin

edit

Each Logstash plugin lives in its own GitHub repository. To create a new repository for your plugin:

  1. Log in to GitHub.
  2. Click the Repositories tab. You’ll see a list of other repositories you’ve forked or contributed to.
  3. Click the green New button in the upper right.
  4. Specify the following settings for your new repo:

    • Repository name — a unique name of the form logstash-output-pluginname.
    • Public or Private — your choice, but the repository must be Public if you want to submit it as an official plugin.
    • Initialize this repository with a README — enables you to immediately clone the repository to your computer.
  5. Click Create Repository.

Copy the output code

edit

Build your local repository:

  1. Clone your plugin. Replace GITUSERNAME with your github username, and MYPLUGINNAME with your plugin name.

    • git clone https://github.com/GITUSERNAME/logstash-output-MYPLUGINNAME.git

      • alternately, via ssh: git clone git@github.com:GITUSERNAME/logstash-output-MYPLUGINNAME.git
    • cd logstash-output-MYPLUGINNAME
  2. Clone the {inputtype} plugin example and copy it to your plugin branch.

    You don’t want to include the example .git directory or its contents, so delete it before you copy the example.

    • cd /tmp
    • git clone https://github.com/logstash-plugins/logstash-output-example.git
    • cd logstash-output-example
    • rm -rf .git
    • cp -R * /path/to/logstash-output-mypluginname/
  3. Rename the following files to match the name of your plugin.

    • logstash-output-example.gemspec
    • example.rb
    • example_spec.rb

      cd /path/to/logstash-output-mypluginname
      mv logstash-output-example.gemspec logstash-output-mypluginname.gemspec
      mv lib/logstash/outputs/example.rb lib/logstash/outputs/mypluginname.rb
      mv spec/outputs/example_spec.rb spec/outputs/mypluginname_spec.rb

Your file structure should look like this:

$ tree logstash-output-mypluginname
├── Gemfile
├── LICENSE
├── README.md
├── Rakefile
├── lib
│   └── logstash
│       └── outputs
│           └── mypluginname.rb
├── logstash-output-mypluginname.gemspec
└── spec
    └── outputs
        └── mypluginname_spec.rb

For more information about the Ruby gem file structure and an excellent walkthrough of the Ruby gem creation process, see http://timelessrepo.com/making-ruby-gems

See what your plugin looks like

edit

Before we dive into the details, open up the plugin file in your favorite text editor and take a look.

# encoding: utf-8
require "logstash/outputs/base"
require "logstash/namespace"

# Add any asciidoc formatted documentation here
# An example output that does nothing.
class LogStash::Outputs::Example < LogStash::Outputs::Base
  config_name "example"

  # If declared logstash will only allow a single instance of this plugin
  # to exist, regardless of how many CPU cores logstash detects. This is best
  # used in cases like the File output, where separate threads writing to a single
  # File would only cause problems.
  #
  # respond_to? check needed for backwards compatibility with < 2.2 Logstashes
  declare_workers_not_supported! if self.respond_to?(:declare_workers_not_supported!)

  # If declared threadsafe logstash will only ever create one
  # instance of this plugin per pipeline.
  # That instance will be shared across all workers
  # It is up to the plugin author to correctly write concurrent code!
  #
  # respond_to? check needed for backwards compatibility with < 2.2 Logstashes
  declare_threadsafe! if self.respond_to?(:declare_threadsafe!)

  public
  def register
    # Does the same thing as declare_workers_not_supported!
    # But works in < 2.2 logstashes
    # workers_not_supported
  end # def register

  public
  # Takes an array of events
  def multi_receive(events)
  end # def multi_receive

  public
  # Needed for logstash < 2.2 compatibility
  # Takes events one at a time
  def receive(event)
  end # def receive

end # class LogStash::Outputs::Example

Coding output plugins

edit

Now let’s take a line-by-line look at the example plugin.

encoding

edit

It seems like a small thing, but remember to specify the encoding at the beginning of your plugin code:

# encoding: utf-8

Logstash depends on things being in UTF-8, so we put this here to tell the Ruby interpreter that we’re going to be using the UTF-8 encoding.

require Statements

edit

Logstash output plugins require parent classes defined in logstash/outputs/base and logstash/namespace:

require "logstash/outputs/base"
require "logstash/namespace"

Of course, the plugin you build may depend on other code, or even gems. Just put them here along with these Logstash dependencies.

Plugin Body

edit

Let’s go through the various elements of the plugin itself.

Inline Documentation

edit

Logstash provides infrastructure to automatically generate documentation for plugins. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html.

All plugin documentation is then rendered and placed in the Logstash section of the Elasticsearch Guide.

The inline documentation can include code blocks and config examples! To include Ruby code, use the asciidoc [source,ruby] directive:

# Using hashes:
# [source,ruby]
# ----------------------------------
# match => {
#  "field1" => "value1"
#  "field2" => "value2"
#  ...
# }
# ----------------------------------

In the rendered HTML document, this block would look like:

Using hashes:

match => {
  "field1" => "value1"
  "field2" => "value2"
  ...
 }

For more asciidoc formatting tips, see the excellent reference at https://github.com/elastic/docs#asciidoc-guide

class Declaration

edit

The output plugin class should be a subclass of LogStash::Outputs::Base:

class LogStash::Outputs::Example < LogStash::Outputs::Base

The class name should closely mirror the plugin name, for example:

LogStash::Outputs::Example

config_name

edit
  config_name "example"

This is the name your plugin will call inside the output configuration block.

If you set config_name "example" in your plugin code, the corresponding Logstash configuration block would need to look like this:

Configuration Parameters

edit
  config :variable_name, :validate => :variable_type, :default => "Default value", :required => boolean, :deprecated => boolean, :obsolete => string

The configuration, or config section allows you to define as many (or as few) parameters as are needed to enable Logstash to process events.

There are several configuration attributes:

  • :validate - allows you to enforce passing a particular data type to Logstash for this configuration option, such as :string, :password, :boolean, :number, :array, :hash, :path (a file-system path), :codec (since 1.2.0), :bytes (starting in 1.5.0). Note that this also works as a coercion in that if I specify "true" for boolean (even though technically a string), it will become a valid boolean in the config. This coercion works for the :number type as well where "1.2" becomes a float and "22" is an integer.
  • :default - lets you specify a default value for a parameter
  • :required - whether or not this parameter is mandatory (a Boolean true or false)
  • :deprecated - informational (also a Boolean true or false)
  • :obsolete - used to declare that a given setting has been removed and is no longer functioning. The idea is to provide an informed upgrade path to users who are still using a now-removed setting.

Plugin Methods

edit

Logstash outputs must implement the `register` and `multi_receive` methods.

register Method

edit
  public
  def register
  end # def register

The Logstash register method is like an initialize method. It was originally created to enforce having super called, preventing headaches for newbies. (Note: It may go away in favor of initialize, in conjunction with some enforced testing to ensure super is called.)

public means the method can be called anywhere, not just within the class. This is the default behavior for methods in Ruby, but it is specified explicitly here anyway.

You can also assign instance variables here (variables prepended by @). Configuration variables are now in scope as instance variables, like @message

Building the Plugin

edit

At this point in the process you have coded your plugin and are ready to build a Ruby Gem from it. The following steps will help you complete the process.

External dependencies

edit

A require statement in Ruby is used to include necessary code. In some cases your plugin may require additional files. For example, the collectd plugin uses the types.db file provided by collectd. In the main directory of your plugin, a file called vendor.json is where these files are described.

The vendor.json file contains an array of JSON objects, each describing a file dependency. This example comes from the collectd codec plugin:

[{
        "sha1": "a90fe6cc53b76b7bdd56dc57950d90787cb9c96e",
        "url": "http://collectd.org/files/collectd-5.4.0.tar.gz",
        "files": [ "/src/types.db" ]
}]
  • sha1 is the sha1 signature used to verify the integrity of the file referenced by url.
  • url is the address from where Logstash will download the file.
  • files is an optional array of files to extract from the downloaded file. Note that while tar archives can use absolute or relative paths, treat them as absolute in this array. If files is not present, all files will be uncompressed and extracted into the vendor directory.

Another example of the vendor.json file is the geoip filter

The process used to download these dependencies is to call rake vendor. This will be discussed further in the testing section of this document.

Another kind of external dependency is on jar files. This will be described in the "Add a gemspec file" section.

Add a Gemfile

edit

Gemfiles allow Ruby’s Bundler to maintain the dependencies for your plugin. Currently, all we’ll need is the Logstash gem, for testing, but if you require other gems, you should add them in here.

See Bundler’s Gemfile page for more details.

source 'https://rubygems.org'
gemspec
gem "logstash", :github => "elastic/logstash", :branch => "2.3"

Add a gemspec file

edit

Gemspecs define the Ruby gem which will be built and contain your plugin.

More information can be found on the Rubygems Specification page.

Gem::Specification.new do |s|
  s.name = 'logstash-output-example'
  s.version = '0.1.0'
  s.licenses = ['Apache License (2.0)']
  s.summary = "This output does x, y, z in Logstash"
  s.description = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
  s.authors = ["Elastic"]
  s.email = 'info@elastic.co'
  s.homepage = "http://www.elastic.co/guide/en/logstash/current/index.html"
  s.require_paths = ["lib"]

  # Files
  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
   # Tests
  s.test_files = s.files.grep(%r{^(test|spec|features)/})

  # Special flag to let us know this is actually a logstash plugin
  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "output" }

  # Gem dependencies
  s.add_runtime_dependency 'logstash-core', '>= 1.4.0', '< 2.0.0'
  s.add_development_dependency 'logstash-devutils'
end

It is appropriate to change these values to fit your plugin. In particular, s.name and s.summary shoud reflect your plugin’s name and behavior.

s.licenses and s.version are also important and will come into play when you are ready to publish your plugin.

Logstash and all its plugins are licensed under Apache License, version 2 ("ALv2"). If you make your plugin publicly available via RubyGems.org, please make sure to have this line in your gemspec:

  • s.licenses = ['Apache License (2.0)']

The gem version, designated by s.version, helps track changes to plugins over time.

Version messaging from Logstash

If you start Logstash with the --verbose flag, you will see messages like these to indicate the relative maturity indicated by the plugin version number:

  • 0.1.x

    This plugin isn't well supported by the community and likely has no maintainer.
  • 0.9.x

    This plugin should work but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin.
  • 1.x.x You will no longer see a message indicating potential code immaturity when a plugin reaches version 1.0.0

Runtime & Development Dependencies

edit

At the bottom of the gemspec file is a section with a comment: Gem dependencies. This is where any other needed gems must be mentioned. If a gem is necessary for your plugin to function, it is a runtime dependency. If a gem are only used for testing, then it would be a development dependency.

You can also have versioning requirements for your dependencies—​including other Logstash plugins:

  # Gem dependencies
  s.add_runtime_dependency 'logstash-core', '>= 1.4.0', '< 2.0.0'
  s.add_development_dependency 'logstash-devutils'

This gemspec has a runtime dependency on the core Logstash gem and requires that it have a version number greater than or equal to version 1.4.0 '>= 1.4.0', and less than version 2.0 '< 2.0.0'.

All plugins have a runtime dependency on the logstash core gem, and a development dependency on logstash-devutils.

Jar dependencies

edit

In some cases, such as the Elasticsearch output plugin, your code may depend on a jar file. In cases such as this, the dependency is added in the gemspec file in this manner:

  # Jar dependencies
  s.requirements << "jar 'org.elasticsearch:elasticsearch', '1.4.0'"
  s.add_runtime_dependency 'jar-dependencies'

With these both defined, the install process will search for the required jar file at http://mvnrepository.com and download the specified version.

Add Tests

edit

Logstash loves tests. Lots of tests. If you’re using your new output plugin in a production environment, you’ll want to have some tests to ensure you are not breaking any existing functionality.

A full exposition on RSpec is outside the scope of this document. Learn more about RSpec at http://rspec.info

For help learning about tests and testing, look in the spec/outputs/ directory of several other similar plugins.

Clone and test!

edit

Now let’s start with a fresh clone of the plugin, build it and run the tests.

  • Clone your plugin into a temporary location Replace GITUSERNAME with your github username, and MYPLUGINNAME with your plugin name.

    • git clone https://github.com/GITUSERNAME/logstash-output-MYPLUGINNAME.git

      • alternately, via ssh: git clone git@github.com:GITUSERNAME/logstash-output-MYPLUGINNAME.git
    • cd logstash-output-MYPLUGINNAME

Then, you’ll need to install your plugins dependencies with bundler:

bundle install

If your plugin has an external file dependency described in vendor.json, you must download that dependency before running or testing. You can do this by running:

rake vendor

And finally, run the tests:

bundle exec rspec

You should see a success message, which looks something like this:

Finished in 0.034 seconds
1 example, 0 failures

Hooray! You’re almost there! (Unless you saw failures…​ you should fix those first).

Building and Testing

edit

Now you’re ready to build your (well-tested) plugin into a Ruby gem.

Build

edit

You already have all the necessary ingredients, so let’s go ahead and run the build command:

gem build logstash-output-example.gemspec

That’s it! Your gem should be built and be in the same path with the name

logstash-output-mypluginname-0.1.0.gem

The s.version number from your gemspec file will provide the gem version, in this case, 0.1.0.

Test installation

edit

You should test install your plugin into a clean installation of Logstash. Download the latest version from the Logstash downloads page.

  1. Untar and cd in to the directory:

    curl -O https://download.elastic.co/logstash/logstash/logstash-2.3.0.tar.gz
    tar xzvf logstash-2.3.0.tar.gz
    cd logstash-2.3.0
  2. Using the plugin tool, we can install the gem we just built.

    • Replace /my/logstash/plugins with the correct path to the gem for your environment, and 0.1.0 with the correct version number from the gemspec file.

      bin/logstash-plugin install /my/logstash/plugins/logstash-output-example/logstash-output-example-0.1.0.gem
    • After running this, you should see feedback from Logstash that it was successfully installed:

      validating /my/logstash/plugins/logstash-output-example/logstash-output-example-0.1.0.gem >= 0
      Valid logstash plugin. Continuing...
      Successfully installed 'logstash-output-example' with version '0.1.0'

      You can also use the Logstash plugin tool to determine which plugins are currently available:

      bin/logstash-plugin list

      Depending on what you have installed, you might see a short or long list of plugins: inputs, codecs, filters and outputs.

  3. Now try running Logstash with a simple configuration passed in via the command-line, using the -e flag.

    Your results will depend on what your output plugin is designed to do.

Congratulations! You’ve built, deployed and successfully run a Logstash output.

Submitting your plugin to RubyGems.org and logstash-plugins

edit

Logstash uses RubyGems.org as its repository for all plugin artifacts. Once you have developed your new plugin, you can make it available to Logstash users by simply publishing it to RubyGems.org.

Licensing

edit

Logstash and all its plugins are licensed under Apache License, version 2 ("ALv2"). If you make your plugin publicly available via RubyGems.org, please make sure to have this line in your gemspec:

  • s.licenses = ['Apache License (2.0)']

Publishing to RubyGems.org

edit

To begin, you’ll need an account on RubyGems.org

After creating an account, obtain an API key from RubyGems.org. By default, RubyGems uses the file ~/.gem/credentials to store your API key. These credentials will be used to publish the gem. Replace username and password with the credentials you created at RubyGems.org:

curl -u username:password https://rubygems.org/api/v1/api_key.yaml > ~/.gem/credentials
chmod 0600 ~/.gem/credentials

Before proceeding, make sure you have the right version in your gemspec file and commit your changes.

  • s.version = '0.1.0'

To publish version 0.1.0 of your new logstash gem:

bundle install
bundle exec rake vendor
bundle exec rspec
bundle exec rake publish_gem

Executing rake publish_gem:

  1. Reads the version from the gemspec file (s.version = '0.1.0')
  2. Checks in your local repository if a tag exists for that version. If the tag already exists, it aborts the process. Otherwise, it creates a new version tag in your local repository.
  3. Builds the gem
  4. Publishes the gem to RubyGems.org

That’s it! Your plugin is published! Logstash users can now install your plugin by running:

bin/logstash-plugin install logstash-output-mypluginname

Contributing your source code to logstash-plugins

edit

It is not required to contribute your source code to logstash-plugins github organization, but we always welcome new plugins!

Benefits

edit

Some of the many benefits of having your plugin in the logstash-plugins repository are:

  • Discovery Your plugin will appear in the Logstash Reference, where Logstash users look first for plugins and documentation.
  • Documentation Your plugin documentation will automatically be added to the Logstash Reference.
  • Testing With our testing infrastructure, your plugin will be continuously tested against current and future releases of Logstash. As a result, users will have the assurance that if incompatibilities arise, they will be quickly discovered and corrected.

Acceptance Guidelines

edit
  • Code Review Your plugin must be reviewed by members of the community for coherence, quality, readability, stability and security.
  • Tests Your plugin must contain tests to be accepted. These tests are also subject to code review for scope and completeness. It’s ok if you don’t know how to write tests — we will guide you. We are working on publishing a guide to creating tests for Logstash which will make it easier. In the meantime, you can refer to http://betterspecs.org/ for examples.

To begin migrating your plugin to logstash-plugins, simply create a new issue in the Logstash repository. When the acceptance guidelines are completed, we will facilitate the move to the logstash-plugins organization using the recommended github process.