How to write a Logstash codec plugin

Each Logstash plugin lives in its own GitHub repository. To create a new repository for your plugin:

You can now create your own Logstash plugin in seconds! The generate subcommand of bin/logstash-plugin creates the foundation for a new Logstash plugin with templatized files. It creates the correct directory structure, gemspec files, and dependencies so you can start adding custom code to process data with Logstash.

For more information, see Generating Plugins

Alternatively, you can use the examples repo we host on github.com

Your file structure should look like this:

$ tree logstash-codec-mypluginname
├── Gemfile
├── LICENSE
├── README.md
├── Rakefile
├── lib
│   └── logstash
│       └── codecs
│           └── mypluginname.rb
├── logstash-codec-mypluginname.gemspec
└── spec
    └── codecs
        └── mypluginname_spec.rb

For more information about the Ruby gem file structure and an excellent walkthrough of the Ruby gem creation process, see http://timelessrepo.com/making-ruby-gems

Before we dive into the details, open up the plugin file in your favorite text editor and take a look.

require "logstash/codecs/base"
require "logstash/codecs/line"

# Add any asciidoc formatted documentation here
class LogStash::Codecs::Example < LogStash::Codecs::Base

  # This example codec will append a string to the message field
  # of an event, either in the decoding or encoding methods
  #
  # This is only intended to be used as an example.
  #
  # input {
  #   stdin { codec => example }
  # }
  #
  # or
  #
  # output {
  #   stdout { codec => example }
  # }
  config_name "example"

  # Append a string to the message
  config :append, :validate => :string, :default => ', Hello World!'

  public
  def register
    @lines = LogStash::Codecs::Line.new
    @lines.charset = "UTF-8"
  end

  public
  def decode(data)
    @lines.decode(data) do |line|
      replace = { "message" => line["message"].to_s + @append }
      yield LogStash::Event.new(replace)
    end
  end # def decode

  public
  def encode(event)
    @on_event.call(event, event.get("message").to_s + @append + NL)
  end # def encode

end # class LogStash::Codecs::Example

Logstash codec plugins require parent classes defined in logstash/codecs/base and logstash/namespace:

require "logstash/codecs/base"
require "logstash/namespace"

Of course, the plugin you build may depend on other code, or even gems. Just put them here along with these Logstash dependencies.

The codec plugin class should be a subclass of LogStash::Codecs::Base:

class LogStash::Codecs::Example < LogStash::Codecs::Base

The class name should closely mirror the plugin name, for example:

LogStash::Codecs::Example

  config_name "example"

This is the name your plugin will call inside the codec configuration block.

If you set config_name "example" in your plugin code, the corresponding Logstash configuration block would need to look like this:

  public
  def register
  end # def register

The Logstash register method is like an initialize method. It was originally created to enforce having super called, preventing headaches for newbies. (Note: It may go away in favor of initialize, in conjunction with some enforced testing to ensure super is called.)

public means the method can be called anywhere, not just within the class. This is the default behavior for methods in Ruby, but it is specified explicitly here anyway.

You can also assign instance variables here (variables prepended by @). Configuration variables are now in scope as instance variables, like @message

  public
  def decode(data)
    @lines.decode(data) do |line|
      replace = { "message" => line["message"].to_s + @append }
      yield LogStash::Event.new(replace)
    end
  end # def decode

The codec’s decode method is where data coming in from an input is transformed into an event. There are complex examples like the collectd codec, and simpler examples like the spool codec.

There must be a yield statement as part of the decode method which will return decoded events to the pipeline.

  public
  def encode(event)
    @on_event.call(event, event.get("message").to_s + @append + NL)
  end # def encode

The encode method takes an event and serializes it (encodes) into another format. Good examples of encode methods include the simple plain codec, the slightly more involved msgpack codec, and even an avro codec.

In most cases, your encode method should have an @on_event.call() statement. This call will output data per event in the described way.

A require statement in Ruby is used to include necessary code. In some cases your plugin may require additional files. For example, the collectd plugin uses the types.db file provided by collectd. In the main directory of your plugin, a file called vendor.json is where these files are described.

The vendor.json file contains an array of JSON objects, each describing a file dependency. This example comes from the collectd codec plugin:

[{
        "sha1": "a90fe6cc53b76b7bdd56dc57950d90787cb9c96e",
        "url": "http://collectd.org/files/collectd-5.4.0.tar.gz",
        "files": [ "/src/types.db" ]
}]

Another example of the vendor.json file is the geoip filter

The process used to download these dependencies is to call rake vendor. This will be discussed further in the testing section of this document.

Another kind of external dependency is on jar files. This will be described in the "Add a gemspec file" section.

As a plugin evolves, an option or feature may no longer serve the intended purpose, and the developer may want to deprecate its usage. Deprecation warns users about the option’s status, so they aren’t caught by surprise when it is removed in a later release.

Logstash 7.6 introduced a deprecation logger to make handling those situations easier. You can use the adapter to ensure that your plugin can use the deprecation logger while still supporting older versions of Logstash. See the readme for more information and for instructions on using the adapter.

Deprecations are noted in the logstash-deprecation.log file in the log directory.

Gemfiles allow Ruby’s Bundler to maintain the dependencies for your plugin. Currently, all we’ll need is the Logstash gem, for testing, but if you require other gems, you should add them in here.

source 'https://rubygems.org'
gemspec
gem "logstash", :github => "elastic/logstash", :branch => "7.10"

At the bottom of the gemspec file is a section with a comment: Gem dependencies. This is where any other needed gems must be mentioned. If a gem is necessary for your plugin to function, it is a runtime dependency. If a gem are only used for testing, then it would be a development dependency.

  # Gem dependencies
  s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
  s.add_development_dependency 'logstash-devutils'

In some cases, such as the Elasticsearch output plugin, your code may depend on a jar file. In cases such as this, the dependency is added in the gemspec file in this manner:

  # Jar dependencies
  s.requirements << "jar 'org.elasticsearch:elasticsearch', '5.0.0'"
  s.add_runtime_dependency 'jar-dependencies'

With these both defined, the install process will search for the required jar file at http://mvnrepository.com and download the specified version.

You already have all the necessary ingredients, so let’s go ahead and run the build command:

gem build logstash-codec-example.gemspec

That’s it! Your gem should be built and be in the same path with the name

logstash-codec-mypluginname-0.1.0.gem

The s.version number from your gemspec file will provide the gem version, in this case, 0.1.0.

You should test install your plugin into a clean installation of Logstash. Download the latest version from the Logstash downloads page.

bin/logstash -e 'input { stdin{ codec => example{}} } output {stdout { codec => rubydebug }}'

The example codec plugin will append the contents of append (which by default appends ", Hello World!")

After starting Logstash, type something, for example "Random output string". The resulting output message field contents should be, "Random output string, Hello World!":

Random output string
{
       "message" => "Random output string, Hello World!",
      "@version" => "1",
    "@timestamp" => "2015-01-27T19:17:18.932Z",
          "host" => "cadenza"
}

Feel free to experiment and test this by changing the append parameter:

bin/logstash -e 'input { stdin{ codec => example{ append => ", I am appending this! }} } output {stdout { codec => rubydebug }}'

Congratulations! You’ve built, deployed and successfully run a Logstash codec.

Logstash and all its plugins are licensed under Apache License, version 2 ("ALv2"). If you make your plugin publicly available via RubyGems.org, please make sure to have this line in your gemspec:

To begin, you’ll need an account on RubyGems.org

After creating an account, obtain an API key from RubyGems.org. By default, RubyGems uses the file ~/.gem/credentials to store your API key. These credentials will be used to publish the gem. Replace username and password with the credentials you created at RubyGems.org:

curl -u username:password https://rubygems.org/api/v1/api_key.yaml > ~/.gem/credentials
chmod 0600 ~/.gem/credentials

Before proceeding, make sure you have the right version in your gemspec file and commit your changes.

To publish version 0.1.0 of your new logstash gem:

bundle install
bundle exec rake vendor
bundle exec rspec
bundle exec rake publish_gem

That’s it! Your plugin is published! Logstash users can now install your plugin by running:

bin/logstash-plugin install logstash-codec-mypluginname

Some of the many benefits of having your plugin in the logstash-plugins repository are:

To begin migrating your plugin to logstash-plugins, simply create a new issue in the Logstash repository. When the acceptance guidelines are completed, we will facilitate the move to the logstash-plugins organization using the recommended github process.