Verifying Logstash Functionality Through Testing

By
Pere Urbon-Bayes

At Logstash, testing is fundamental to our engineering process. In this blog, we want to share this process with our amazing community. We’ll share how we do testing, including process and tools we use, and for Logstash plugin developers out there, we'll briefly describe how to write tests.

How do we test?

As you might know, Logstash is written in Ruby, actually in JRuby, the Ruby on top of the JVM. For this reason, all the tools used here are going to be Ruby and Java specific. In any software project there are multiple test approaches, one for each phase, lets take for example, the V model. Let's introduce this concept before moving forward.


vmodel.png

Like most software projects, we use unit testing to verify individual units of code and integration testing to check individual components working together. But there are other test phases like acceptance, functional, regression, all working together to strengthen the overall quality.


Inside Logstash we use rspec as a test runner, this means tests should use the rspec syntax. We use Jenkins as our Continuous Integration (CI) platform, to host our different test phases, like our regression tests,  integration tests using  docker and kitchen-ci. In addition to these, we are iteratively  introducing  advanced  testing procedures like randomize testing through the ruby-flores library.

Testing with RSpec

After an small introduction to our tests, it is time to go more in details. In this section we’ll review how to write and run tests for both plugins and the Logstash core.

How to run tests

Feedback is an important part of writing tests, so let’s start showing how to run them in the Logstash core and the plugins. Please keep in mind that tests are not packaged (as it used to be in previous versions) with the different release artifacts, so we assume you are in development mode.

The Logstash core

In the core codebase, we have a set of rake tasks available to run the tests, which is described in the table below.

Task command

Description

rake bootstrap

Bootstrap a Logstash with the minimum dependencies to run in development.

rake test:install-core

Install all dependencies necessary to run the core tests.

rake test:install-default

Install all dependencies to run the Integration test with all bundled plugins.

rake test:install-all

Install all available plugins, and dependencies, from logstash-plugins organization in github.

rake test:core

Run the core tests, this is a Unit test.

rake test:plugins

Run the plugins test, this is an Integrations test (defaults, all, subset)

In the plugins land

If you’re developing a plugin, running tests is simpler, you usually just need to run the bundle exec rspec command in the plugin's repository. This will run all files under the spec directory that have tests.


There are different options you can use to complement this command, the most common are:

-I

Add a directory to the $LOAD_PATH

ref

--tag

Used to enable test under a given tag

ref

--format

Select a format for the result output

ref

--pattern

Run only the files that match the pattern

ref

More info can be found in the rspec documentation.

How to write a test

A detailed guide to writing rspec tests is out of scope for this blog, however we’ll try to explain few basics with help of a sample test. Let’s take a look at this example. 


describe "foo" do
  subject   { described_class.new(options) }
  let(:var) { "zzzz"}
  before(:each) do
    # do something
  end
  context "when bar" do
    it "is sleeping" do
      expect(var).to eq("zzzz")
    end
  end
end


describe/context sections

Wrap up a test section, or a subsection (context).

ref

subject

Describe the subject of this test suite.

ref

let

Defines memoized variables, cached across different test.

ref

before/after

This are hooks that run around test and suites.

ref

it

Build a concrete test example.

ref

expect

Matchers used to verify conditions in test, more examples in the documentation.

ref

Test directory structure

In Logstash you will find all our tests inside the spec folder, which is standard convention for rspec. Inside this folder you will find a set of subfolders that provide namespacing for different test modules.


Nowadays, in Logstash core, we’ve this directories for tests:

  • core: This tests are validating core pipeline features.
  • { inputs / filters / outputs }: Tests for the basic plugins functionalities.
  • logstash: Tests for Logstash integrations and patches.
  • pluginmanager: Test for the plugin manager, the tool that lets you install plugins.
  • util: Inside LogStash we need some utilities, for example for accessors, the charsets, etc. Here life the test for them.

There are also other tests inside the main directory which does not fit on any “category”, in the main spec folder there are also the helpers, small modules that provide utilities for the rest of the tests.


But this was for Logstash core, plugins have a more simple directory structure. Inside plugins you will encounter:

  • { inputs / codecs / filters / outputs }: Unit test for the plugin.
  • integrations: Testing the integration with third party components.
  • { support / helpers }: Set of helpers and support classes need to run the test.

As a plugin developer, ff you wonder where to place your new tests, the last lines should help you.

Conclusion

In this post we briefly introduced the way we do testing in Logstash, described our tests infrastructure and organization, and provided a detailed description on how to run tests, write tests, store tests, etc. Automated testing is a very important aspect to all our Elastic products, so please stay tuned for more blogs in this area!


If you are interested in  learning more about testing, we recommend you check these resources: