How to write a Java output plugin
editHow to write a Java output plugin
editThis functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.
To develop a new Java output for Logstash, you write a new Java class that conforms to the Logstash Java Outputs API, package it, and install it with the logstash-plugin utility. We’ll go through each of those steps.
Set up your environment
editCopy the example repo
editStart by copying the example output plugin. The plugin API is currently part of the
Logstash codebase so you must have a local copy of that available. You can
obtain a copy of the Logstash codebase with the following git
command:
git clone --branch <branch_name> --single-branch https://github.com/elastic/logstash.git <target_folder>
The branch_name
should correspond to the version of Logstash containing the
preferred revision of the Java plugin API.
The beta version of the Java plugin API is available in the 6.7
branch of the Logstash codebase.
Specify the target_folder
for your local copy of the Logstash codebase. If you
do not specify target_folder
, it defaults to a new folder called logstash
under your current folder.
Generate the .jar file
editAfter you have obtained a copy of the appropriate revision of the Logstash
codebase, you need to compile it to generate the .jar file containing the Java
plugin API. From the root directory of your Logstash codebase ($LS_HOME), you
can compile it with ./gradlew assemble
(or gradlew.bat assemble
if you’re
running on Windows). This should produce the
$LS_HOME/logstash-core/build/libs/logstash-core-x.y.z.jar
where x
, y
, and
z
refer to the version of Logstash.
After you have successfully compiled Logstash, you need to tell your Java plugin
where to find the logstash-core-x.y.z.jar
file. Create a new file named
gradle.properties
in the root folder of your plugin project. That file should
have a single line:
LOGSTASH_CORE_PATH=<target_folder>/logstash-core
where target_folder
is the root folder of your local copy of the Logstash codebase.
Code the plugin
editThe example output plugin prints events to the console using the event’s
toString
method. Let’s look at the main class in the example output:
@LogstashPlugin(name = "java_output_example") public class JavaOutputExample implements Output { public static final PluginConfigSpec<String> PREFIX_CONFIG = PluginConfigSpec.stringSetting("prefix", ""); private final String id; private String prefix; private PrintStream printer; private final CountDownLatch done = new CountDownLatch(1); private volatile boolean stopped = false; public JavaOutputExample(final String id, final Configuration configuration, final Context context) { this(id, configuration, context, System.out); } JavaOutputExample(final String id, final Configuration config, final Context context, OutputStream targetStream) { this.id = id; prefix = config.get(PREFIX_CONFIG); printer = new PrintStream(targetStream); } @Override public void output(final Collection<Event> events) { Iterator<Event> z = events.iterator(); while (z.hasNext() && !stopped) { String s = prefix + z.next(); printer.println(s); } } @Override public void stop() { stopped = true; done.countDown(); } @Override public void awaitStop() throws InterruptedException { done.await(); } @Override public Collection<PluginConfigSpec<?>> configSchema() { return Collections.singletonList(PREFIX_CONFIG); } @Override public String getId() { return id; } }
Let’s step through and examine each part of that class.
Class declaration
edit@LogstashPlugin(name="java_output_example") public class JavaOutputExample implements Output {
Notes about the class declaration:
-
All Java plugins must be annotated with the
@LogstashPlugin
annotation. Additionally:-
The
name
property of the annotation must be supplied and defines the name of the plugin as it will be used in the Logstash pipeline definition. For example, this output would be referenced in the output section of the Logstash pipeline definition asoutput { java_output_example => { .... } }
-
The value of the
name
property must match the name of the class excluding casing and underscores.
-
The
-
The class must implement the
co.elastic.logstash.api.Output
interface.
Plugin settings
editThe snippet below contains both the setting definition and the method referencing it:
public static final PluginConfigSpec<String> PREFIX_CONFIG = PluginConfigSpec.stringSetting("prefix", ""); @Override public Collection<PluginConfigSpec<?>> configSchema() { return Collections.singletonList(PREFIX_CONFIG); }
The PluginConfigSpec
class allows developers to specify the settings that a
plugin supports complete with setting name, data type, deprecation status,
required status, and default value. In this example, the prefix
setting
defines an optional prefix to include in the output of the event. The setting is
not required and if it is not explicitly set, it defaults to the empty string.
The configSchema
method must return a list of all settings that the plugin
supports. In a future phase of the Java plugin project, the Logstash execution
engine will validate that all required settings are present and that no
unsupported settings are present.
Constructor and initialization
editprivate final String id; private String prefix; private PrintStream printer; public JavaOutputExample(final String id, final Configuration configuration, final Context context) { this(configuration, context, System.out); } JavaOutputExample(final String id, final Configuration config, final Context context, OutputStream targetStream) { this.id = id; prefix = config.get(PREFIX_CONFIG); printer = new PrintStream(targetStream); }
All Java output plugins must have a constructor taking a String
id and a
Configuration
and Context
argument. This is the constructor that will be
used to instantiate them at runtime. The retrieval and validation of all plugin
settings should occur in this constructor. In this example, the values of the
prefix
setting is retrieved and stored in a local variable for later use in
the output
method. In this example, a second, pacakge private constructor is
defined that is useful for unit testing with a Stream
other than System.out
.
Any additional initialization may occur in the constructor as well. If there are any unrecoverable errors encountered in the configuration or initialization of the output plugin, a descriptive exception should be thrown. The exception will be logged and will prevent Logstash from starting.
Output method
edit@Override public void output(final Collection<Event> events) { Iterator<Event> z = events.iterator(); while (z.hasNext() && !stopped) { String s = prefix + z.next(); printer.println(s); } }
Outputs may send events to local sinks such as the console or a file or to remote systems such as Elasticsearch or other external systems. In this example, the events are printed to the local console.
Stop and awaitStop methods
editprivate final CountDownLatch done = new CountDownLatch(1); private volatile boolean stopped; @Override public void stop() { stopped = true; done.countDown(); } @Override public void awaitStop() throws InterruptedException { done.await(); }
The stop
method notifies the output to stop sending events. The stop mechanism
may be implemented in any way that honors the API contract though a volatile
boolean
flag works well for many use cases. Because this output example is so
simple, its output
method does not check for the stop flag.
Outputs stop both asynchronously and cooperatively. Use the awaitStop
method
to block until the output has completed the stop process. Note that this method
should not signal the output to stop as the stop
method does. The
awaitStop mechanism may be implemented in any way that honors the API contract
though a CountDownLatch
works well for many use cases.
getId method
edit@Override public String getId() { return id; }
For output plugins, the getId
method should always return the id that was provided to the plugin through its
constructor at instantiation time.
Unit tests
editLastly, but certainly not least importantly, unit tests are strongly encouraged. The example output plugin includes an example unit test that you can use as a template for your own.
Package and deploy
editJava plugins are packaged as Ruby gems for dependency management and interoperability with Ruby plugins.
One of the goals for Java plugin support is to eliminate the need for any
knowledge of Ruby or its toolchain for Java plugin development. Future phases of
the Java plugin project will automate the packaging of Java plugins as Ruby gems
so no direct knowledge of or interaction with Ruby will be required. In the
current phase, Java plugins must still be manually packaged as Ruby gems
and installed with the logstash-plugin
utility.
Compile to JAR file
editThe Java plugin should be compiled and assembled into a fat jar with the
vendor
task in the Gradle build file. This will package all Java dependencies
into a single jar and write it to the correct folder for later packaging into a
Ruby gem.
To build the jar file, run the following command from the plugin directory:
./gradlew vendor
Manually package as Ruby gem
editSeveral Ruby source files are required to package the jar file as a Ruby gem. These Ruby files are used only at Logstash startup time to identify the Java plugin and are not used during runtime event processing.
These Ruby source files will be automatically generated in a future release.
logstash-output-<output-name>.gemspec
Gem::Specification.new do |s| s.name = 'logstash-output-java_output_example' s.version = PLUGIN_VERSION s.licenses = ['Apache-2.0'] s.summary = "Example output using Java plugin API" s.description = "" s.authors = ['Elasticsearch'] s.email = 'info@elastic.co' s.homepage = "http://www.elastic.co/guide/en/logstash/current/index.html" s.require_paths = ['lib', 'vendor/jar-dependencies'] # Files s.files = Dir["lib/**/*","spec/**/*","*.gemspec","*.md","CONTRIBUTORS","Gemfile","LICENSE","NOTICE.TXT", "vendor/jar-dependencies/**/*.jar", "vendor/jar-dependencies/**/*.rb", "VERSION", "docs/**/*"] # Special flag to let us know this is actually a logstash plugin s.metadata = { 'logstash_plugin' => 'true', 'logstash_group' => 'output'} # Gem dependencies s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99" s.add_runtime_dependency 'jar-dependencies' s.add_development_dependency 'logstash-devutils' end
You can use this file with the following modifications:
-
s.name
must follow thelogstash-output-<output-name>
pattern -
s.version
must match theproject.version
specified in thebuild.gradle
file. Both versions should be set to be read from theVERSION
file in this example.
lib/logstash/outputs/<output-name>.rb
# encoding: utf-8 require "logstash/outputs/base" require "logstash/namespace" require "logstash-output-java_output_example_jars" require "java" class LogStash::Outputs::JavaOutputExample < LogStash::Outputs::Base config_name "java_output_example" def self.javaClass() org.logstash.javaapi.JavaOutputExample.java_class; end end
Modify these items in the file above:
- Change the name to correspond with the output name.
-
Change
require "logstash-output-java_output_example_jars"
to reference the appropriate "jars" file as described below. -
Change
class LogStash::Outputs::JavaOutputExample < LogStash::Outputs::Base
to provide a unique and descriptive Ruby class name. -
Change
config_name "java_output_example"
to match the name of the plugin as specified in thename
property of the@LogstashPlugin
annotation. -
Change
def self.javaClass() org.logstash.javaapi.JavaOutputExample.java_class; end
to return the class of the Java output.
lib/logstash-output-<output-name>_jars.rb
require 'jar_dependencies' require_jar('org.logstash.javaapi', 'logstash-output-java_output_example', '0.0.1')
In the file above:
- Rename the file to correspond to the output name.
-
Change the
require_jar
directive to correspond to thegroup
specified in the Gradle build file, the name of the output JAR file, and the version as specified in both the gemspec and Gradle build file.
After you have created the previous files and the plugin JAR file, build the gem using the following command:
gem build logstash-output-<output-name>.gemspec
Installing the Java plugin in Logstash
editAfter you have packaged your Java plugin as a Ruby gem, you can install it in Logstash with this command:
bin/logstash-plugin install --no-verify --local /path/to/javaPlugin.gem
For Windows platforms: Substitute backslashes for forward slashes as appropriate in the command.
Running Logstash with the Java output plugin
editThe following is a minimal Logstash configuration that can be used to test that the Java output plugin is correctly installed and functioning.
input { generator { message => "Hello world!" count => 1 } } output { java_output_example {} }
Copy the above Logstash configuration to a file such as java_output.conf
.
Logstash should then be started with:
bin/logstash --java-execution -f /path/to/java_output.conf
The --java-execution
flag to enable the Java execution engine is
required as Java plugins are not supported in the Ruby execution engine.
The expected Logstash output (excluding initialization) with the configuration above is:
{"@timestamp":"yyyy-MM-ddTHH:mm:ss.SSSZ","message":"Hello world!","@version":"1","host":"<yourHostname>","sequence":0}
Feedback
editIf you have any feedback on Java plugin support in Logstash, please comment on our main Github issue or post in the Logstash forum.