Jms input plugin
editJms input plugin
edit- Plugin version: v3.1.2
- Released on: 2019-12-05
- Changelog
For other versions, see the Versioned plugin docs.
Getting Help
editFor questions about the plugin, open a topic in the Discuss forums. For bugs or feature requests, open an issue in Github. For the list of Elastic supported plugins, please consult the Elastic Support Matrix.
Description
editRead events from a Jms Broker. Supports both Jms Queues and Topics.
For more information about Jms, see https://javaee.github.io/tutorial/jms-concepts.html. For more information about the Ruby Gem used, see http://github.com/reidmorrison/jruby-jms.
JMS configurations can be done either entirely in the Logstash configuration file, or in a mixture of the Logstash configuration file, and a specified yaml file. Simple configurations that do not need to make calls to implementation specific methods on the connection factory can be specified entirely in the Logstash configuration, whereas more complex configurations, should also use the combination of yaml file and Logstash configuration.
Sample Configuration using Logstash Configuration Only
editConfigurations can be configured either entirely in Logstash configuration, or via a combination of Logstash configuration and yaml file, which can be useful for sharing similar configurations across multiple inputs and outputs. The JMS plugin can also be configured using JNDI if desired.
Logstash Confuguration for Non-JNDI Connection
editinput { jms { broker_url => 'failover:(tcp://host1:61616,tcp://host2:61616)?initialReconnectDelay=100' destination => 'myqueue' factory => 'org.apache.activemq.ActiveMQConnectionFactory' pub_sub => false use_jms_timestamp => false # JMS provider credentials if needed username => 'username' password => 'secret' # JMS provider keystore and truststore details keystore => '/Users/logstash-user/security/keystore.jks' keystore_password => 'another_secret' truststore => '/Users/logstash-user/security/truststore.jks' truststore_password => 'yet_another_secret' # Parts of the JMS message to be included include_header => false include_properties => false include_body => true # Message selector selector => "string_property = 'this' OR int_property < 3" # Connection factory specific settings factory_settings => { exclusive_consumer => true } # Jar Files to include require_jars => ['/usr/share/jms/activemq-all-5.15.9.jar'] } }
Url of the broker to connect to. Please consult your JMS provider documentation for the exact syntax to use here, including how to enable failover. |
|
Name of the topic or queue that the plugin will listen to events from. |
|
Full name (including package name) of Java connection factory used to create a connection with your JMS provider. |
|
Determines whether the event source is a queue or a topic, set to |
|
Determines whether the JMSTimestamp header is used to populate the |
|
Credentials to use when connecting to the JMS provider, if required. |
|
Keystore and Truststore to use when connecting to the JMS provider, if required. |
|
Parts of the JMS Message to include in the event - headers, properties and the message body can be included or excluded from the event. |
|
Message selector: Use this to filter messages to be processed. The whole selector query should be double-quoted, string property values should be single quoted, and numeric property vaues should not be quoted. See JMS provider documentation for exact syntax. |
|
Additional settings that are set directly on the ConnectionFactory object can be added here. |
|
List of jars required by the JMS provider. Paths should be the fully qualified location of all jar files required by the JMS provider. This list may also include dependent jars as well as specific jars from the JMS provider. |
Logstash Configuration for JNDI Connection
editinput { jms { # Logstash Configuration Settings. include_header => false include_properties => false include_body => true use_jms_timestamp => false destination => "myqueue" pub_sub => false # JNDI Settings jndi_name => /jms/cf/default jndi_context => { 'java.naming.factory.initial' => com.solacesystems.jndi.SolJNDIInitialContextFactory 'java.naming.security.principal' => solace-cloud-client@user 'java.naming.provider.url' => tcp://address.messaging.solace.cloud:20608 'java.naming.security.credentials' => verysecret } # Jar files to be imported require_jars=> ['/usr/share/jms/commons-lang-2.6.jar', '/usr/share/jms/sol-jms-10.5.0.jar', '/usr/share/jms/geronimo-jms_1.1_spec-1.1.1.jar', '/usr/share/jms/commons-lang-2.6.jar]' } }
Configuration settings. Note that there is no |
|
JNDI name for this connection. |
|
JNDI context settings hash. Contains details of how to connect to JNDI server. See your JMS provider documentation for implementation specific details. |
|
List of jars required by the JMS provider. Paths should be the fully qualified location of all jar files required by the JMS provider. This list may also include dependent jars as well as specific jars from the JMS provider. |
Sample Configuration using Logstash Configuration and Yaml File
editNon-JNDI Connection
editThis section contains sample configurations for connecting to a JMS provider that is not using JNDI using a combination of the Logstash configuration and the yaml file
Logstash Configuration for Non-JNDI Connection (for configs including yaml)
editinput { jms { # Logstash Configuration File Settings include_header => false include_properties => false include_body => true use_jms_timestamp => false destination => "myqueue" pub_sub => false # JMS Provider credentials username => xxxx password => xxxx # Location of yaml file, and which section to use for configuration yaml_file => "~/jms.yml" yaml_section => "mybroker" } }
Yaml File for Non-JNDI Connection
editmybroker: :broker_url: 'ssl://localhost:61617' :factory: org.apache.activemq.ActiveMQConnectionFactory :exclusive_consumer: true :require_jars: - /usr/share/jms/activemq-all-5.15.9.jar - /usr/share/jms/log4j-1.2.17.jar
Section name for this broker definition. This should be the value of |
|
Full url of the broker. See your JMS Provider documentation for details. |
|
Full name (including package name) of Java connection factory used to create a connection with your JMS provider. |
|
Implementation specific configuration parameters to be used with the connection factory specified.
in <3>. Each JMS Provider will have its own set of parameters that can be used here. These parameters are mapped to
|
|
List of jars required by the JMS provider. Paths should be the fully qualified location of all jar files required by the JMS provider. This list may also include dependent jars as well as specific jars from the JMS provider. |
JNDI Connection
editThis section contains sample configurations for connecting to a JMS provider that is using JNDI using a combination of the Logstash configuration and the yaml file
Logstash Configuration for JNDI Connection (for configs including yaml)
editinput { jms { # Logstash specific configuration settings include_header => false include_properties => false include_body => true use_jms_timestamp => false destination => "myqueue" pub_sub => false # Location of yaml file, and which section to use for configuration yaml_file => "~/jms.yml" yaml_section => "mybroker" } }
Yaml File for JNDI Connection
editsolace: :jndi_name: /jms/cf/default :jndi_context: java.naming.factory.initial: com.solacesystems.jndi.SolJNDIInitialContextFactory java.naming.security.principal: solace-cloud-client@user java.naming.provider.url: tcp://address.messaging.solace.cloud:20608 java.naming.security.credentials: verysecret :require_jars: - /usr/share/jms/commons-lang-2.6.jar - /usr/share/jms/sol-jms-10.5.0.jar - /usr/share/jms/geronimo-jms_1.1_spec-1.1.1.jar - /usr/share/jms/commons-lang-2.6.jar
Section name for this broker definition. This should be the value of |
|
Name of JNDI entry at which the Factory can be found |
|
JNDI context settings. Contains details of how to connect to JNDI server. See your JMS provider documentation for implementation specific details. |
|
List of jars required by the JMS provider. Paths should be the fully qualified location of all jar files required by the JMS provider. This list may also include dependent jars as well as specific jars from the JMS provider. |
Jar files
editIn order to communicate with a JMS broker, the plugin must load the jar files necessary for each client type.
This can be set in the yaml file, or in the main configuration if a yaml file is not necessary. The require_jars
setting should include the full path for each jar file required for the client. Eg
Logstash configuration
editinput { jms { : [snip] require_jars => ['/usr/share/jms/commons-lang-2.6.jar', '/usr/share/jms/sol-jms-10.5.0.jar', '/usr/share/jms/geronimo-jms_1.1_spec-1.1.1.jar', '/usr/share/jms/commons-lang-2.6.jar'] } }
Troubleshooting
editThis section includes some common issues that a user may have when integrating this plugin with their JMS provider.
Missing Jar files
editThe most common issue is missing jar files, which may be jar files provided by the JMS vendor, or jar files that the JMS vendor requires to run. This issue can manifest in different ways, depending on where the missing jar file is discovered.
Example log output:
Failed to load JMS Connection, likely because a JMS Provider is not on the Logstash classpath or correctly specified by the plugin's `require_jars` directive {:exception=>"cannot load Java class javax.jms.DeliveryMode"
JMS Consumer Died {:exception=>"Java::JavaxNaming::NoInitialContextException", :exception_message=>"Cannot instantiate class:"
warning: thread "[main]<jms" terminated with exception (report_on_exception is true): java.lang.NoClassDefFoundError: org/apache/commons/lang/exception/NestableException
JMS Consumer Died {:exception=>"Java::JavaxJms::JMSException", :exception_message=>"io/netty/channel/epoll/Epoll", :root_cause=>{:exception=>"Java::JavaLang::NoClassDefFoundError", :exception_message=>"io/netty/channel/epoll/Epoll"}
If any of these issues occur, check the list of require_jars
in either the Logstash configuration or yaml configuration
files.
Setting System Properties
editMany JMS providers allow or expect System properties to be set to configure certain properties when using JMS, for
example, the Apache qpid JMS client allows the connection factory lookup to be stored there, and the Solace JMS client
allows many properties, such as number of connection retries to be set as System properties. Any system properties that
are set should be set in the Logstash jvm.options
file.
Multiple JMS inputs/outputs in the same Logstash process
editThe use of multiple JMS consumers and producers in the same Logstash process is unsupported if:
- System properties need to be different for any of the consumers/producers
- Different keystores or truststores are required for any of the consumers/producers
Message Selectors unexpectedly filtering out all messages
editIncorrect message selector syntax can have two effects - either the syntax is incorrect and the selector parser from the JMS provider will throw an exception causing the plugin to fail OR the syntax will be accepted, but the messages will be silently dropped - this can happen with incorrect quoting of string properties in the selector definition. All selector definitions must be double quoted in the Logstash configuration file, and string property values must be single quoted, and numeric property values not quoted at all.
Failed to create Event with MissingConverterException
editMessages from certain JMS providers may contain headers or properties that Logstash cannot interpret, which can lead to error messages such as:
[2019-11-25T08:04:28,769][ERROR][logstash.inputs.jms ] Failed to create event {:message=>Java::ComSolacesystemsJmsMessage::SolTextMessage: ... Attributes: {:jms_correlation_id=>"xxxx", :jms_delivery_mode_sym=>:non_persistent, :jms_destination=>"destination", :jms_expiration=>0, :jms_message_id=>"xxxxxx", :jms_priority=>0, :jms_redelivered=>false, :jms_reply_to=>#<Java::ComSolacesystemsJmsImpl::SolTopicImpl:0xdeadbeef>, :jms_timestamp=>1574669008862, :jms_type=>nil} Properties: nil, :exception=>org.logstash.MissingConverterException: Missing Converter handling for full class name=com.solacesystems.jms.impl.SolTopicImpl, simple name=SolTopicImpl, :backtrace=>["org.logstash.Valuefier.fallbackConvert(Valuefier.java:98)..."]} :exception=>org.logstash.MissingConverterException: Missing Converter handling for full class name=com.solacesystems.jms.impl.SolTopicImpl
To get around this, use the skip_headers
or skip_properties
configuration setting to avoid attempting to process
the offending header or property in the message.
In the example shown above, this attribute is causing the
MissingConverterException
:
jms_reply_to=>#<Java::ComSolacesystemsJmsImpl::SolTopicImpl:0xdeadbeef>
To avoid this error, the configuration should include the following line:
skip_headers => ["jms_reply_to"]
Jms Input Configuration Options
editThis plugin supports the following configuration options plus the Common Options described later.
Setting | Input type | Required |
---|---|---|
No |
||
Yes |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
a valid filesystem path |
No |
|
No |
||
No |
||
No |
||
No |
||
No |
||
Deprecated |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
a valid filesystem path |
No |
|
No |
||
No |
||
No |
||
No |
||
No |
Also see Common Options for a list of options supported by all input plugins.
broker_url
edit- Value type is string
- There is no default value for this setting.
Url to use when connecting to the JMS provider. This is only relevant for non-JNDI configurations.
destination
edit- This is a required setting.
- Value type is string
- There is no default value for this setting.
Name of the destination queue or topic to use.
durable_subscriber
edit- Value type is boolean
-
This is
false
by default -
Requires
pub_sub
to be true
Setting this value to true
will make subscriptions to topics "durable", which allowing messages that arrived on the
specified topic while Logstash is not running to still be read. Without setting this value, any messages sent to a topic
while Logstash is not actively listening will be lost. A durable subscriber specifies a unique identity consisting of the
topic (destination
), the client id (durable_subscriber_client_id
) and subscriber name
(durable_subscriber_subscriber_name
). See your JMS Provider documentation for any further requirements/limitations
around these settings.
- Note that a durable subscription can only have one active subscriber at a time.
-
Note that this setting is only permitted when
pub_sub
is set to true, and will generate a configuration error otherwise
durable_subscriber_client_id
edit- Value type is string
-
If
durable_subscriber
is set, the default value for this setting is Logstash, otherwise this setting has no effect
This represents the value of the client ID for a durable subscribtion, and is only used if durable_subscriber
is set to true
.
durable_subscriber_name
edit- Value type is string
-
If
durable_subscriber
is set, the default value for this setting will be the same value as thedestination
setting, otherwise this setting has no effect.
This represents the value of the subscriber name for a durable subscribtion, and is only used if durable_subscriber
is set to true
. Please consult your JMS Provider documentation for constraints and requirements for this setting.
factory
edit- Value type is string
- There is no default value for this setting.
Full name (including package name) of Java connection factory used to create a connection with your JMS provider.
factory_settings
edit- Value type is hash
- There is no default value for this setting.
Hash of implementation specific configuration values to set on the connection factory of the JMS provider. Each JMS
Provider will have its own set of parameters that can be used here. These parameters are mapped to set
methods on
the provided connection factory, and can be supplied in either snake or camel case. For example, a hash including
exclusive_consumer => true
would call setExclusiveConsumer(true)
on the supplied connection factory.
See your JMS provider documentation for implementation specific details.
include_body
edit- Value type is boolean
-
Default value is
true
Include JMS Message Body in the event. Supports TextMessage, MapMessage and ByteMessage.
If the JMS Message is a TextMessage or ByteMessage, then the value will be in the "message" field of the event. If the JMS Message is a MapMessage, then all the key/value pairs will be added in the Hashmap of the event.
StreamMessage and ObjectMessage are not supported.
include_header
edit- Value type is boolean
-
Default value is
true
A JMS message has three parts:
- Message Headers (required)
- Message Properties (optional)
- Message Bodies (optional)
You can tell the input plugin which parts should be included in the event produced by Logstash.
Include JMS Message Header Field values in the event.
include_properties
edit- Value type is boolean
-
Default value is
true
Include JMS Message Properties Field values in the event.
interval
edit- Value type is number
-
Default value is
10
Polling interval in seconds. This is the time sleeping between asks to a consumed Queue. This parameter has non influence in the case of a subcribed Topic.
jndi_context
edit- Value type is hash
- There is no default value for this setting.
Only used if using JNDI lookup. Key value pairs to determine how to connect the JMS message brokers if JNDI is being used. Consult your JMS provider documentation for the correct values to use here.
jndi_name
edit- Value type is string
- There is no default value for this setting.
Only used if using JNDI lookup. Name of JNDI entry at which the Factory can be found.
keystore
edit- Value type is path
- There is no default value for this setting.
If you need to use a custom keystore (.jks
) specify it here. This does not work with .pem keys
keystore_password
edit- Value type is password
- There is no default value for this setting.
Specify the keystore password here. Note, most .jks files created with keytool require a password
oracle_aq_buffered_messages
edit- Value type is boolean
-
Default value is
false
Receive Oracle AQ buffered messages. In this mode persistent Oracle AQ JMS messages will not be received. Only for use with Oracle AQ
password
edit- Value type is string
- There is no default value for this setting.
Password to use when connecting to the JMS provider.
pub_sub
edit- Value type is boolean
-
Default value is
false
If pub-sub (topic) style should be used. Note that if pub_sub
is set to true, threads
must be set to 1.
require_jars
edit- Value type is array
- There is no default value for this setting.
If you do not use an yaml configuration use either the factory or jndi_name. An optional array of Jar file names to load for the specified JMS provider. By using this option it is not necessary to put all the JMS Provider specific jar files into the java CLASSPATH prior to starting Logstash.
runner
edit- DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
selector
edit- Value type is string
- There is no default value for this setting.
JMS message selector. Use in conjunctions with message headers or properties to filter messages to be processed. Only messages that match the query specified here will be processed. Check with your JMS provider for the correct JMS message selector syntax.
skip_headers
edit- Value type is array
- There is no default value for this setting.
If include_headers
is set, a list of headers to skip processing on can be specified here.
skip_properties
edit- Value type is array
- There is no default value for this setting.
If include_properties
is set, a list of properties to skip processing on can be specified here.
system_properties
edit- Value type is hash
- There is no default value for this setting.
Any System properties that the JMS provider requires can be set either in a Hash here, or in jvm.options
threads
edit- Value type is number
-
Default value is
1
- Note that if pub_sub is set to true, this value must be 1. A configuration error will be thrown otherwise
truststore
edit- Value type is path
- There is no default value for this setting.
If you need to use a custom truststore (.jks
) specify it here. This does not work with .pem certs.
truststore_password
edit- Value type is password
- There is no default value for this setting.
Specify the truststore password here.
- Note, most .jks files created with keytool require a password.
use_jms_timestamp
edit- Value type is boolean
-
Default value is
false
Convert the JMSTimestamp header field to the @timestamp value of the event
username
edit- Value type is string
- There is no default value for this setting.
Username to use for connecting to JMS provider.
yaml_section
edit- Value type is string
- There is no default value for this setting.
Yaml config file section name For some known examples, see jms.yml examples.
Common Options
editThe following configuration options are supported by all input plugins:
Details
edit
codec
edit- Value type is codec
-
Default value is
"plain"
The codec used for input data. Input codecs are a convenient method for decoding your data before it enters the input, without needing a separate filter in your Logstash pipeline.
enable_metric
edit- Value type is boolean
-
Default value is
true
Disable or enable metric logging for this specific plugin instance by default we record all the metrics we can, but you can disable metrics collection for a specific plugin.
id
edit- Value type is string
- There is no default value for this setting.
Add a unique ID
to the plugin configuration. If no ID is specified, Logstash will generate one.
It is strongly recommended to set this ID in your configuration. This is particularly useful
when you have two or more plugins of the same type, for example, if you have 2 jms inputs.
Adding a named ID in this case will help in monitoring Logstash when using the monitoring APIs.
input { jms { id => "my_plugin_id" } }
Variable substitution in the id
field only supports environment variables
and does not support the use of values from the secret store.
tags
edit- Value type is array
- There is no default value for this setting.
Add any number of arbitrary tags to your event.
This can help with processing later.
type
edit- Value type is string
- There is no default value for this setting.
Add a type
field to all events handled by this input.
Types are used mainly for filter activation.
The type is stored as part of the event itself, so you can also use the type to search for it in Kibana.
If you try to set a type on an event that already has one (for example when you send an event from a shipper to an indexer) then a new input will not override the existing type. A type set at the shipper stays with that event for its life even when sent to another Logstash server.