Google_pubsub input plugin

For plugins not bundled by default, it is easy to install by running bin/logstash-plugin install logstash-input-google_pubsub. See Working with plugins for more details.

For questions about the plugin, open a topic in the Discuss forums. For bugs or feature requests, open an issue in Github. For the list of Elastic supported plugins, please consult the Elastic Support Matrix.

Author: Eric Johnson <erjohnso@google.com> Date: 2016-06-01

Copyright 2016 Google Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Google deps This is a Logstash input plugin for Google Pub/Sub. The plugin can subscribe to a topic and ingest messages.

The main motivation behind the development of this plugin was to ingest Stackdriver Logging messages via the Exported Logs feature of Stackdriver Logging.

You must first create a Google Cloud Platform project and enable the Google Pub/Sub API. If you intend to use the plugin ingest Stackdriver Logging messages, you must also enable the Stackdriver Logging API and configure log exporting to Pub/Sub. There is plentiful information on https://cloud.google.com/ to get started:

Currently, this module requires you to create a topic manually and specify it in the logstash config file. You must also specify a subscription, but the plugin will attempt to create the pull-based subscription on its own.

All messages received from Pub/Sub will be converted to a logstash event and added to the processing pipeline queue. All Pub/Sub messages will be acknowledged and removed from the Pub/Sub topic (please see more about Pub/Sub concepts.

It is generally assumed that incoming messages will be in JSON and added to the logstash event as-is. However, if a plain text message is received, the plugin will return the raw text in as raw_message in the logstash event.

You have two options for authentication depending on where you run Logstash.

If you intend to use the logstash plugin for Stackdriver Logging message ingestion, you must first manually set up the Export option to Cloud Pub/Sub and the manually create the topic. Please see the more detailed instructions at, https://cloud.google.com/logging/docs/export/using_exported_logs [Exported Logs] and ensure that the necessary permissions have also been manually configured.

Logging messages from Stackdriver Logging exported to Pub/Sub are received as JSON and converted to a logstash event as-is in this format.

Below is a copy of the included example.conf-tmpl file that shows a basic configuration for this plugin.

input {
    google_pubsub {
        # Your GCP project id (name)
        project_id => "my-project-1234"

        # The topic name below is currently hard-coded in the plugin. You
        # must first create this topic by hand and ensure you are exporting
        # logging to this pubsub topic.
        topic => "logstash-input-dev"

        # The subscription name is customizeable. The plugin will attempt to
        # create the subscription (but use the hard-coded topic name above).
        subscription => "logstash-sub"

        # If you are running logstash within GCE, it will use
        # Application Default Credentials and use GCE's metadata
        # service to fetch tokens.  However, if you are running logstash
        # outside of GCE, you will need to specify the service account's
        # JSON key file below.
        #json_key_file => "/home/erjohnso/pkey.json"

        # Should the plugin attempt to create the subscription on startup?
        # This is not recommended for security reasons but may be useful in
        # some cases.
        #create_subscription => false
    }
}
output { stdout { codec => rubydebug } }

The original Pub/Sub message is preserved in the special Logstash [@metadata][pubsub_message] field so you can fetch:

You MUST extract any fields you want in a filter prior to the data being sent to an output because Logstash deletes @metadata fields otherwise.

See the PubsubMessage documentation for a full description of the fields.

Example to get the message ID:

input {google_pubsub {...}}

filter {
  mutate {
    add_field => { "messageId" => "%{[@metadata][pubsub_message][messageId]}" }
  }
}

output {...}

This plugin supports the following configuration options plus the Common options described later.

Also see Common options for a list of options supported by all input plugins.

These configuration options are supported by all input plugins:

input {
  google_pubsub {
    id => "my_plugin_id"
  }
}