Custom GCS (Google Cloud Storage) Input

Use the Google Cloud Storage input to read content from files stored in buckets that reside on your Google Cloud. The input can be configured to work with and without polling, though currently, if polling is disabled it will only perform a one-time passthrough, list the file contents and end the process. Polling is generally recommended for most cases even though it can get expensive with dealing with a very large number of files.

To mitigate errors and ensure a stable processing environment, this input employs the following features :

This is a one-time download JSON key file that you get after adding a key to a GCP service account. If you are just starting out creating your GCS bucket, do the following:

A sample JSON Credentials file looks as follows:

{
  "type": "dummy_service_account",
  "project_id": "dummy-project",
  "private_key_id": "dummy-private-key-id",
  "private_key": "-----BEGIN PRIVATE KEY-----\nDummyPrivateKey\n-----END PRIVATE KEY-----\n",
  "client_email": "dummy-service-account@example.com",
  "client_id": "12345678901234567890",
  "auth_uri": "https://dummy-auth-uri.com",
  "token_uri": "https://dummy-token-uri.com",
  "auth_provider_x509_cert_url": "https://dummy-auth-provider-cert-url.com",
  "client_x509_cert_url": "https://dummy-client-cert-url.com",
  "universe_domain": "dummy-universe-domain.com"
}

NOTE:

Assuming you have GCS buckets already set up and the service account key available, please refer to the input documentation here for further details on specific parameters used by the integration.

This integration includes the ECS Dynamic Template, all fields that follow the ECS Schema will get assigned the correct index field mapping and do not need to be added manually.

Custom ingest pipelines may be added by adding the name to the pipeline configuration option, creating custom ingest pipelines can be done either through the API or the Ingest Node Pipeline UI (use the global search field to search for "ingest pipelines").