Custom Azure Blob Storage Input

Collect log data from configured Azure Blob Storage Container with Elastic Agent.

Version
2.1.0 (View all)
Compatible Kibana version(s)
8.13.0 or higher
Supported Serverless project types

Security
Observability
Subscription level
Basic
Level of support
Elastic

Use the Azure Blob Storage Input to read content from files stored in containers that reside on your Azure Cloud. The input can be configured to work with and without polling, though currently, if polling is disabled it will only perform a one-time passthrough, list the file contents and end the process. Polling is generally recommended for most cases even though it can get expensive with dealing with a very large number of files.

To mitigate errors and ensure a stable processing environment, this input employs the following features :

  1. When processing Azure blob containers, if suddenly there is any outage, the process will be able to resume post the last file it processed for which it was successfully able to save the state.

  2. If any errors occur for certain files, they will be logged appropriately, but the rest of the files will continue to be processed normally.

  3. If any major error occurs that stops the main thread, the logs will be appropriately generated, describing said error.

[id="supported-types"] NOTE: Currently only JSON and NDJSON are supported blob/file formats. Blobs/files may also be gzip compressed. As for authentication types, we currently have support for shared access keys and connection strings.

Configuring The Input:

Assuming you have Azure Blob Containers already set up and account credentials available, please refer to the input documentation here for further details on specific parameters used by the integration.

ECS Field Mapping

This integration includes the ECS Dynamic Template, all fields that follow the ECS Schema will get assigned the correct index field mapping and do not need to be added manually.

Ingest Pipelines

Custom ingest pipelines may be added by adding the name to the pipeline configuration option, creating custom ingest pipelines can be done either through the API or the Ingest Node Pipeline UI.

Changelog

VersionDetailsKibana version(s)

2.1.0

Enhancement View pull request
ECS version updated to 8.11.0. Removed import_mappings. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.

8.13.0 or higher

2.0.0

Enhancement View pull request
Converted Azure Blob Storage to input package type.

8.12.0 or higher

1.1.0

Enhancement View pull request
Set sensitive values as secret.

8.12.0 or higher

1.0.1

Enhancement View pull request
Changed owners

8.11.0 or higher

1.0.0

Enhancement View pull request
Made integration GA. Refactored integration by adding support for new parameters and updated documentation.

8.11.0 or higher

0.9.0

Enhancement View pull request
ECS version updated to 8.11.0.

0.8.0

Enhancement View pull request
ECS version updated to 8.10.0.

0.7.0

Enhancement View pull request
The format_version in the package manifest changed from 2.11.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added 'owner.type: elastic' to package manifest.

0.6.0

Enhancement View pull request
Add tags.yml file so that integration's dashboards and saved searches are tagged with "Security Solution" and displayed in the Security Solution UI.

0.5.0

Enhancement View pull request
Update package to ECS 8.9.0.

0.4.0

Enhancement View pull request
Update package to ECS 8.8.0.

0.3.0

Enhancement View pull request
Update package-spec version to 2.7.0.

0.2.0

Enhancement View pull request
Update package to ECS 8.7.0.

0.1.0

Enhancement View pull request
Initial Release

On this page