Configuration
editConfiguration
editThis functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.
Upgrade the Logstash specification
editYou can upgrade the Logstash version or change settings by editing the YAML specification. ECK applies the changes by performing a rolling restart of Logstash Pods.
Logstash configuration
editDefine the Logstash configuration (the ECK equivalent to logstash.yml
) in the spec.config
section:
apiVersion: logstash.k8s.elastic.co/v1alpha1 kind: Logstash metadata: name: quickstart spec: version: 8.17.0 count: 1 elasticsearchRefs: - name: quickstart clusterName: qs config: pipeline.workers: 4 log.level: debug
Alternatively, you can provide the configuration through a Secret specified in the spec.configRef
section. The Secret must have an logstash.yml
entry with these settings:
apiVersion: logstash.k8s.elastic.co/v1alpha1 kind: Logstash metadata: name: quickstart spec: version: 8.17.0 count: 1 elasticsearchRefs: - name: quickstart clusterName: qs configRef: secretName: quickstart-config --- apiVersion: v1 kind: Secret metadata: name: quickstart-config stringData: logstash.yml: |- pipeline.workers: 4 log.level: debug
Configuring Logstash pipelines
editDefine Logstash pipelines in the spec.pipelines
section (the ECK equivalent to pipelines.yml
):
apiVersion: logstash.k8s.elastic.co/v1alpha1 kind: Logstash metadata: name: quickstart spec: version: 8.17.0 count: 1 elasticsearchRefs: - clusterName: qs name: quickstart pipelines: - pipeline.id: main config.string: | input { beats { port => 5044 } } output { elasticsearch { hosts => [ "${QS_ES_HOSTS}" ] user => "${QS_ES_USER}" password => "${QS_ES_PASSWORD}" ssl_certificate_authorities => "${QS_ES_SSL_CERTIFICATE_AUTHORITY}" } }
Alternatively, you can provide the pipeline configuration through a Secret specified in the spec.pipelinesRef
element. The Secret must have a logstash.yml
entry with this configuration:
apiVersion: logstash.k8s.elastic.co/v1alpha1 kind: Logstash metadata: name: quickstart spec: version: 8.17.0 count: 1 elasticsearchRefs: - clusterName: qs name: quickstart pipelinesRef: secretName: quickstart-pipeline --- apiVersion: v1 kind: Secret metadata: name: quickstart-pipeline stringData: pipelines.yml: |- - pipeline.id: main config.string: | input { beats { port => 5044 } } output { elasticsearch { hosts => [ "${QS_ES_HOSTS}" ] user => "${QS_ES_USER}" password => "${QS_ES_PASSWORD}" ssl_certificate_authorities => "${QS_ES_SSL_CERTIFICATE_AUTHORITY}" } }
Logstash on ECK supports all options present in pipelines.yml
, including settings to update the number of workers, and
the size of the batch that the pipeline will process. This also includes using path.config
to point to volumes
mounted on the Logstash container:
apiVersion: logstash.k8s.elastic.co/v1alpha1 kind: Logstash metadata: name: quickstart spec: version: 8.17.0 count: 1 elasticsearchRefs: - clusterName: qs name: quickstart pipelines: - pipeline.id: main config.string: | input { beats { port => 5044 } } output { elasticsearch { hosts => [ "${QS_ES_HOSTS}" ] user => "${QS_ES_USER}" password => "${QS_ES_PASSWORD}" ssl_certificate_authorities => "${QS_ES_SSL_CERTIFICATE_AUTHORITY}" } }
Logstash persistent queues (PQs) and dead letter queues (DLQs) are not currently managed by the Logstash operator, and using them will require you to create and manage your own Volumes and VolumeMounts
Defining data volumes for Logstash
edit[2.9.0] Added in 2.9.0.
Volume support for Logstash is a breaking change to earlier versions of ECK and requires you to recreate your Logstash resources.
Specifying the volume claim settings
editA PersistentVolume called logstash-data
is created by default.
It maps to /usr/share/logstash/data
for persistent storage, which is typically used for storage from plugins.
By default, the logstash-data
volume claim is a 1.5Gi
volume, using the standard StorageClass of your Kubernetes cluster.
You can override the default by adding a spec.volumeClaimTemplate
section named logstash-data
.
For production workloads, you should define your own volume claim template with the desired storage capacity and (optionally) the Kubernetes storage class to associate with the persistent volume. To override this volume claim for data
usages, the name of this volume claim must be logstash-data
.
This example updates the default data template to increase the storage to 2Gi
for the Logstash data folder:
apiVersion: logstash.k8s.elastic.co/v1alpha1 kind: Logstash metadata: name: logstash spec: # some configuration attributes omitted for brevity here volumeClaimTemplates: - metadata: name: logstash-data # Do not change this name unless you set up a volume mount for the data path. spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi
The default volume size will likely be insufficient for production workloads, especially when you are using:
- the persistent queue (PQ) feature
- dead letter queues (DLQ), or
- Logstash plugins that make heavy use of temporary storage.
Increase the storage capacity, or consider creating separate volumes for these use cases.
You can add separate storage by including an additional spec.volumeClaimTemplate
along with a corresponding spec.podTemplate.spec.containers.volumeMount
for each requested volume.
This example shows how to setup separate storage for a PQ:
apiVersion: logstash.k8s.elastic.co/v1alpha1 kind: Logstash metadata: name: logstash spec: # some configuration attributes omitted for brevity here volumeClaimTemplates: - metadata: name: pq spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi podTemplate: spec: containers: - name: logstash volumeMounts: - mountPath: /usr/share/logstash/pq name: pq readOnly: false config: log.level: info queue.type: persisted path.queue: /usr/share/logstash/pq
The |
|
Set the |
This example shows how to configure Logstash with a Dead Letter Queue setup on the main pipeline, and a separate pipeline to read items from the DLQ.
apiVersion: logstash.k8s.elastic.co/v1alpha1 kind: Logstash metadata: name: logstash spec: # some configuration attributes omitted for brevity here podTemplate: spec: containers: - name: logstash volumeMounts: - mountPath: /usr/share/logstash/dlq name: dlq readOnly: false volumeClaimTemplates: - metadata: name: dlq spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi pipelines: - pipeline.id: beats dead_letter_queue.enable: true path.dead_letter_queue: /usr/share/logstash/dlq config.string: | input { beats { port => 5044 } } output { elasticsearch { hosts => [ "${ECK_ES_HOSTS}" ] user => "${ECK_ES_USER}" password => "${ECK_ES_PASSWORD}" ssl_certificate_authorities => "${ECK_ES_SSL_CERTIFICATE_AUTHORITY}" } } - pipeline.id: dlq_read dead_letter_queue.enable: false config.string: | input { dead_letter_queue { path => "/usr/share/logstash/dlq" commit_offsets => true pipeline_id => "beats" clean_consumed => true } } filter { mutate { remove_field => "[geoip][location]" } } output { elasticsearch { hosts => [ "${ECK_ES_HOSTS}" ] user => "${ECK_ES_USER}" password => "${ECK_ES_PASSWORD}" ssl_certificate_authorities => "${ECK_ES_SSL_CERTIFICATE_AUTHORITY}" } }
The |
|
Set the |
Updating the volume claim settings
editIf the storage class allows volume expansion, you can increase the storage requests size in spec.volumeClaimTemplates
.
ECK updates the existing PersistentVolumeClaims accordingly, and recreates the StatefulSet automatically.
If the volume driver supports ExpandInUsePersistentVolumes
, the filesystem is resized online.
In this case, you do not need to restart the Logstash process or re-create the Pods.
If the volume driver does not support ExpandInUsePersistentVolumes
, you must manually delete Pods after the resize so that they can be recreated automatically with the expanded filesystem.
Any other changes in the volumeClaimTemplates—such as changing the storage class or decreasing the volume size—are not allowed. To make changes such as these, you must fully delete the Logstash resource, delete and recreate or resize the volume, and create a new Logstash resource.
Before you delete a persistent queue (PQ) volume, ensure that the queue is empty.
When using the PQ, we recommend setting queue.drain: true
on the Logstash Pods to ensure that the queue is drained when Pods are shutdown.
Note that you should also increase the terminationGracePeriodSeconds
to a large enough value to allow the queue to drain.
This example shows how to configure a Logstash resource to drain the queue and increase the termination grace period.
apiVersion: logstash.k8s.elastic.co/v1alpha1 kind: Logstash metadata: name: logstash spec: # some configuration attributes omitted for brevity here config: queue.drain: true podTemplate: spec: terminationGracePeriodSeconds: 604800
A Kubernetes known issue: Kubernetes may not honor terminationGracePeriodSeconds
settings greater than 600.
A queue of a terminated Pod may not be fully drained, even when queue.drain: true
is set and a high terminationGracePeriodSeconds
is configured.
In this technical preview, there is currently no way to drain a dead letter queue (DLQ) automatically before Logstash shuts down. To manually drain the queue, first stop sending data to it, by either disabling the DLQ feature, or disabling any pipelines that send to a DLQ. Then wait for events to stop flowing through any pipelines reading from the input.
EmptyDir
editIf you are not concerned about data loss, you can use an emptyDir
volume for Logstash data.
The use of emptyDir
in a production environment may cause permanent data loss.
Do not use with persistent queues (PQs), dead letter queues (DLQs), or with any plugin that requires persistent storage to keep track of state between restarts of Logstash.
Plugins that require persistent storage include any plugin that stores state locally.
These plugins typically have a configuration parameter that includes the name path
or directory
, not including paths to static content, such as certificates or keystores.
Examples include the sincedb_path
setting for the file
, dead_letter_queue
and s3
inputs, the last_run_metadata_path
for the JDBC
input, aggregate_maps_path
for the aggregate
filter, and temporary_directory
for the s3
output, used to aggregate content before uploading to s3.
spec: count: 5 podTemplate: spec: volumes: - name: logstash-data emptyDir: {}
Using Elasticsearch in Logstash pipelines
editThe spec.elasticsearchRefs
section provides a mechanism to help configure Logstash to establish a secured connection to one or more ECK managed Elasticsearch clusters. By default, each elasticsearchRef
will target all nodes in its referenced Elasticsearch cluster. If you want to direct traffic to specific nodes of your Elasticsearch cluster, refer to Traffic Splitting for more information and examples.
When you use elasticsearchRefs
in a Logstash pipeline, the Logstash operator creates the necessary resources from the associated Elasticsearch cluster, and provides environment variables to allow these resources to be accessed from the pipeline configuration.
Environment variables are replaced at runtime with the appropriate values.
The environment variables have a fixed naming convention:
-
NORMALIZED_CLUSTERNAME_ES_HOSTS
-
NORMALIZED_CLUSTERNAME_ES_USER
-
NORMALIZED_CLUSTERNAME_ES_PASSWORD
-
NORMALIZED_CLUSTERNAME_ES_SSL_CERTIFICATE_AUTHORITY
where NORMALIZED_CLUSTERNAME is the value taken from the clusterName
field of the elasticsearchRef
property, capitalized, and -
transformed to _
- eg, prod-es, would becomed PROD_ES.
The clusterName
value should be unique across all referenced Elasticsearches in the same Logstash spec.
The Logstash ECK operator creates a user called eck_logstash_user_role
when an elasticsearchRef
is specified. This user has the following permissions:
"cluster": ["monitor", "manage_ilm", "read_ilm", "manage_logstash_pipelines", "manage_index_templates", "cluster:admin/ingest/pipeline/get",], "indices": [ { "names": [ "logstash", "logstash-*", "ecs-logstash", "ecs-logstash-*", "logs-*", "metrics-*", "synthetics-*", "traces-*" ], "privileges": ["manage", "write", "create_index", "read", "view_index_metadata"] }
You can update user permissions to include more indices if the Elasticsearch plugin is expected to use indices other than the default. Check out Logstash configuration with a custom index sample configuration that creates a user that writes to a custom index.
This example demonstrates how to create a Logstash deployment that connects to different Elasticsearch instances, one of which is in a separate namespace:
apiVersion: logstash.k8s.elastic.co/v1alpha1 kind: Logstash metadata: name: quickstart spec: version: 8.17.0 count: 1 elasticsearchRefs: - clusterName: prod-es name: prod - clusterName: qa-es name: qa namespace: qa pipelines: - pipeline.id: main config.string: | input { beats { port => 5044 } } output { elasticsearch { hosts => [ "${PROD_ES_ES_HOSTS}" ] user => "${PROD_ES_ES_USER}" password => "${PROD_ES_ES_PASSWORD}" ssl_certificate_authorities => "${PROD_ES_ES_SSL_CERTIFICATE_AUTHORITY}" } elasticsearch { hosts => [ "${QA_ES_ES_HOSTS}" ] user => "${QA_ES_ES_USER}" password => "${QA_ES_ES_PASSWORD}" ssl_certificate_authorities => "${QA_ES_ES_SSL_CERTIFICATE_AUTHORITY}" } }
Define Elasticsearch references in the CRD. This will create the appropriate Secrets to store certificate details and the rest of the connection information, and create environment variables to allow them to be referred to in Logstash pipeline configurations. |
|
This refers to an Elasticsearch cluster residing in the same namespace as the Logstash instances. |
|
This refers to an Elasticsearch cluster residing in a different namespace to the Logstash instances. |
|
Elasticsearch output definitions - use the environment variables created by the Logstash operator when specifying an |
Connect to an external Elasticsearch cluster
editLogstash can connect to external Elasticsearch cluster that is not managed by ECK.
You can reference a Secret instead of an Elasticsearch cluster in the elasticsearchRefs
section through the secretName
attribute:
apiVersion: v1 kind: Secret metadata: name: external-es-ref stringData: url: https://abcd-42.xyz.elastic-cloud.com:443 username: logstash_user password: REDACTED ca.crt: REDACTED --- apiVersion: logstash.k8s.elastic.co/v1alpha1 kind: Logstash metadata: name: quickstart spec: version: 8.17.0 count: 1 elasticsearchRefs: - clusterName: prod-es secretName: external-es-ref monitoring: metrics: elasticsearchRefs: - secretName: external-es-ref logs: elasticsearchRefs: - secretName: external-es-ref
The URL to reach the Elasticsearch cluster. |
|
The username of the user to be authenticated to the Elasticsearch cluster. |
|
The password of the user to be authenticated to the Elasticsearch cluster. |
|
The CA certificate in PEM format to secure communication to the Elasticsearch cluster (optional). |
|
The |
Please always specify the port in URL when connecting to an external Elasticsearch Cluster.
Expose services
editBy default, the Logstash operator creates a headless Service for the metrics endpoint to enable metric collection by the Metricbeat sidecar for Stack Monitoring:
kubectl get service quickstart-ls-api
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE quickstart-ls-api ClusterIP None <none> 9600/TCP 48s
Additional services can be added in the spec.services
section of the resource:
services: - name: beats service: spec: ports: - port: 5044 name: "winlogbeat" protocol: TCP - port: 5045 name: "filebeat" protocol: TCP
Pod configuration
editYou can customize the Logstash Pod using a Pod template, defined in the spec.podTemplate
section of the configuration.
This example demonstrates how to create a Logstash deployment with increased heap size and resource limits.
apiVersion: logstash.k8s.elastic.co/v1alpha1 kind: Logstash metadata: name: logstash-sample spec: version: 8.17.0 count: 1 elasticsearchRefs: - name: "elasticsearch-sample" clusterName: "sample" podTemplate: spec: containers: - name: logtash env: - name: LS_JAVA_OPTS value: "-Xmx2g -Xms2g" resources: requests: memory: 1Gi cpu: 0.5 limits: memory: 4Gi cpu: 2
The name of the container in the Pod template must be logstash
.