This documentation contains work-in-progress information for future Elastic Stack and Cloud releases. Use the version selector to view supported release docs. It also contains some Elastic Cloud serverless information. Check out our serverless docs for more details.
« Storage recommendations Virtual memory »
Elastic Docs ›Elastic Cloud on Kubernetes [master] ›Orchestrating Elastic Stack applications ›Run Elasticsearch on ECK

Transport settings

edit

Transport settings

edit

The transport module in Elasticsearch is used for internal communication between nodes within the cluster as well as communication between remote clusters. Check the Elasticsearch documentation for details. For customization options of the HTTP layer, check Services and TLS certificates.

Customize the Transport Service

edit

In the spec.transport.service. section, you can change the Kubernetes service used to expose the Elasticsearch transport module:

spec:
  transport:
    service:
      metadata:
        labels:
          my-custom: label
      spec:
        type: LoadBalancer

Check the Kubernetes Publishing Services (ServiceTypes) that are currently available.

When you change the clusterIP setting of the service, ECK deletes and re-creates the service, as clusterIP is an immutable field. This will cause a short network disruption, but in most cases it should not affect existing connections as the transport module uses long-lived TCP connections.

Configure a custom Certificate Authority

edit

Elasticsearch uses X.509 certificates to establish encrypted and authenticated connections across nodes in the cluster. By default, ECK creates a self-signed CA certificate to issue a certificate for each node in the cluster.

You can use a Kubernetes secret to provide your own CA instead of the self-signed certificate that ECK will then use to create node certificates for transport connections. The CA certificate must be stored in the secret under ca.crt and the private key must be stored under ca.key.

You need to reference the name of a secret that contains the TLS private key and the CA certificate, in the spec.transport.tls.certificate section, as shown in this example:

spec:
  transport:
    tls:
      certificate:
        secretName: custom-ca

Customize the node transport certificates

edit

The operator generates a self-signed TLS certificates for each node in the cluster. You can add extra IP addresses or DNS names to the generated certificates as follows:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 8.15.3
  transport:
    tls:
      subjectAltNames:
      - ip: 1.2.3.4
      - dns: hulk.example.com
  nodeSets:
  - name: default
    count: 3

Issue node transport certificates with third-party tools

edit

When following the instructions in Configure a custom Certificate Authority the issuance of certificates is orchestrated by the ECK operator and the operator needs access to the CAs private key. If this is undesirable it is also possible to configure node transport certificates without involving the ECK operator. The following two pre-requisites apply:

  1. The tooling used must be able to issue individual certificates for each Elasticsearch node and dynamically add or remove certificates as the cluster scales up and down.
  2. The ECK operator must be configured to be aware of the CA in use for the remote cluster support to work.

The following example configuration using cert-manager csi-driver and trust-manager meets these two requirements:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: es
spec:
  version: 8.15.3
  transport:
    tls:
      certificateAuthorities:
        configMapName: trust 
      selfSignedCertificates:
        disabled: true 
  nodeSets:
  - name: default
    count: 3
    config:
      xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/cert-manager-certs/tls.key
      xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/cert-manager-certs/tls.crt
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          volumeMounts:
          - name: transport-certs
            mountPath: /usr/share/elasticsearch/config/cert-manager-certs
        volumes:
        - name: transport-certs
          csi:
            driver: csi.cert-manager.io
            readOnly: true
            volumeAttributes:
              csi.cert-manager.io/issuer-name: ca-cluster-issuer 
              csi.cert-manager.io/issuer-kind: ClusterIssuer
              csi.cert-manager.io/dns-names: "${POD_NAME}.${POD_NAMESPACE}.svc.cluster.local"

Disables the self-signed certificates generated by ECK for the transport layer.

The example assumes that a ClusterIssuer by the name of ca-cluster-issuer exists and a PEM encoded version of the CA certificate is available in a ConfigMap (in the example named trust). The CA certificate must be in a file called ca.crt inside the ConfigMap in the same namespace as the Elasticsearch resource.

If the remote cluster server is enabled, then the DNS names must also include both:

  • The DNS name for the related Kubernetes Service: <cluster-name>-es-remote-cluster.${POD_NAMESPACE}.svc
  • The Pod DNS name: ${POD_NAME}.<cluster-name>-es-<nodeset-name>.${POD_NAMESPACE}.svc

The following manifest is only provided to illustrate how these certificates can be configured in principle, using the trust-manager Bundle resource and cert-manager provisioned certificates:

apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: trust
spec:
  sources:
  - secret:
      name: "root-ca-secret"
      key: "tls.crt"
  target:
    configMap:
      key: "ca.crt"
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: selfsigned-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: selfsigned-ca
  secretName: root-ca-secret
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-issuer
    kind: ClusterIssuer
    group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {} 
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ca-cluster-issuer
spec:
  ca:
    secretName: root-ca-secret
...

This example uses a self-signed issuer for the root CA and a second issuer for the Elasticsearch cluster transport certificates as the cert-manager CSI driver does not support self-signed CAs.

When transitioning from a configuration that uses externally provisioned certificates back to ECK-managed self-signed transport certificates it is important to ensure that the externally provisioned CA remains configured as a trusted CA through the .spec.transport.tls.certificateAuthorities attribute until all nodes in the cluster have been updated to use the ECK-managed certificates. When transitioning from ECK-managed certificates to externally provisioned ones, ECK ensures automatically that the ECK CA remains configured until the transition has been completed.

« Storage recommendations Virtual memory »