Istio

The following sections describe how to connect the operator and managed resources to the Istio service mesh. It is assumed that Istio is already installed and configured on your Kubernetes cluster. If you are new to Istio, refer to the product documentation for more information and installation instructions.

These instructions have been tested with Istio 1.6.1. Some details may differ on older or newer versions of Istio and might require additional configuration steps not documented here.

Some Elastic Stack features such as Kibana alerting and actions rely on the Elasticsearch API keys feature which requires TLS to be enabled at the application level. If you intend to make use of these features, you should not disable the self-signed certificate on the Elasticsearch resource and enable PERMISSIVE mode for the Elasticsearch service through a DestinationRule or PeerAuthentication resource. Strict mTLS mode is currently not compatible with Elastic Stack features requiring TLS to be enabled for the Elasticsearch HTTP layer.

If you are using a Kubernetes distribution such as Minikube which does not have support for issuing third-party security tokens, you should explicitly set automountServiceAccountToken field to true in the pod templates to allow Istio to fallback to default service account tokens. Refer to Istio security best practices for more information.

Connect the operator to the Istio service mesh

edit

The operator itself must be connected to the service mesh in order to deploy and manage Elastic Stack resources that you wish to connect to the service mesh. This is achieved by injecting an Istio sidecar to the ECK operator pods. The following instructions assume that automatic sidecar injection is enabled on your cluster via a mutating admissions webhook. Refer to Istio injection documentation if you prefer a different method of injection.

Create the elastic-system namespace and enable sidecar injection:

kubectl create namespace elastic-system
kubectl label namespace elastic-system istio-injection=enabled

Install ECK:

kubectl apply -f https://download.elastic.co/downloads/eck/1.2.2/all-in-one.yaml

Check the configuration to ensure that the installation has been successful:

kubectl get pod elastic-operator-0 -n elastic-system -o=jsonpath='{range .spec.containers[*]}{.name}{"\n"}'

If the output of the above command contains both manager and istio-proxy, ECK has been successfully installed with the Istio sidecar injected.

Connect Elastic Stack applications to the Istio service mesh

edit

This section assumes that you are deploying Elasticsearch, Kibana, or APM Server resources to a namespace that has automatic sidecar injection enabled.

If you have configured Istio in permissive mode, examples defined elsewhere in the ECK documentation will continue to work without requiring any modifications. However, if you have enabled strict mutual TLS authentication between services either via global (MeshPolicy) or namespace-level (Policy) configuration, the following modifications to the resource manifests are necessary for correct operation.

Elasticsearch

edit

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elastic-istio
spec:
  version: 8.15.3
  http:
    tls: 
      selfSignedCertificate:
        disabled: true
  nodeSets:
  - name: default
    count: 3
    podTemplate:
      metadata:
        annotations:
          traffic.sidecar.istio.io/includeInboundPorts: "*"
          traffic.sidecar.istio.io/excludeOutboundPorts: "9300" 
          traffic.sidecar.istio.io/excludeInboundPorts: "9300"
      spec:
        automountServiceAccountToken: true

	Disable the default self-signed certificate generated by the operator and allow TLS to be managed by Istio. Note that disabling the self-signed certificate might interfere with some features such as Kibana Alerting and Actions.
	Exclude the transport port (port 9300) from being proxied. Currently ECK does not support switching off X-Pack security and TLS for the Elasticsearch transport port. If Istio is allowed to proxy the transport port, the traffic will be encrypted twice and communication between Elasticsearch nodes will be disrupted.
	Optional. Only set `automountServiceAccountToken` to `true` if your Kubernetes cluster does not have support for issuing third-party security tokens.

If you do not have automatic mutual TLS enabled, you may need to create a Destination Rule to allow the operator to communicate with the Elasticsearch cluster. A communication issue between the operator and the managed Elasticsearch cluster can be detected by looking at the operator logs to see if there are any errors reported with the text 503 Service Unavailable.

kubectl logs -f -n elastic-system -c manager statefulset.apps/elastic-operator

If the operator logs indicate a communications problem, create a DestinationRule to enable mutual TLS between the operator and the affected Elasticsearch cluster. For example, the following rule enables mutual TLS for a specific Elasticsearch cluster named elastic-istio deployed to the default namespace.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: elastic-istio
spec:
  host: "elastic-istio-es-http.default.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

Refer to the Istio documentation for more information about other configuration options affecting authentication between services.

Using init containers with Istio CNI

edit

There are known issues with init containers when Istio CNI is configured. If you use init containers to install Elasticsearch plugins or perform other initialization tasks that require network access, they may fail due to outbound traffic being blocked by the CNI plugin. To work around this issue, explicitly allow the external ports used by the init containers.

Installing plugins using an init container.

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elastic-istio
spec:
  version: 8.15.3
  http:
    tls:
      selfSignedCertificate:
        disabled: true
  nodeSets:
  - name: default
    count: 3
    podTemplate:
      metadata:
        annotations:
          traffic.sidecar.istio.io/includeInboundPorts: "*"
          traffic.sidecar.istio.io/excludeOutboundPorts: "9300,443" 
          traffic.sidecar.istio.io/excludeInboundPorts: "9300"
      spec:
        automountServiceAccountToken: true
        initContainers:
          - name: install-plugins
            command:
              - sh
              - -c
              - |
                bin/elasticsearch-plugin install --batch repository-gcs

Plugins are downloaded over the HTTPS port (443) and needs to be allowed when Istio CNI is installed.

Kibana

edit

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: elastic-istio
spec:
  version: 8.15.3
  count: 1
  elasticsearchRef:
    name: elastic-istio
  http:
    tls: 
      selfSignedCertificate:
        disabled: true
  podTemplate:
    spec:
      automountServiceAccountToken: true

	Disable the default self-signed certificate generated by the operator and allow TLS to be managed by Istio.
	Optional. Only set `automountServiceAccountToken` to `true` if your Kubernetes cluster does not have support for issuing third-party security tokens.

APM Server

edit

apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: elastic-istio
spec:
  version: 8.15.3
  count: 1
  elasticsearchRef:
    name: elastic-istio
  http:
    tls: 
      selfSignedCertificate:
        disabled: true
  podTemplate:
    metadata:
      annotations:
        sidecar.istio.io/rewriteAppHTTPProbers: "true" 
    spec:
      automountServiceAccountToken: true

	Disable the default self-signed certificate generated by the operator and allow TLS to be managed by Istio.
	Automatically re-write the health checks to go through the proxy.
	Optional. Only set `automountServiceAccountToken` to `true` if your Kubernetes cluster does not have support for issuing third-party security tokens.

« Service meshes Linkerd »