ECK in production environment
This article will guide you on how to spin up an ECK environment ready for production which contains,
- dedicated stack monitoring,
- fleet-server & elastic-agent (with kubernetes integration),
- elasticsearch autoscaling,
- nodeAffinity & nodeSelector,
- SAML with auth0,
- hot, warm, cold, frozen architecture,
- heartbeat monitor SSL certificate expiration,
- external-dns (with CloudFlare integration),
- cert-manager (with let's encrypt integration),
- ingress controller (using Cloudflare to register the endpoints),
- Optional: esrally to validate autoscaling and run benchmark against es cluster
Remember you must have basic or enterprise license to run ECK.
Demo
To see the demo and files, please access https://github.com/framsouza/eck-ready-for-production
cert-manager
cert-manager adds certificates and certificate issuers as resource types in Kubernetes, and simplifies the process of obtaining, renewing and using those certificates. It's very command Kubernetes administrator use cert-manager to handle certificate, and on this example we are going to use cert-manager with let's encrypt to access Kibana. Remember that, TLS certificates for the Elasticsearch transport layer that are used for internal communications between Elasticsearch nodes are managed by ECK and cannot be changed.
ingress
Ingress controller is specialized load balancer for Kubernetes, which accepts traffic from outside the Kubernetes cluster and balances it to pods.
external-dns
It's a addon that configures public DNS servers about exposed Kubernetes services, on this examples we are integrating external-dns with Cloudflare. For each Ingress/Service resource you us, a DNS entry will created on Cloudflare with the respective IP address, on external-dns logs you should be able to see the following:
level=info msg="Using inCluster-config based on serviceaccount-token"
level=info msg="Created Kubernetes client https://10.76.0.1:443"
level=info msg="Changing record." action=CREATE record=kibana.framsouza.co ttl=1 type=A zone=4cd4c7c1cb8f7bf3a7482749654ae6fb
level=info msg="Changing record." action=CREATE record=monitoring.framsouza.co ttl=1 type=TXT zone=4cd4c7c1cb8f7bf3a7482749654ae6fb
How-to setup
Make sure to respect the commands execution order:
- Create GKE cluster with Kubernetes type hot, warm, cold, frozen for each dedicated node pool, make sure you will have enough resource to run the pods in the nodes.
- Create a cluster role mapping that gives you permission to install ECK operator
- kubectl create clusterrolebinding cluster-admin-binding --cluster-role=cluster-admin --user=<USERNAME>
- Install ECK operator
- helm repo add elastic https://helm.elastic.co && helm repo update && helm install elastic-operator elastic/eck-operator -n elastic-system --create-namespace
- Create dedicated storage class by applying files storageclass-hot.yaml and storageclass-warm.yaml
- Download your license and apply it via secret (or apply the license.yaml)
- kubectl create secret generic eck-license --from-file <LICENSE-PATH> -n elastic-system && kubectl label secret eck-license "license.k8s.elastic.co/scope"=operator -n elastic-system
- Create the monitoring cluster (it will create a ns call monitoring) by applying monitoring-es.yaml
- Create elasticsearch resource, elasticsearch.yaml
- Create kibana resource, kibana.yaml
- Create fleet resource, fleet.yaml
- Create heartbeat, heartbeat.yaml
- Install external-dns
- kubectl apply -f external-dns.yml
- Install cert-manager
- kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.7.1/cert-manager.yaml
- Install ingress-nginx ,
- helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace
- Create ingress resource, ingress.yaml
- Create cluster issuer apply, clusterissuer.yaml
- Create let's encrypt certificate, certificate.yaml
Accessing
For this example, I am using a domain call framsouza.co and as I am using external-dns, the DNS entry will be automatically added to Cloudflare, https://kibana.framsouza.co and https://monitoring.framsouza.co. You can check the connection is safe and we are using a valid certificate by let's encrypt.
If you want to login using SAML, make sure to adjust the saml session on elasticsearch.yml according to your environment.
Autoscaling validation
(Optional)
To confirm if autoscaling is working as expected, you can use esrally to test that and run benchmark against your cluster. Make sure to have params-file.json with the following content:
{
"number_of_replicas": 1,
"number_of_shards": 2
}
then you just need to run and wait a couple of hours until the test is finished.
docker run -v /tmp/params-file.json:/tmp/params-file.json elastic/rally race --track=http_logs --target-hosts=${IP}:9200 --pipeline=benchmark-only --client-options="timeout:60,use_ssl:true,verify_certs:false,basic_auth_user:'elastic',basic_auth_password:'${PASSWORD}'" --track-params=/tmp/params-file.json
Have a look at esrally-result.txt.