Configure the validating webhook
editConfigure the validating webhook
editECK can be configured to provide a validating webhook that validates Elastic custom resources (Elasticsearch, Kibana, APM Server, Enterprise Search, Beats, Elastic Agent, Elastic Maps Server, and Logstash) before they are created or updated. Validating webhooks provide immediate feedback if a submitted manifest contains invalid or illegal configuration — which can help you catch errors early and save time that would otherwise be spent on troubleshooting.
Validating webhooks are defined using a ValidatingWebhookConfiguration
object that defines the following:
- Type of resource to validate (Elasticsearch, Kibana and so on)
- Type of actions to validate (create, update, delete)
-
Connection details to the webhook
- Kubernetes service name and namespace
- Request path
- CA certificate for verifying the server
- Failure policy if the webhook is unavailable (block the operation or continue without validation)
Defaults provided by ECK
editWhen using the default operator.yaml
manifest, ECK is installed with a ValidatingWebhookConfiguration
configured as follows:
- Validate all known Elastic custom resources (Elasticsearch, Kibana, APM Server, Enterprise Search, Beats, Elastic Agent, Elastic Maps Server, and Logstash) on create and update.
-
The operator itself is the webhook server — which is exposed through a service named
elastic-webhook-server
in theelastic-system
namespace. -
The operator generates a certificate for the webhook and stores it in a secret named
elastic-webhook-server-cert
in theelastic-system
namespace. This certificate is automatically rotated by the operator when it is due to expire.
Manual configuration
editIf you installed ECK without the webhook and want to enable it later on, or if you want to customise the configuration such as providing your own certificates, this section describes the options available to you.
Configuration options
editYou can customise almost all aspects of the webhook setup by changing the operator configuration.
Configuration option | Default value | Description |
---|---|---|
|
false |
This must be set to |
|
true |
Set to |
|
/tmp/k8s-webhook-server/serving-certs |
Path to mount the certificate. |
|
elastic-webhook.k8s.elastic.co |
Name of the |
|
elastic-webhook-server-cert |
Name of the secret containing the certificate for the webhook server. |
|
9443 |
Port to listen for incoming validation requests. |
Using your own certificates
editThis section describes how you can use your own certificates for the webhook instead of letting the operator manage them automatically. There are a few important things to be aware of when going down this route:
- You have to keep track of the expiry dates and manage the certificate rotation yourself. Expired certificates may stop the webhook from working.
- The secret containing the custom certificate must be available when the operator starts.
-
You must update the
caBundle
fields in theValidatingWebhookConfiguration
yourself. This must be done at the beginning and whenever the certificate is rotated.
Use a certificate signed by your own CA
edit-
The certificate must have a Subject Alternative Name (SAN) of the form
<service_name>.<namespace>.svc
(for exampleelastic-webhook-server.elastic-system.svc
). A typical OpenSSL command to generate such a certificate would be as follows:openssl req -x509 -sha256 -nodes -newkey rsa:4096 -days 365 -subj "/CN=elastic-webhook-server" -addext "subjectAltName=DNS:elastic-webhook-server.elastic-system.svc" -keyout tls.key -out tls.crt
-
Create a secret in the namespace the operator would be deployed to. This secret must contain the certificate under the
tls.crt
key and the private key under thetls.key
key.kubectl create secret -n elastic-system generic elastic-webhook-server-custom-cert --from-file=tls.crt=tls.crt --from-file=tls.key=tls.key
-
Encode your CA trust chain as base64 and set it as the value of the
caBundle
fields in theValidatingWebhookConfiguration
. Note that there are multiplecaBundle
fields in the webhook configuration. -
Install the operator with the following options:
-
Set
manage-webhook-certs
tofalse
-
Set
webhook-secret
to the name of the secret you have just created (elastic-webhook-server-custom-cert
)
-
Set
If you are using the Helm chart installation method, you can install the operator by running this command:
helm install elastic-operator elastic/eck-operator -n elastic-system --create-namespace \ --set=webhook.manageCerts=false \ --set=webhook.certsSecret=elastic-webhook-server-custom-cert \ --set=webhook.caBundle=$(base64 -w 0 ca.crt)
Use a certificate from cert-manager
editThis section describes how to use cert-manager to manage the webhook certificate. It assumes that there is a ClusterIssuer
named self-signing-issuer
available.
-
Create a new certificate
apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: elastic-webhook-server-cert namespace: elastic-system spec: dnsNames: - elastic-webhook-server.elastic-system.svc issuerRef: kind: ClusterIssuer name: self-signing-issuer secretName: elastic-webhook-server-cert subject: organizations: - example
-
Annotate the
ValidatingWebhookConfiguration
to inject the CA bundle:apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration metadata: annotations: cert-manager.io/inject-ca-from: elastic-system/elastic-webhook-server-cert name: elastic-webhook.k8s.elastic.co webhooks: [...]
-
Install the operator with the following options:
-
Set
manage-webhook-certs
tofalse
-
Set
webhook-secret
to the name of the certificate secret (elastic-webhook-server-cert
)
-
Set
If you are using the Helm chart installation method, you can install the operator by running the following command:
helm install elastic-operator elastic/eck-operator -n elastic-system --create-namespace \ --set=webhook.manageCerts=false \ --set=webhook.certsSecret=elastic-webhook-server-cert \ --set=webhook.certManagerCert=elastic-webhook-server-cert
Disable the webhook
editTo disable the webhook, set the enable-webhook
operator configuration flag to false
and remove the ValidatingWebhookConfiguration
named elastic-webhook.k8s.elastic.co
:
kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io elastic-webhook.k8s.elastic.co
Troubleshooting
editYou might get errors in your Kubernetes API server logs indicating that it cannot reach the operator service (elastic-webhook-server
). This could be because no operator pods are available to handle request or because a network policy or a firewall rule is preventing the control plane from accessing the service. To help with troubleshooting, you can change the failurePolicy
of the webhook configuration to Fail
. This will cause create or update operations to fail if there is an error contacting the webhook. Usually the error message will contain helpful information about the failure that will allow you to diagnose the root cause.
Resource creation taking too long or timing out
editWebhooks require network connectivity between the Kubernetes API server and the operator. If the creation of an Elasticsearch resource times out with an error message similar to the following, then the Kubernetes API server might be unable to connect to the webhook to validate the manifest.
Error from server (Timeout): error when creating "elasticsearch.yaml": Timeout: request did not complete within requested timeout 30s
If you get this error, try re-running the command with a higher request timeout as follows:
kubectl --request-timeout=1m apply -f elasticsearch.yaml
As the default failurePolicy
of the webhook is Ignore
, this command should succeed after about 30 seconds. This is an indication that the API server cannot contact the webhook server and has foregone validation when creating the resource.
On GKE private clusters, you may have to add a firewall rule allowing access to port 9443 from the API server so that it can contact the webhook. Check the GKE documentation on firewall rules and the Kubernetes issue for more details.
It is possible that a network policy is blocking any incoming requests to the webhook server. Consult your system administrator to determine whether that is the case, and create an appropriate policy to allow communication between the Kubernetes API server and the webhook server. For example, the following network policy simply opens up the webhook port to the world:
kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: allow-webhook-access-from-any namespace: elastic-system spec: podSelector: matchLabels: control-plane: elastic-operator ingress: - from: [] ports: - port: 9443
If you want to restrict the webhook access only to the Kubernetes API server, you must know the IP address of the API server, that you can obtain through this command:
kubectl cluster-info | grep master
Assuming that the API server IP address is 10.1.0.1
, the following policy restricts webhook access to just the API server.
kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: allow-webhook-access-from-apiserver namespace: elastic-system spec: podSelector: matchLabels: control-plane: elastic-operator ingress: - from: - ipBlock: cidr: 10.1.0.1/32 ports: - port: 9443
Updates failing due to validation errors
editIf your attempts to update a resource fail with an error message similar to the following, you can force the webhook to ignore it by removing the kubectl.kubernetes.io/last-applied-configuration
annotation from your resource.
admission webhook "elastic-es-validation-v1.k8s.elastic.co" denied the request: Elasticsearch.elasticsearch.k8s.elastic.co "quickstart" is invalid: some-misspelled-field: Invalid value: "some-misspelled-field": some-misspelled-field field found in the kubectl.kubernetes.io/last-applied-configuration annotation is unknown