Prepare the Kubernetes environment and deploy a sample application

edit

Prepare the Kubernetes environment and deploy a sample application

edit

Assign Kubernetes roles

edit

Logging and metrics tools like kube-state-metrics, Filebeat, Fluentd, Metricbeat, Prometheus, etc. get deployed in the kube-system namespace and have access to all namespaces. Create the cluster wide role binding to allow the deployment of kube-state-metrics and the Beats DaemonSets using the Role Based Access Control (RBAC) api:

kubectl create clusterrolebinding cluster-admin-binding  \
  --clusterrole=cluster-admin --user=$(gcloud config get-value account)

Deploy kube-state-metrics

edit

Kube-state-metrics is a service that exposes metrics and events about the state of the nodes, pods, containers, etc. The Metricbeat kubernetes module will connect to kube-state-metrics. Check to see if kube-state-metrics is running:

kubectl get pods --namespace=kube-system | grep kube-state

Create it if needed (by default it will not be there).

git clone https://github.com/kubernetes/kube-state-metrics.git
kubectl create -f kube-state-metrics/kubernetes
kubectl get pods --namespace=kube-system | grep kube-state

Clone the Elastic examples Github repo

edit

git clone https://github.com/elastic/examples.git

The remainder of the steps will refer to files from this repo. Change directory into examples/GKE-on-Prem-logging-and-metrics.

Example application

edit

If you are just getting started with GKE On-Prem and do not have anything running you can use a sample guestbook application from the Kubernetes engine documentation. The YAML has been concatenated into a single manifest and some changes have been made to serve as an example for enabling Beats to autodiscover the components of the application. Whether or not you deploy the example application, this documentation will refer to specific parts of the guestbook.yaml manifest file.

Network considerations

edit

Before you deploy the sample application manifest, have a look at the frontend service in GKE-on-Prem-logging-and-metrics/guestbook.yaml. You may need to edit this service so that the service is exposed to your internal network. The network topology of the lab where this example was developed has a load balancer in front of the GKE On-Prem environment. Therefore the service specifies an IP address associated with the load balancer. Your configuration will likely be different.

apiVersion: v1
kind: Service
metadata:
  name: frontend
  labels:
    app: guestbook
    tier: frontend
spec:
  type: LoadBalancer
  ports:
  - port: 80
    protocol: TCP
  selector:
    app: guestbook
    tier: frontend
  loadBalancerIP: 10.0.10.42

Edit the file guestbook.yaml as appropriate to integrate with your network.

Label your application pods

edit

The Beats autodiscover functionality is facilitated by Kubernetes metadata. In the example manifest there are metadata labels assigned to the deployments and the Filebeat and Metricbeat configurations are updated to expect this metadata.

These lines from the guestbook.yaml manifest file add the app: redis label to the Redis deployments:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: redis-master
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: redis

	This label is added to the metadata for the k8s deployment and is applied to each pod in the deployment.
	You should create labels that are appropriate for your use case, `app: redis` is only an example.

These lines from the filebeat-kubernetes.yaml manifest file configure Filebeat to autodiscover Redis pods that have the appropriate label:

   filebeat.autodiscover:
      providers:
        - type: kubernetes
          templates:
            - condition.contains: 
                kubernetes.labels.app: redis 
              config:
                - module: redis

	Specifies that the condition is looking for a substring and not an exact match
	The label to inspect, and the substring to look for
	The module to use when collecting, parsing, indexing, and visualizing logs from pods that meet the condition

If you are using the example application to get started with GKE On-Prem and the Elastic Stack, deploy the sample application.

kubectl create -f guestbook.yaml

If you are ready to manage logs and metrics from your own application, examine your pods for existing labels and update the Filebeat and Metricbeat autodiscover configuration within filebeat-kubernetes.yaml and metricbeat-kubernetes.yaml respectively. See the documentation for configuring Filebeat autodiscover and Metricbeat autodiscover. You will also need the list of Filebeat modules and Metricbeat modules.

« Architecture Deploy Beats »