Elastic doubles down on cloud native with Helm charts and CNCF membership
The shift to microservices and cloud native application architectures is a liberating paradigm for developers; but the dynamic nature of these newer frameworks also introduce a number of interesting challenges ranging from observability to inventory management. Users are turning to the Elastic Stack to solve these problems because the open source and flexible nature of Elastic products allows them to extend into uncharted space in a fast and agile manner.
For instance, eBay was the early adopter of Beats for monitoring its massive Kubernetes clusters and became an active contributor and thought partner for features like auto-discovery of services running on top of Kubernetes environments. Fairfax Media adopted a similar strategy as part of its operational use of the Elastic Stack. And, of course, our own infrastructure engineers and SREs make heavy use of the Elastic Stack to keep Elastic’s Kubernetes clusters humming.
And we're excited to make two announcements in this space today:
- The alpha release of Helm charts for Elasticsearch and Kibana, and
- Our membership in Cloud Native Computing Foundation (CNCF).
Observability for cloud native applications
While Kubernetes can dramatically simplify the task of application deployment in containers, it also adds some complexity for the day-to-day tasks of managing application performance and gaining visibility into services. In the dynamic world of microservices, containers, and Kubernetes, we need to rethink how we monitor logs, metrics, and tracing data -- often called the three pillars of observability -- in cloud native environments. Elastic supports all three.
Elastic Stack natively integrates with Kubernetes and makes it operationally simple to dynamically scale collection of logs and metrics from these environments. The Elastic data collectors for logs and metrics, called Filebeat and Metricbeat, deploy as a DaemonSet on top of Kubernetes. This ensures that these collectors only need to run on every Kubernetes node, not inside each container. They automatically add Kubernetes and Docker metadata about which container or pod the log or metric came from. They also support hints-based Kubernetes service auto-discovery, which enables configuration snippets to automatically collect and parse metrics and logs from common Kubernetes services, such as Prometheus exporters, Envoy service proxy, Traefik web proxy, and etcd datastore, as well as various applications reliant on those services.
Elastic APM is an open source approach to application performance management. If you are already housing logs and metrics in Elastic Stack, you can add APM to your deployment in a few lines of code using auto-instrumentation provided by Elastic APM agents. Elastic APM supports distributed tracing. Distributed tracing enables you to analyze performance bottlenecks in your microservices architecture all in one view. To accomplish this, the agents work together to pass a consistent trace ID from the initial web request to your front-end service to queries made to your back-end services. The agents also support OpenTracing, a set of vendor-neutral APIs that can be used to log request spans to a number of pluggable backends. Elastic APM translates calls to the OpenTracing API to Elastic APM and thus allows you to reuse existing instrumentation.
The Elastic Stack comes with powerful data visualization capabilities. Filebeat and Metricbeat modules, as well as, Elastic APM ship with pre-built Kibana dashboards that serve as a great starting point for exploring logs, metrics, and APM data in Kibana. On top of that, the Infrastructure, Logs, and APM UIs enable common workflows for correlating the data coming from different operational contexts. For instance, the Infrastructure UI has dedicated Kubernetes and Docker views that enable workflows specific to operators of large container orchestration deployments. Similarly, the APM UI has dedicated views for stringing together data coming from distributed traces across many microservices.
Announcing Helm charts for Elasticsearch and Kibana
We are excited to announce the initial public alpha of our Elasticsearch and Kibana Helm charts. These charts make it incredibly simple to deploy Elasticsearch and Kibana to Kubernetes in a manner of minutes.
As Kubernetes has continued to mature, the ability to effectively run stateful workloads has finally started become a reality. With this foundation in place, the timing was right for us to create these Helm charts as a way to help enable our users to run the Elastic Stack using modern, cloud-native deployment models and technologies.
Helm (an open source, CNCF-maintained project) is often touted as the missing package manager for Kubernetes. Helm charts are rapidly becoming a standard way of creating, publishing, and sharing Kubernetes applications. We love them because they provide a great way for us to package and share best practices and default configuration alongside our software.
To get started with Elasticsearch and Kibana Helm charts:
- Add the Elastic Helm Chart Repo:
helm repo add elastic https://helm.elastic.co - Install Elasticsearch:
helm install --name elasticsearch elastic/elasticsearch - Install Kibana:
helm install --name kibana elastic/kibana
For more information about the charts, the README files for the Elasticsearch and Kibana charts can be viewed on GitHub. For more information about how to use your new cluster, see Getting Started with Elasticsearch and Getting Started with Kibana.
A quick reminder that with great power comes great responsibility. These charts are still in alpha. We recommend sticking to non-production deployments. There is no guarantee that alpha versions will be compatible with subsequent release versions.
We want to hear from you! Please reach out if you encounter any issues or if you have ideas for improvements to the charts. Head to the discuss forums with any questions. You can open issues directly on the Helm chart GitHub repository.
Finally, we’re always looking know more about how you are using (or plan to use) Kubernetes. If you are interested in sharing, please fill out this short survey.
Elastic joins CNCF
Joining Cloud Native Computing Foundation (CNCF) is a natural evolution for us, given our open source history of building products and ongoing efforts around integrating with cloud native technologies.
CNCF is an open source software foundation that supports and promotes the use of open technologies and standards to build the next generation of applications and services. CNCF is home to projects such as Kubernetes, which are the basis of key trends in the industry, such as breaking up monolithic applications into microservices. Among its many efforts, CNCF has laid out a trailmap for how organizations should use its projects to transform the way they build applications.
Elastic is already aligned with the CNCF mission to promote open cloud native technologies. We build our products in an open and transparent way, and we support integrations with many key CNCF projects and CNCF member projects. We also have already adopted standards promoting vendor-neutral integrations, such as OpenTracing, in our products.
On becoming a member, Elastic is looking forward to working with CNCF on open tracing and metrics standards that help promote open source technologies and companies. We also look forward to deeper engagement with the CNCF community!
Come see us at KubeCon + CloudNativeCon North America 2018
We’re at KubeCon North America 2018 in Seattle this week. We will be showcasing Elasticsearch and Kibana Helm charts, as well as integration with Kubernetes, Prometheus, and OpenTracing for full stack observability of cloud native applications.
If you happen to be here too, please stop by booth S58 in the sponsor showcase to learn more about these announcements, and meet Elastic experts.