Tips to secure Elasticsearch clusters for free with encryption, users, and more

With the release of the Elastic Stack 6.8 and 7.1, we’ve made a number of Elasticsearch security features more widely available to everyone by making them free in the default distribution. We’re really excited to share this, but one of the new challenges now is to update all of our documentation and guidance on how to secure the Elastic Stack. Gone are the days of relying on just a proxy server in front of the Stack to secure it!

I’ve gone through most of our old popular security-focused blog posts, and I’m going to update the guidance, dispel some myths, point out some new ideas, and make sure you can secure your Stack better than ever before.

I’m not going to be linking to any of the old articles; some give out advice that we now consider dangerous due to how the Elastic Stack has changed over the years. (Be wary of  security advice from old blogs, things change pretty quickly.) We will be forwarding all those links to this page, but if you come upon another site serving cached versions of those articles, forward them a link to this one.

TLS encryption

TLS encryption is now part of our free Elasticsearch and Kibana security within the default distribution. The purpose of TLS is to encrypt network communications. We encrypt the communication between the nodes in your cluster, between the clients and the cluster, as well as connections from Kibana to the users.

We also use TLS certificates as a way to ensure a node should be allowed to join a cluster. Without TLS, it’s possible for anyone to add a node to a cluster. Although this could be abused by an attacker for a variety of nefarious purposes, the far more common case is when a node accidentally joins the wrong cluster. TLS can help avoid these problems because without the proper certificate a node cannot join the cluster.

Prior to 6.8 and 7.1, if you needed TLS for your Elastic Stack while using a Basic license you would have to rely on some sort of proxy, which adds a considerable amount of setup. TLS is already pretty difficult to configure even without having to set up a proxy between the nodes and the clients. By adding a proxy, you would have taken a hard problem and made it substantially harder. We can now leverage the TLS support directly in the Elastic Stack.

We have quite a bit of documentation on how to configure TLS. If you’ve worked with TLS, you know it’s never easy to set up certificates, but we have built-in tools to help generate cluster-wide certificates and certificate authorities to make things easier.  Watch this space for some future blog posts that go into detail about how to best configure TLS in all of the Elastic Stack components. The two easiest ways to get started today? Check out this Getting started with Elasticsearch security blog post, or take a look at our official Kubernetes Operator.

Authentication

With versions 6.8 and 7.1, native authentication is available to all. The purpose of authentication is to allow clients to identify themselves to the Elastic Stack. This is often known as the username and password you use to log in to a system. It’s generally a bad idea to leave a cluster full of data open to the whole world. It’s especially dangerous if the cluster is connected directly to the Internet where anyone can connect without using a password.

In the past, the advice was to use an Nginx server with basic auth enabled between the client and the cluster. Getting this configuration correct could be incredibly difficult. Since every node can service external requests, if you forgot to firewall off a node, anyone could connect to the cluster without authentication.

We allow authentication using the native and file realms for free now. The file realm stores user information in a file on every node. You have to keep the files synced on all the nodes. The native realm is a more streamlined experience and does about what one would expect: The user data is stored in an index on the cluster every node has access to. You add usernames and passwords via a REST API, and then those accounts can log in. See the Getting started with Elasticsearch security blog for more details.

If you have more complex authentication needs, such as LDAP, Active Directory, SAML, or something else, we offer a Cloud (SaaS) service with some of these as well as other commercial features baked right in, or you can get in touch with us to discuss possible on-prem solutions. To see the different authentication options we offer, be sure to check out the authentication docs.

Authorization

Authorization and authentication tend to go hand in hand; the Elastic Stack is no different. We have a very flexible system that can define authorization rules for the cluster and indices, all the way down to documents and fields. We even have something called an authorization engine where you can build your own authorization plugin.

Historically, advice has been pretty rough in this space. There are some ways that have been suggested in the past to write rather complex filtering rules with a proxy like Nginx, and then trying to leverage filtered aliases to achieve something that almost looks like authorization. We strongly suggest everyone avoids using these methods for authorization. They are not security features, probably won’t do what you want, and are incredibly fragile to maintain. These solutions won’t do what you really want anymore because over time the number of Elasticsearch APIs has grown substantially. There are many ways to access and query indices and the data they contain. There’s no way to ensure you’ve blocked or restricted every single request. The second reason is related to the first, because there are so many endpoints to protect and we’re always adding new ones. It’s going to be a massive amount of work to keep those filters updated.

Luckily, you don’t have to worry about this anymore. In 6.8 and 7.1, you have access to a substantial number of authorization features. There are too many to describe here in this post, but the Elastic Stack security docs do a great job of explaining all of the existing privileges. We will spend time in future blog posts to explain some of the solutions that can be built using our permission model.

Like authentication, we have more advanced authorization features that can be applied to individual documents and fields. You can build your own authentication and authorization plugins, even log the security events on the cluster. You can make use of these as part of our Cloud service or get in touch with us to discuss your on-prem options.

Put your Stack behind a VPN and/or firewall

There has been a fair amount of past advice suggesting you place the Elastic Stack behind a firewall or VPN. The purpose behind much of this advice was because without TLS and authentication it could be difficult to protect a cluster using only a proxy, so just blocking access to everyone was easier than trying to get all the moving parts properly configured.

If this is something you still want to do, even with TLS and authentication enabled, it can still be a great idea. Security works best when there are multiple layers. The one nice part of this now is the VPN and firewall aren’t the only layer.

Restricted scripts

We once long ago had a blog post about restricting scripting in Elasticsearch. With the creation of the Painless scripting language it’s much harder to bring down a cluster, even with scripts with ill intent. It’s not impossible, though, and we have some best practices in the scripting security docs that discuss our current stance on this topic. Once again though, you should always review your environment and take the steps that make the most sense to you. Security works best in layers, so there isn’t one single best option.

Isolation

There’s been a number of times that isolating your cluster and nodes for security reasons has been mentioned. This can be everything from containers to firewalls to IP filtering.

This is still great advice and isolating everything you can is a great security tip. In the last few years, we’ve done a lot to make this easier than ever before. You can use our official container images published on Docker Hub. You can leverage the Elasticsearch Service, and we’ll take care of all the heavy lifting. And we just announced an official Kubernetes Operator that you can take for a spin.

Back up your indices (especially .kibana because it can be easily changed)

Backing up your data is still just as important as it’s ever been. Security doesn’t change that in any way, but we do get to teach the old dog some new tricks.

The biggest advantage is we can use Kibana Spaces to restrict who can modify certain dashboards. This, too, has been released for free as part of 6.8 and 7.1. In the past, if you could access Kibana you could probably change anything you wanted. It would not be uncommon for someone to mess up the dashboards or in some cases even accidentally delete them all. The advice was to back up your .kibana index on a regular basis so you could recover from mistakes quickly. Restoring an index is a lot faster than rebuilding all of your dashboards. Backing up .kibana is still important, but you shouldn’t need to restore it on a regular basis thanks to spaces and authorization.

Another great advantage to security is the ability to control who can take snapshots, create an index, and modify the data. Without security, a user who should only be able to read could make catastrophic changes to the data. With security enabled, we can take some steps to hopefully prevent accidental changes to important data.

What now?

The purpose behind this post was to revisit some of our previous advice on how to secure your Elastic Stack deployments. In the world of security, the only constant is change. Even the advice in this blog will be old and wrong someday. Security in the Elastic Stack can be daunting due to the number of possible options. (That’s a feature, not a bug.) Be sure to ask questions in the forums, and read our blogs and documentation.

If you’d prefer an easy button for securing your Elastic Stack, we have a few options. Our Elasticsearch Service comes with TLS configured for all clusters, authentication and authorization enabled, as well as our Enterprise security features enabled by default. If cloud isn’t your thing, you can get many of the same features in the Elasticsearch Service in your own datacenter with Elastic Cloud Enterprise (ECE). We also have a Kubernetes Operator that can take care of some of the heavy lifting, such as enabling TLS and configuring authentication for you by default. And of course if you’re an Enterprise customer running your own Stack, you can use our fantastic support team to help you sort out any problems that may arise.

Stay tuned for everything we have in store in the future. We have a lot of How Tos planned and even plenty of new security features you’ll see in the future. It’s a very exciting time to be using the Elastic Stack!