Serverless Elasticsearch Curator on AWS Lambda
In this post, we demonstrate how Elastic's Infrastructure team runs Elasticsearch Curator as a serverless application on AWS Lambda. We share the rationale, the tools, and of course, the code.
Servers, Services, and Serverless
As the team responsible for managing Elastic's internal systems, we want to build great systems without increasing the burden of system management. If we deploy a server instance to manage our existing infrastructure, we are creating overhead, which ultimately makes us less effective. To keep our overhead low and our effectiveness high, we often eschew servers for services and more recently, serverless computing.
Elastic Cloud
We use Elasticsearch to store and analyse all sorts of things. We keep system metrics from Metricbeat, logging from Puppet, even a running history of our GitHub issues to track our performance as a service team.
We use Elastic Cloud for our Elasticsearch clusters. The Cloud service gives us Elasticsearch, without expanding our server footprint, and thus our management overhead.
Serverless
We also run a selection of tasks as "serverless" processes on Lambda. This article focuses on our serverless approach to running Curator. With all the time-series and log data we collect, we certainly have a need for Curator, but it would be a shame to run servers just to host it.
Lambkin
Getting a function into Lambda in a repeatable, automated way is a reasonably complex task. We use our open-source tool Lambkin to reduce that complexity. Lambkin creates skeleton functions on demand and helps us publish, run, and schedule Lambda functions.
The Procedure
Here, we will step through a process for setting up a serverless Elasticsearch Curator system identical to the one we use internally. The example is implemented in Python, the language used for both Curator and Lambkin. To follow along, you'll need a Python environment with the "pip" command available. Deep knowledge of Python is not required, however. The solution provided will run as-is, and is configurable for your environment by editing a simple YAML file.
Let's go:
Install Lambkin and virtualenv
sudo pip install lambkin virtualenv
Configure AWS credentials and default region
If you don't already have your AWS account configured, a simple way to do it is:
sudo pip install awscli aws configure
More detail is available in the AWS CLI documentation.
Create a new skeleton Lambda function
lambkin create serverless-curator cd serverless-curator
In the new serverless-curator directory, you'll find some skeleton files. These files make up a valid, ready-to-run Lambda function in Python, and a context for managing any dependencies it might have.
Create the Python function
The primary file is serverless-curator.py. It contains the body of our Python function. Feel free to examine it if you're interested.
The function doesn't do much yet, but it's already possible to publish and run it on Lambda:
lambkin publish --description='Just a test.' lambkin run
You should then see some output from Lambda, ending with a JSON object returned by the sample function. Like this:
{"from": "Python", "hello": "World"}
Replace the skeleton function
It's time to replace the example function with something more useful. Full source code for a working Curator function is provided in this Gist.
The function makes use of a YAML configuration file where you can declare index patterns across multiple Elasticsearch clusters. Be sure to create the file serverless-curator.yaml. A configuration example is also provided in the Gist.
Install Python requirements
Our new function requires some library packages from the Python Package Index, so we need to ensure they will be available in Lambda. Edit the requirements.txt file, changing its contents to:
certifi==2016.8.8 elasticsearch-curator==4.0.6 PyYAML==3.11
Then install the packages:
lambkin build
The function's requirements are now installed. Lambkin uses virtualenv to ensure that each function gets its own isolated dependencies.
Publish and try the new version
We'll also update the description, and set a long timeout in case we will be processing a lot of indices:
lambkin publish --description='Elasticsearch Curator' --timeout=300 lambkin run
The function will return (in JSON format), a dictionary showing which indices were deleted (if any).
Schedule the function
If you're happy with the results, Lambkin can arrange to have the function run on a regular schedule:
lambkin schedule --rate '1 hour'
At this point, you have a reliable, regularly scheduled Curator job running in a serverless environment.
Now would be a great time to check your new function into version control. When you want to change the configuration, perhaps to accommodate new indices, just edit the YAML config, and do a "lambkin publish".
Wrap Up
If you'd like to remove the function from Lambda, try these commands:
lambkin list-published lambkin unpublish --help
If you'd like to explore Lambkin further, try "lambkin --help", or come and join the conversation on GitHub.