Discovering the Need for an Indexing Strategy in Multi-Tenant Applications
UPDATE: This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.
There are many buzzwords that may be applied to Elasticsearch. Multi-tenancy is one. Getting started with a multi-tenant use case can be deceptively easy - there are some pitfalls that will require a little careful design.
What is Multi-Tenancy?
Multi-tenancy means that one has a system with multiple tenants. Depending on the project you might refer to a tenant as a user, an organisation, a project, an application, a client or something different. If each of your tenants use Elasticsearch for entirely different things then what you’re actually doing is providing Elasticsearch as service with a shared cluster model - not a good idea. If you wonder why we think shared clusters is a bad idea, please see our article on Elasticsearch in production.
If, however, each tenant has an instance of the same application that relies on Elasticsearch, there is a commonality between the tenants, both in terms of queries executed and how data is structured. Even stronger commonality exists when the application itself is multi-tenant, like a software as a service scenario. In either case, an important criteria of multi-tenant solutions is to ensure that one tenant is not able to observe any data belonging to another tenant. I will simply refer to this as the data separation constraint.
Example: Blog Hosting as a Service
As an example througout this article we use a fictive blog hosting provider. It is fictive in the sense that it I’m not thinking of anybody in particular - and any similarity is accidental - however I wouldn’t be suprised if you find a similar story on the tech blog of an actual blog hosting provider. As their competitors, they provide a freemium model, where the smallest package is free.
Using One Index Per Tenant
For their proof of concept they started out with separate indexes per tenant. By simply appending the unique ID of each tenant to the index name, their proof of concept was up and running in no time and the data separation constraint was satisified.
Once the proof of concept was completed they wanted to put Elasticsearch to the test and started loading one percent of their production data into a test cluster. Lucky that they actually did this test, as the results where no good. The test cluster ran out of memory half way during the import. The engineers where anxious about these results because they had convinced the management that Elasticsearch would be so much faster.
After some analysis the engineers figured out that when the the first out-of-memory error happened, the amount of data indexed actually corresponded to only a fraction of the memory available to the instance. Clearly, there was an issue with configuration here, and they started reading Sizing Elasticsearch only to learn that there is a base cost of memory for each shard and that this base cost is constant, even if the shard contains no documents. They did the test again with only one shard per index (default is five) and this helped, but their total shard count per Elasticsearch instance was still too high. Since every index has to have at least one shard, their freemium model made it impossible to have one index per tenant. Or as one of the team members put it: “We cannot reserve memory for idle customers that don’t pay us any money.”
Custom Routing
The fact that this was not an issue with the one percent dataset led them to realize that the CPU cost for warming the cache for a single tenant was not related to the amount of data that tenant had, but in fact it was related to the size of the entire data set and the total number of shards. Both of these parameters were quite different between the initial performance test and the full-scale test. After reading Elasticsearch from the top down they understood more about the way routing works in Elasticsearch and the problem became obvious: “For every tenant cache that needed to be populated, no matter how small the tenant actually is, every shard in the cluster was queried.”
Being used to the advanced query optimizers common in relational databases they felt they had moved to an inferior technology, but they soon discovered that Elasticsearch has support for defining custom routing. They changed the index settings to use the tenant ID as a routing key, reindexed the full data set and added the new routing key to all the queries in the application as well. Now they finally got similar performance regardless of the number of tenants in the data set.
They realized, of course, that some tenants are larger than others and wanted to reduce the probability of two large tenants ending up in the same shard and possibly making that shard too large for a single instance. This led them to choose a higher than otherwise necessary number of shards for the indexes.
Tenants Grow Both in Data and Traffic
After about a year in production, the entire organization was very happy with the move to Elasticsearch. The developers used new aggregations features and the operations team had good routines for replacing and adding instances as necessary. Elasticsearch even took the size of shards into consideration during allocation so that the total amount of data on each instance remained fairly even.
Of course, things are never that easy in the long run… The largest tenant kept pushing the size of the shard it was in and operations realized that it was getting more and more expensive and difficult to scale up the instance that shard was residing on. This led to a hotfix of the application with a bunch of special cases for that tenant ID so that it could have an index of its own with multiple shards for that tenant. After a while they also realized that this enabled them to increase the number of replicas for just this tenant, which helped to distribute the search load incurred by this tenant over more instances.
Nobody Can Tell the Future
The developers were not happy about the special cases in their code relating to a specific tenant, but they realized that the extreme differences in size required them to do so. It probably did not hurt that the revenue from this one customer paid a huge chunck of their salaries.
After a while though, they realized that bad code could end up hurting all customers and not just their own job satisfaction, wich led them to revisit the issue once more. This time, somebody asked the key question: “What happens when our second largest customer outgrows a single shard, and which customer will that be?”
This question led them to the realization that they had to make it configurable for the operations team to decide which index or perhaps even indexes a tenant has data in. They ended up with a config scheme that looked something like this:
[ { "tenant-id-of-large-tenant":{ "write":"index1-largetenant", "read":[ "index*-largetenant" ] } }, { ".*":{ "write":"index-main", "read":[ "index-main" ] } } ]
Regarding Transitions and Moving Tenants Around
Elasticsearch has a nifty feature called index aliases. They’re great when migrating a tenant from an old to a new index. Not only do they support targeting multiple indexes, either as a list or with a pattern, but they allow you to define filters to be included when querying.
It is even possible to use them in such a way that your application might think there is only one index per tenant, when in reality there is one index alias per tenant. In the long run however, there will be a scaling issue with this as well. The reason is that the list of aliases is part of the cluster state that is replicated to all nodes and a large cluster state can threaten the stability of your cluster. Still, it is a simple solution that can take you pretty far.
Conclusion
In the long run, it’s a huge benefit for multitenant systems to have independence between shards and tenants. In order to scale well, some tenants need to share a shard and others need multiple shards. By designing your application so that tenants can be moved out to a different index you not only make it possible to add the hardware required for the largest tenants, but you also ensure that the existence of a large tenant does not increase the cost of adding a small tenant.
The final configuration scheme in this story may seem overengineered for many applications and you may feel that you would prefer your application not having to know about this. Mitigating the latter is best done by encapsulating it in the data access layer of your application. Regarding overengineering, you will probably not need to be this flexible in the beginning when your total data volume is low, but you should still have this situation in mind and consider how you can transition to such a scheme. I would also recommend that you once in a while check the size of your top 10 tenants and compare that to the average size of the other tenants. It is not uncommon with distributions where the top 10% of the largest tenants make up 90% of the total volume.
The trickiest part is probably the fact that you don’t know which tenants that will grow, requiring you to be able to make changes along the way. The exact way to do so, can be very different from project to project. How much downtime is acceptable for a tenant? Is Elasticsearch your primary storage? Is it acceptable to see duplicates in searches while transitioning or is it better that new documents don’t show up until the transition is complete? These and more are questions you will have to ask yourself.
I hope you enjoyed the read!