How can I customize the components of my deployment?

When you create or edit an existing deployment, you can fine-tune the capacity, add extensions, and select additional features.

Autoscaling reduces some of the manual effort required to manage a deployment by adjusting the capacity as demands on the deployment change. Currently, autoscaling is supported to scale Elasticsearch data tiers upwards, and to scale machine learning nodes both upwards and downwards. Check Deployment autoscaling to learn more.

Depending upon how much data you have and what queries you plan to run, you need to select a cluster size that fits your needs. There is no silver bullet for deciding how much memory you need other than simply testing it. The cluster performance metrics in the Elasticsearch Add-On for Heroku console can tell you if your cluster is sized appropriately. You can also enable deployment monitoring for more detailed performance metrics. Fortunately, you can change the amount of memory allocated to the cluster later without any downtime for HA deployments.

To change a cluster’s topology, from deployment management, select Edit deployment from the Actions dropdown. Next, select a storage and RAM setting from the Size per zone drop-down list, and save your changes. When downsizing the cluster, make sure to have enough resources to handle the current load, otherwise your cluster will be under stress.

Currently, half the memory is assigned to the JVM heap (a bit less when monitoring is activated). For example, on a 32 GB cluster, 16 GB are allotted to heap. The disk-to-RAM ratio currently is 1:24, meaning that you get 24 GB of storage space for each 1 GB of RAM. All clusters are backed by SSD drives.

The CPU resources assigned to a cluster are relative to the size of your cluster, meaning that a 32 GB cluster gets twice as much CPU resources as a 16 GB cluster. All clusters are guaranteed their share of CPU resources, as we do not overcommit resources. Smaller clusters up to and including 8 GB of RAM benefit from temporary CPU boosting to improve performance when needed most.

High availability is achieved by running a cluster with replicas in multiple data centers (availability zones), to prevent against downtime when infrastructure problems occur or when resizing or upgrading deployments. We offer the options of running in one, two, or three data centers.

Running in two data centers or availability zones is our default high availability configuration. It provides reasonably high protection against infrastructure failures and intermittent network problems. You might want three data centers if you need even higher fault tolerance. Just one zone might be sufficient, if the cluster is mainly used for testing or development.

Like many other changes, you change the level of fault tolerance while the cluster is running. For example, when you prepare a new cluster for production use, you can first run it in a single data center and then add another data center right before deploying to production.

While multiple data centers or availability zones increase a cluster’s fault tolerance, they do not protect against problematic searches that cause nodes to run out of memory. For a cluster to be highly reliable and available, it is also important to have enough memory.

The node capacity you choose is per data center. The reason for this is that there is no point in having two data centers if the failure of one will result in a cascading error because the remaining data center cannot handle the total load. Through the allocation awareness in Elasticsearch, we configure the nodes so that your Elasticsearch cluster will automatically allocate replicas between each availability zone.

You can get an at-a-glance status of all the shards in the deployment on the Elasticsearch page.

We recommend that you read Size your shards before you change the number of shards.