Storage and sizing guide
editStorage and sizing guide
editAPM Server handles a few different kinds of events. Since processing and storage costs are largely dominated by transactions and spans, only those will be covered here.
When the sampling rate is very small, transactions will be the dominate storage cost.
Typical transactions
editThe size of a transaction depends on the language, agent settings, and what services the agent instruments. For instance, an agent auto-instrumenting a service with a popular tech stack (web framework, database, caching library, etc.) is more likely to generate bigger transactions.
In addition, all agents support manual instrumentation. How little or much you use these APIs will also impact what a typical transaction looks like.
Due to the high variability of transactions, it’s difficult to classify a transaction as typical. Regardless, here’s a speculative reference:
| Transaction size | Number of Spans | Number of Frames |
|---|---|---|
Small |
5-10 |
5-10 |
Medium |
15-20 |
15-20 |
Large |
30-40 |
30-40 |
There will always be transaction outliers with hundreds of spans or frames, but those are very rare. Small transactions are the most common.
Typical storage
editConsider the following typical storage reference. These numbers do not account for Elasticsearch compression.
- 1 unsampled transaction is ~1 Kb
- 1 span with 10 frames is ~4 Kb
- 1 span with 50 frames is ~20 Kb
- 1 transaction with 10 spans, each with 10 frames is ~50 Kb
- 1 transaction with 25 spans, each with 25 spans is 250-300 Kb
- 100 transactions with 10 spans, each with 10 frames, sampled at 90% is 600 Kb
APM data compresses quite well, so the storage cost in Elasticsearch will be considerably less:
- Indexing 100 unsampled transactions per second for 1 hour results in 360,000 documents. These documents use around 50 Mb of disk space.
- Indexing 10 transactions per second for 1 hour, each transaction with 10 spans, each span with 10 frames, results in 396,000 documents. These documents use around 200 Mb of disk space.
- Indexing 25 transactions per second for 1 hour, each transaction with 25 spans, each span with 25 frames, results in 2,340,000 documents. These documents use around 1.2 Gb of disk space.
These examples were indexing the same data over and over with minimal variation. Because of that, the compression ratios observed of 80-90% are somewhat optimistic.