Configure tail-based sampling
editConfigure tail-based sampling
editEnable tail-based sampling with Enable tail-based sampling. When enabled, trace events are mapped to sampling policies. Each sampling policy must specify a sample rate, and can optionally specify other conditions. All of the policy conditions must be true for a trace event to match it.
Trace events are matched to policies in the order specified.
Each policy list must conclude with a default policy — one that only specifies a sample rate.
This default policy is used to catch remaining trace events that don’t match a stricter policy.
Requiring this default policy ensures that traces are only dropped intentionally.
If you enable tail-based sampling and send a transaction that does not match any of the policies,
APM Server will reject the transaction with the error no matching policy
.
Please note that from version 8.3.1
APM Server implements a default storage limit of 3GB,
but, due to how the limit is calculated and enforced the actual disk space may still grow slightly
over the limit.
Example configuration
editThis example defines three tail-based sampling polices:
- sample_rate: 1 service.environment: production trace.name: "GET /very_important_route" - sample_rate: .01 service.environment: production trace.name: "GET /not_important_route" - sample_rate: .1
Samples 100% of traces in |
|
Samples 1% of traces in |
|
Default policy to sample all remaining traces at 10%, e.g. traces in a different environment, like |
Configuration reference
editTop-level tail-based sampling settings:
Set to true
to enable tail based sampling.
Disabled by default. (bool)
APM Server binary |
|
Fleet-managed |
|
Synchronization interval for multiple APM Servers.
Should be in the order of tens of seconds or low minutes.
Default: 1m
(1 minute). (duration)
APM Server binary |
|
Fleet-managed |
|
Criteria used to match a root transaction to a sample rate.
Policies map trace events to a sample rate.
Each policy must specify a sample rate.
Trace events are matched to policies in the order specified.
All policy conditions must be true for a trace event to match.
Each policy list should conclude with a policy that only specifies a sample rate.
This final policy is used to catch remaining trace events that don’t match a stricter policy.
([]policy
)
APM Server binary |
|
Fleet-managed |
|
The amount of storage space allocated for trace events matching tail sampling policies. Caution: Setting this limit higher than the allowed space may cause APM Server to become unhealthy.
If the configured storage limit is insufficient, it logs "configured storage limit reached". The event will bypass sampling and will always be indexed when storage limit is reached.
Default: 3GB
. (text)
APM Server binary |
|
Fleet-managed |
|
Policy settings:
sample_rate
The sample rate to apply to trace events matching this policy. Required in each policy.
The sample rate must be greater than or equal to 0
and less than or equal to 1
.
For example, a sample_rate
of 0.01
means that 1% of trace events matching the policy will be sampled.
A sample_rate
of 1
means that 100% of trace events matching the policy will be sampled. (int)
trace.name
The trace name for events to match a policy.
A match occurs when the configured trace.name
matches the transaction.name
of the root transaction of a trace.
A root transaction is any transaction without a parent.id
. (string)
trace.outcome
The trace outcome for events to match a policy.
A match occurs when the configured trace.outcome
matches a trace’s event.outcome
field.
Trace outcome can be success
, failure
, or unknown
. (string)
service.name
The service name for events to match a policy. (string)
service.environment
The service environment for events to match a policy. (string)