Create a datafeed Added in 5.4.0
Datafeeds retrieve data from Elasticsearch for analysis by an anomaly detection job.
You can associate only one datafeed with each anomaly detection job.
The datafeed contains a query that runs at a defined interval (frequency
).
If you are concerned about delayed data, you can add a delay (query_delay
) at each interval.
When Elasticsearch security features are enabled, your datafeed remembers which roles the user who created it had
at the time of creation and runs the query using those same roles. If you provide secondary authorization headers,
those credentials are used instead.
You must use Kibana, this API, or the create anomaly detection jobs API to create a datafeed. Do not add a datafeed
directly to the .ml-config
index. Do not give users write
privileges on the .ml-config
index.
Path parameters
-
A numerical character string that uniquely identifies the datafeed. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters.
Query parameters
-
allow_no_indices boolean
If true, wildcard indices expressions that resolve into no concrete indices are ignored. This includes the
_all
string or when no indices are specified. -
expand_wildcards string | array[string]
Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values.
-
If true, concrete, expanded, or aliased indices are ignored when frozen.
Body Required
-
aggregations object
If set, the datafeed performs aggregation searches. Support for aggregations is limited and should be used only with low cardinality data.
-
chunking_config object
Additional properties are allowed.
-
delayed_data_check_config object
Additional properties are allowed.
-
frequency string
A duration. Units can be
nanos
,micros
,ms
(milliseconds),s
(seconds),m
(minutes),h
(hours) andd
(days). Also accepts "0" without a unit and "-1" to indicate an unspecified value. -
indices string | array[string]
-
indices_options object
Additional properties are allowed.
-
job_id string
-
max_empty_searches number
If a real-time datafeed has never seen any data (including during any initial training period), it automatically stops and closes the associated job after this many real-time searches return no documents. In other words, it stops after
frequency
timesmax_empty_searches
of real-time operation. If not set, a datafeed with no end time that sees no data remains started until it is explicitly stopped. By default, it is not set. -
query object
Additional properties are allowed.
-
query_delay string
A duration. Units can be
nanos
,micros
,ms
(milliseconds),s
(seconds),m
(minutes),h
(hours) andd
(days). Also accepts "0" without a unit and "-1" to indicate an unspecified value. -
runtime_mappings object
-
script_fields object
Specifies scripts that evaluate custom expressions and returns script fields to the datafeed. The detector configuration objects in a job can contain functions that use these script fields.
-
scroll_size number
The size parameter that is used in Elasticsearch searches when the datafeed does not use aggregations. The maximum value is the value of
index.max_result_window
, which is 10,000 by default. -
headers object
curl \
-X PUT http://api.example.com/_ml/datafeeds/{datafeed_id} \
-H "Content-Type: application/json" \
-d '{"aggregations":{},"chunking_config":{"mode":"auto","time_span":"string"},"delayed_data_check_config":{"check_window":"string","enabled":true},"frequency":"string","indices":"string","indices_options":{"allow_no_indices":true,"expand_wildcards":"string","ignore_unavailable":true,"ignore_throttled":true},"job_id":"string","max_empty_searches":42.0,"query":{},"query_delay":"string","runtime_mappings":{"additionalProperty1":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"},"additionalProperty2":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"}},"script_fields":{"additionalProperty1":{"script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"ignore_failure":true},"additionalProperty2":{"script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"ignore_failure":true}},"scroll_size":42.0,"headers":{}}'
{
"aggregations": {},
"chunking_config": {
"mode": "auto",
"time_span": "string"
},
"delayed_data_check_config": {
"check_window": "string",
"enabled": true
},
"frequency": "string",
"indices": "string",
"indices_options": {
"allow_no_indices": true,
"expand_wildcards": "string",
"ignore_unavailable": true,
"ignore_throttled": true
},
"job_id": "string",
"max_empty_searches": 42.0,
"query": {},
"query_delay": "string",
"runtime_mappings": {
"additionalProperty1": {
"fields": {
"additionalProperty1": {
"type": "boolean"
},
"additionalProperty2": {
"type": "boolean"
}
},
"fetch_fields": [
{
"field": "string",
"format": "string"
}
],
"format": "string",
"input_field": "string",
"target_field": "string",
"target_index": "string",
"script": {
"source": "string",
"id": "string",
"params": {
"additionalProperty1": {},
"additionalProperty2": {}
},
"": "painless",
"options": {
"additionalProperty1": "string",
"additionalProperty2": "string"
}
},
"type": "boolean"
},
"additionalProperty2": {
"fields": {
"additionalProperty1": {
"type": "boolean"
},
"additionalProperty2": {
"type": "boolean"
}
},
"fetch_fields": [
{
"field": "string",
"format": "string"
}
],
"format": "string",
"input_field": "string",
"target_field": "string",
"target_index": "string",
"script": {
"source": "string",
"id": "string",
"params": {
"additionalProperty1": {},
"additionalProperty2": {}
},
"": "painless",
"options": {
"additionalProperty1": "string",
"additionalProperty2": "string"
}
},
"type": "boolean"
}
},
"script_fields": {
"additionalProperty1": {
"script": {
"source": "string",
"id": "string",
"params": {
"additionalProperty1": {},
"additionalProperty2": {}
},
"": "painless",
"options": {
"additionalProperty1": "string",
"additionalProperty2": "string"
}
},
"ignore_failure": true
},
"additionalProperty2": {
"script": {
"source": "string",
"id": "string",
"params": {
"additionalProperty1": {},
"additionalProperty2": {}
},
"": "painless",
"options": {
"additionalProperty1": "string",
"additionalProperty2": "string"
}
},
"ignore_failure": true
}
},
"scroll_size": 42.0,
"headers": {}
}
{
"aggregations": {},
"authorization": {
"api_key": {
"id": "string",
"name": "string"
},
"roles": [
"string"
],
"service_account": "string"
},
"chunking_config": {
"mode": "auto",
"time_span": "string"
},
"delayed_data_check_config": {
"check_window": "string",
"enabled": true
},
"datafeed_id": "string",
"frequency": "string",
"indices": [
"string"
],
"job_id": "string",
"indices_options": {
"allow_no_indices": true,
"expand_wildcards": "string",
"ignore_unavailable": true,
"ignore_throttled": true
},
"max_empty_searches": 42.0,
"query": {},
"query_delay": "string",
"runtime_mappings": {
"additionalProperty1": {
"fields": {
"additionalProperty1": {
"type": "boolean"
},
"additionalProperty2": {
"type": "boolean"
}
},
"fetch_fields": [
{
"field": "string",
"format": "string"
}
],
"format": "string",
"input_field": "string",
"target_field": "string",
"target_index": "string",
"script": {
"source": "string",
"id": "string",
"params": {
"additionalProperty1": {},
"additionalProperty2": {}
},
"": "painless",
"options": {
"additionalProperty1": "string",
"additionalProperty2": "string"
}
},
"type": "boolean"
},
"additionalProperty2": {
"fields": {
"additionalProperty1": {
"type": "boolean"
},
"additionalProperty2": {
"type": "boolean"
}
},
"fetch_fields": [
{
"field": "string",
"format": "string"
}
],
"format": "string",
"input_field": "string",
"target_field": "string",
"target_index": "string",
"script": {
"source": "string",
"id": "string",
"params": {
"additionalProperty1": {},
"additionalProperty2": {}
},
"": "painless",
"options": {
"additionalProperty1": "string",
"additionalProperty2": "string"
}
},
"type": "boolean"
}
},
"script_fields": {
"additionalProperty1": {
"script": {
"source": "string",
"id": "string",
"params": {
"additionalProperty1": {},
"additionalProperty2": {}
},
"": "painless",
"options": {
"additionalProperty1": "string",
"additionalProperty2": "string"
}
},
"ignore_failure": true
},
"additionalProperty2": {
"script": {
"source": "string",
"id": "string",
"params": {
"additionalProperty1": {},
"additionalProperty2": {}
},
"": "painless",
"options": {
"additionalProperty1": "string",
"additionalProperty2": "string"
}
},
"ignore_failure": true
}
},
"scroll_size": 42.0
}