Create transform API
editCreate transform API
editInstantiates a transform.
Request
editPUT _transform/<transform_id>
Prerequisites
editIf the Elasticsearch security features are enabled, you must have the following built-in roles and privileges:
-
transform_admin
-
kibana_admin
(UI only) -
source index:
read
,view_index_metadata
-
destination index:
read
,create_index
,manage
andindex
-
cluster:
monitor
(UI only)
For more information, see Security privileges and Built-in roles.
Description
editThis API defines a transform, which copies data from source indices,
transforms it, and persists it into an entity-centric destination index. The
entities are defined by the set of group_by
fields in the pivot
object. You
can also think of the destination index as a two-dimensional tabular data
structure (known as a data frame). The ID for each document in the
data frame is generated from a hash of the entity, so there is a unique row
per entity. For more information, see Transforming data.
When the transform is created, a series of validations occur to
ensure its success. For example, there is a check for the existence of the
source indices and a check that the destination index is not part of the source
index pattern. You can use the defer_validation
parameter to skip these
checks.
Deferred validations are always run when the transform is started, with the exception of privilege checks. When Elasticsearch security features are enabled, the transform remembers which roles the user that created it had at the time of creation and uses those same roles. If those roles do not have the required privileges on the source and destination indices, the transform fails when it attempts unauthorized operations.
You must use Kibana or this API to create a transform.
Do not put a transform directly into any
.transform-internal*
indices using the Elasticsearch index API.
If Elasticsearch security features are enabled, do not give users any
privileges on .transform-internal*
indices. If you used transforms
prior 7.5, also do not give users any privileges on
.data-frame-internal*
indices.
Path parameters
edit-
<transform_id>
- (Required, string) Identifier for the transform. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters.
Query parameters
edit-
defer_validation
-
(Optional, boolean) When
true
, deferrable validations are not run. This behavior may be desired if the source index does not exist until after the transform is created.
Request body
edit-
description
- (Optional, string) Free text description of the transform.
-
dest
-
(Required, object) The destination for the transform.
Properties of
dest
-
index
- (Required, string) The destination index for the transform.
-
pipeline
- (Optional, string) The unique identifier for a pipeline.
-
-
frequency
-
(Optional, time units)
The interval between checks for changes in the source indices when the
transform is running continuously. Also determines the retry interval in the
event of transient failures while the transform is searching or indexing. The
minimum value is
1s
and the maximum is1h
. The default value is1m
.
-
pivot
-
(Required, object) The method for transforming the data. These objects define the pivot function
group by
fields and the aggregation to reduce the data.Properties of
pivot
-
aggregations
oraggs
-
(Required, object) Defines how to aggregate the grouped data. The following composite aggregations are supported:
Transforms support a subset of the functionality in composite aggregations. See Limitations.
-
group_by
-
(Required, object) Defines how to group the data. More than one grouping can be defined per pivot. The following groupings are supported:
-
max_page_search_size
-
(Optional, integer)
Defines the initial page size to use for the composite aggregation for each
checkpoint. If circuit breaker exceptions occur, the page size is dynamically
adjusted to a lower value. The minimum value is
10
and the maximum is10,000
. The default value is500
.
-
-
source
-
(Required, object) The source of the data for the transform.
Properties of
source
-
index
-
(Required, string or array)
The source indices for the transform. It can be a single index, an index
pattern (for example,
"myindex*"
), an array of indices (for example,["index1", "index2"]
), or an array of index patterns (for example,["myindex1-*", "myindex2-*"]
. -
query
- (Optional, object) A query clause that retrieves a subset of data from the source index. See Query DSL.
-
-
sync
-
(Optional, object) Defines the properties transforms require to run continuously.
Properties of
sync
-
time
-
(Required, object) Specifies that the transform uses a time field to synchronize the source and destination indices.
Properties of
time
-
delay
-
(Optional, time units)
The time delay between the current time and the latest input data time. The
default value is
60s
. -
field
-
(Required, string) The date field that is used to identify new documents in the source.
In general, it’s a good idea to use a field that contains the ingest timestamp. If you use a different field, you might need to set the
delay
such that it accounts for data transmission delays.
-
-
Examples
editPUT _transform/ecommerce_transform { "source": { "index": "kibana_sample_data_ecommerce", "query": { "term": { "geoip.continent_name": { "value": "Asia" } } } }, "pivot": { "group_by": { "customer_id": { "terms": { "field": "customer_id" } } }, "aggregations": { "max_price": { "max": { "field": "taxful_total_price" } } } }, "description": "Maximum priced ecommerce data by customer_id in Asia", "dest": { "index": "kibana_sample_data_ecommerce_transform", "pipeline": "add_timestamp_pipeline" }, "frequency": "5m", "sync": { "time": { "field": "order_date", "delay": "60s" } } }
When the transform is created, you receive the following results:
{ "acknowledged" : true }