This documentation contains work-in-progress information for future Elastic Stack and Cloud releases. Use the version selector to view supported release docs. It also contains some Elastic Cloud serverless information. Check out our serverless docs for more details.
« Get transform statistics API Reset transform API »
Elastic Docs ›Elasticsearch Guide [master] ›REST APIs ›Transform APIs

Preview transform API

edit

Preview transform API

edit

Previews a transform.

Request

edit

GET _transform/<transform_id>/_preview

POST _transform/<transform_id>/_preview

GET _transform/_preview

POST _transform/_preview

Prerequisites

edit

Requires the following privileges:

  • cluster: manage_transform (the transform_admin built-in role grants this privilege)
  • source indices: read, view_index_metadata.

Description

edit

This API generates a preview of the results that you will get when you run the create transforms API with the same configuration. It returns a maximum of 100 results. The calculations are based on all the current data in the source index.

It also generates a list of mappings and settings for the destination index. If the destination index does not exist when you start a transform, these are the mappings and settings that are used. These values are determined based on the field types of the source index and the transform aggregations.

There are some limitations that might result in poor mappings. As a work-around, create the destination index or an index template with your preferred mappings before you start the transform.

You must choose either the latest or pivot method for your transform; you cannot use both in a single transform.

When you preview a transform, it uses the credentials of the user calling the API. When you start a transform, it uses the roles of the last user to create or update it. If the two sets of roles differ, the preview may not accurately reflect the behavior of the transform. To avoid such problems, the same user that creates or updates the transform should preview it to ensure it is returning the expected data. Alternatively, use secondary authorization headers to supply the credentials

Path parameters

edit
<transform_id>

(Optional, string) Id of the transform to preview.

If you provide the <transform_id> as a path parameter, you cannot provide transform configuration details in the request body.

Query parameters

edit
timeout
(Optional, time) Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error. Defaults to 30s.

Request body

edit
description
(Optional, string) Free text description of the transform.
dest

(Optional, object) The destination for the transform.

Properties of dest
index
(Optional, string) The destination index for the transform.

In the case of a pivot transform, the mappings of the destination index are deduced based on the source fields when possible. If alternate mappings are required, use the Create index API prior to starting the transform.

In the case of a latest transform, the mappings are never deduced. If dynamic mappings for the destination index are undesirable, use the Create index API prior to starting the transform.

pipeline
(Optional, string) The unique identifier for an ingest pipeline.
frequency
(Optional, time units) The interval between checks for changes in the source indices when the transform is running continuously. The minimum value is 1s and the maximum is 1h. The default value is 1m.
latest

(Required*, object) The latest method transforms the data by finding the latest document for each unique key.

Properties of latest
sort
(Required, string) Specifies the date field that is used to identify the latest documents.
unique_key
(Required, array of strings) Specifies an array of one or more fields that are used to group the data.
pivot

(Required, object) The pivot method transforms the data by aggregating and grouping it. These objects define the group by fields and the aggregation to reduce the data.

Properties of pivot
aggregations or aggs

(Required, object) Defines how to aggregate the grouped data. The following aggregations are currently supported:

group_by

(Required, object) Defines how to group the data. More than one grouping can be defined per pivot. The following groupings are currently supported:

The grouping properties can optionally have a missing_bucket property. If it’s true, documents without a value in the respective group_by field are included. Defaults to false.

retention_policy

(Optional, object) Defines a retention policy for the transform. Data that meets the defined criteria is deleted from the destination index.

Properties of retention_policy
time

(Required, object) Specifies that the transform uses a time field to set the retention policy. Data is deleted if time.field for the retention policy exists and contains data older than max.age.

Properties of time
field
(Required, string) The date field that is used to calculate the age of the document. Set time.field to an existing date field.
max_age
(Required, time units) Specifies the maximum age of a document in the destination index. Documents that are older than the configured value are removed from the destination index.
source

(Required, object) The source of the data for the transform.

Properties of source
index

(Required, string or array) The source indices for the transform. It can be a single index, an index pattern (for example, "my-index-*"), an array of indices (for example, ["my-index-000001", "my-index-000002"]), or an array of index patterns (for example, ["my-index-*", "my-other-index-*"]. For remote indices use the syntax "remote_name:index_name".

If any indices are in remote clusters then the master node and at least one transform node must have the remote_cluster_client node role.

query
(Optional, object) A query clause that retrieves a subset of data from the source index. See Query DSL.
runtime_mappings
(Optional, object) Definitions of search-time runtime fields that can be used by the transform. For search runtime fields all data nodes, including remote nodes, must be 7.12 or later.
sync

(Optional, object) Defines the properties transforms require to run continuously.

Properties of sync
time

(Optional, object) Specifies that the transform uses a time field to synchronize the source and destination indices.

Properties of time
delay
(Optional, time units) The time delay between the current time and the latest input data time. The default value is 60s.
field
(Optional, string) The date field that is used to identify new documents in the source.
settings

(Optional, object) Defines optional transform settings.

Properties of settings
align_checkpoints
(Optional, boolean) Specifies whether the transform checkpoint ranges should be optimized for performance. Such optimization can align checkpoint ranges with date histogram interval when date histogram is specified as a group source in the transform config. As an effect, less document updates in the destination index will be performed thus improving overall performance. The default value is true, which means the checkpoint ranges will be optimized if possible.
dates_as_epoch_millis
(Optional, boolean) Defines if dates in the output should be written as ISO formatted string (default) or as millis since epoch. epoch_millis has been the default for transforms created before version 7.11. For compatible output set this to true. The default value is false.
deduce_mappings
(Optional, boolean) Specifies whether the transform should deduce the destination index mappings from the transform config. The default value is true, which means the destination index mappings will be deduced if possible.
docs_per_second
(Optional, float) Specifies a limit on the number of input documents per second. This setting throttles the transform by adding a wait time between search requests. The default value is null, which disables throttling.
max_page_search_size
(Optional, integer) Defines the initial page size to use for the composite aggregation for each checkpoint. If circuit breaker exceptions occur, the page size is dynamically adjusted to a lower value. The minimum value is 10 and the maximum is 65,536. The default value is 500.
unattended
(Optional, boolean) If true, the transform runs in unattended mode. In unattended mode, the transform retries indefinitely in case of an error which means the transform never fails. Setting the number of retries other than infinite fails in validation. Defaults to false.

Response body

edit
generated_dest_index

(object) Contains details about the destination index.

Properties of generated_dest_index
aliases
(object) The aliases for the destination index.
mappings
(object) The mappings for each document in the destination index.
settings
(object) The index settings for the destination index.
preview
(array) An array of documents. In particular, they are the JSON representation of the documents that would be created in the destination index by the transform.

Examples

edit
resp = client.transform.preview_transform(
    source={
        "index": "kibana_sample_data_ecommerce"
    },
    pivot={
        "group_by": {
            "customer_id": {
                "terms": {
                    "field": "customer_id",
                    "missing_bucket": True
                }
            }
        },
        "aggregations": {
            "max_price": {
                "max": {
                    "field": "taxful_total_price"
                }
            }
        }
    },
)
print(resp)
const response = await client.transform.previewTransform({
  source: {
    index: "kibana_sample_data_ecommerce",
  },
  pivot: {
    group_by: {
      customer_id: {
        terms: {
          field: "customer_id",
          missing_bucket: true,
        },
      },
    },
    aggregations: {
      max_price: {
        max: {
          field: "taxful_total_price",
        },
      },
    },
  },
});
console.log(response);
POST _transform/_preview
{
  "source": {
    "index": "kibana_sample_data_ecommerce"
  },
  "pivot": {
    "group_by": {
      "customer_id": {
        "terms": {
          "field": "customer_id",
          "missing_bucket": true
        }
      }
    },
    "aggregations": {
      "max_price": {
        "max": {
          "field": "taxful_total_price"
        }
      }
    }
  }
}

The data that is returned for this example is as follows:

{
  "preview" : [
    {
      "max_price" : 171.0,
      "customer_id" : "10"
    },
    {
      "max_price" : 233.0,
      "customer_id" : "11"
    },
    {
      "max_price" : 200.0,
      "customer_id" : "12"
    }
    ...
  ],
  "generated_dest_index" : {
    "mappings" : {
      "_meta" : {
        "_transform" : {
          "transform" : "transform-preview",
          "version" : {
            "created" : "7.7.0"
          },
          "creation_date_in_millis" : 1584738236757
        },
        "created_by" : "transform"
      },
      "properties" : {
        "max_price" : {
          "type" : "half_float"
        },
        "customer_id" : {
          "type" : "keyword"
        }
      }
    },
    "settings" : {
      "index" : {
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-1"
      }
    },
    "aliases" : { }
  }
}
« Get transform statistics API Reset transform API »