Compact and aligned text (CAT)

The compact and aligned text (CAT) APIs aim are intended only for human consumption using the Kibana console or command line. They are not intended for use by applications. For application consumption, it's recommend to use a corresponding JSON API. All the cat commands accept a query string parameter help to see all the headers and info they provide, and the /_cat command alone lists all the available commands.

































































Get data frame analytics jobs Added in 7.7.0

GET /_cat/ml/data_frame/analytics/{id}

Get configuration and usage information about data frame analytics jobs.

IMPORTANT: CAT APIs are only intended for human consumption using the Kibana console or command line. They are not intended for use by applications. For application consumption, use the get data frame analytics jobs statistics API.

Path parameters

  • id string Required

    The ID of the data frame analytics to fetch

Query parameters

  • Whether to ignore if a wildcard expression matches no configs. (This includes _all string or when no configs have been specified)

  • bytes string

    The unit in which to display byte values

    Values are b, kb, mb, gb, tb, or pb.

  • h string | array[string]

    Comma-separated list of column names to display.

  • s string | array[string]

    Comma-separated list of column names or column aliases used to sort the response.

  • time string

    Unit used to display time values.

    Values are nanos, micros, ms, s, m, h, or d.

Responses

GET /_cat/ml/data_frame/analytics/{id}
curl \
 -X GET http://api.example.com/_cat/ml/data_frame/analytics/{id}
Response examples (200)
[
  {
    "id": "string",
    "type": "string",
    "create_time": "string",
    "version": "string",
    "source_index": "string",
    "dest_index": "string",
    "description": "string",
    "model_memory_limit": "string",
    "state": "string",
    "failure_reason": "string",
    "progress": "string",
    "assignment_explanation": "string",
    "node.id": "string",
    "node.name": "string",
    "node.ephemeral_id": "string",
    "node.address": "string"
  }
]
















Get trained models Added in 7.7.0

GET /_cat/ml/trained_models

Get configuration and usage information about inference trained models.

IMPORTANT: CAT APIs are only intended for human consumption using the Kibana console or command line. They are not intended for use by applications. For application consumption, use the get trained models statistics API.

Query parameters

  • Specifies what to do when the request: contains wildcard expressions and there are no models that match; contains the _all string or no identifiers and there are no matches; contains wildcard expressions and there are only partial matches. If true, the API returns an empty array when there are no matches and the subset of results when there are partial matches. If false, the API returns a 404 status code when there are no matches or only partial matches.

  • bytes string

    The unit used to display byte values.

    Values are b, kb, mb, gb, tb, or pb.

  • h string | array[string]

    A comma-separated list of column names to display.

  • s string | array[string]

    A comma-separated list of column names or aliases used to sort the response.

  • from number

    Skips the specified number of transforms.

  • size number

    The maximum number of transforms to display.

  • time string

    Unit used to display time values.

    Values are nanos, micros, ms, s, m, h, or d.

Responses

GET /_cat/ml/trained_models
curl \
 -X GET http://api.example.com/_cat/ml/trained_models
Response examples (200)
[
  {
    "id": "string",
    "created_by": "string",
    "": "string",
    "operations": "string",
    "license": "string",
    "version": "string",
    "description": "string",
    "ingest.pipelines": "string",
    "ingest.count": "string",
    "ingest.time": "string",
    "ingest.current": "string",
    "ingest.failed": "string",
    "data_frame.id": "string",
    "data_frame.create_time": "string",
    "data_frame.source_index": "string",
    "data_frame.analysis": "string",
    "type": "string"
  }
]




Get node attribute information

GET /_cat/nodeattrs

Get information about custom node attributes. IMPORTANT: cat APIs are only intended for human consumption using the command line or Kibana console. They are not intended for use by applications. For application consumption, use the nodes info API.

Query parameters

  • local boolean

    If true, the request computes the list of selected nodes from the local cluster state. If false the list of selected nodes are computed from the cluster state of the master node. In both cases the coordinating node will send requests for further information to each selected node.

  • Period to wait for a connection to the master node.

Responses

GET /_cat/nodeattrs
curl \
 -X GET http://api.example.com/_cat/nodeattrs
Response examples (200)
[
  {
    "node": "string",
    "id": "string",
    "pid": "string",
    "host": "string",
    "ip": "string",
    "port": "string",
    "attr": "string",
    "value": "string"
  }
]

Get node information

GET /_cat/nodes

Get information about the nodes in a cluster. IMPORTANT: cat APIs are only intended for human consumption using the command line or Kibana console. They are not intended for use by applications. For application consumption, use the nodes info API.

Query parameters

  • bytes string

    The unit used to display byte values.

    Values are b, kb, mb, gb, tb, or pb.

  • full_id boolean | string

    If true, return the full node ID. If false, return the shortened node ID.

  • If true, the response includes information from segments that are not loaded into memory.

  • Period to wait for a connection to the master node.

  • time string

    Unit used to display time values.

    Values are nanos, micros, ms, s, m, h, or d.

Responses

GET /_cat/nodes
curl \
 -X GET http://api.example.com/_cat/nodes
Response examples (200)
[
  {
    "id": "string",
    "pid": "string",
    "ip": "string",
    "port": "string",
    "http_address": "string",
    "version": "string",
    "flavor": "string",
    "type": "string",
    "build": "string",
    "jdk": "string",
    "": "string",
    "heap.current": "string",
    "heap.max": "string",
    "ram.current": "string",
    "ram.max": "string",
    "file_desc.current": "string",
    "file_desc.max": "string",
    "cpu": "string",
    "load_1m": "string",
    "load_5m": "string",
    "load_15m": "string",
    "uptime": "string",
    "node.role": "string",
    "master": "string",
    "name": "string",
    "completion.size": "string",
    "fielddata.memory_size": "string",
    "fielddata.evictions": "string",
    "query_cache.memory_size": "string",
    "query_cache.evictions": "string",
    "query_cache.hit_count": "string",
    "query_cache.miss_count": "string",
    "request_cache.memory_size": "string",
    "request_cache.evictions": "string",
    "request_cache.hit_count": "string",
    "request_cache.miss_count": "string",
    "flush.total": "string",
    "flush.total_time": "string",
    "get.current": "string",
    "get.time": "string",
    "get.total": "string",
    "get.exists_time": "string",
    "get.exists_total": "string",
    "get.missing_time": "string",
    "get.missing_total": "string",
    "indexing.delete_current": "string",
    "indexing.delete_time": "string",
    "indexing.delete_total": "string",
    "indexing.index_current": "string",
    "indexing.index_time": "string",
    "indexing.index_total": "string",
    "indexing.index_failed": "string",
    "merges.current": "string",
    "merges.current_docs": "string",
    "merges.current_size": "string",
    "merges.total": "string",
    "merges.total_docs": "string",
    "merges.total_size": "string",
    "merges.total_time": "string",
    "refresh.total": "string",
    "refresh.time": "string",
    "refresh.external_total": "string",
    "refresh.external_time": "string",
    "refresh.listeners": "string",
    "script.compilations": "string",
    "script.cache_evictions": "string",
    "script.compilation_limit_triggered": "string",
    "search.fetch_current": "string",
    "search.fetch_time": "string",
    "search.fetch_total": "string",
    "search.open_contexts": "string",
    "search.query_current": "string",
    "search.query_time": "string",
    "search.query_total": "string",
    "search.scroll_current": "string",
    "search.scroll_time": "string",
    "search.scroll_total": "string",
    "segments.count": "string",
    "segments.memory": "string",
    "segments.index_writer_memory": "string",
    "segments.version_map_memory": "string",
    "segments.fixed_bitset_memory": "string",
    "suggest.current": "string",
    "suggest.time": "string",
    "suggest.total": "string",
    "bulk.total_operations": "string",
    "bulk.total_time": "string",
    "bulk.total_size_in_bytes": "string",
    "bulk.avg_time": "string",
    "bulk.avg_size_in_bytes": "string"
  }
]
















Get snapshot repository information Added in 2.1.0

GET /_cat/repositories

Get a list of snapshot repositories for a cluster. IMPORTANT: cat APIs are only intended for human consumption using the command line or Kibana console. They are not intended for use by applications. For application consumption, use the get snapshot repository API.

Query parameters

  • local boolean

    If true, the request computes the list of selected nodes from the local cluster state. If false the list of selected nodes are computed from the cluster state of the master node. In both cases the coordinating node will send requests for further information to each selected node.

  • Period to wait for a connection to the master node.

Responses

GET /_cat/repositories
curl \
 -X GET http://api.example.com/_cat/repositories
Response examples (200)
[
  {
    "id": "string",
    "type": "string"
  }
]

















































































































Get cluster statistics Added in 1.3.0

GET /_cluster/stats

Get basic index metrics (shard numbers, store size, memory usage) and information about the current nodes that form the cluster (number, roles, os, jvm versions, memory usage, cpu and installed plugins).

Query parameters

  • Include remote cluster data into the response

  • timeout string

    Period to wait for each node to respond. If a node does not respond before its timeout expires, the response does not include its stats. However, timed out nodes are included in the response’s _nodes.failed property. Defaults to no timeout.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • _nodes object

      Additional properties are allowed.

      Hide _nodes attributes Show _nodes attributes object
      • failures array[object]
        Hide failures attributes Show failures attributes object
        • type string Required

          The type of error

        • reason string

          A human-readable explanation of the error, in English.

        • The server stack trace. Present only if the error_trace=true parameter was sent with the request.

        • Additional properties are allowed.

        • root_cause array[object]

          Additional properties are allowed.

        • suppressed array[object]

          Additional properties are allowed.

      • total number Required

        Total number of nodes selected by the request.

      • successful number Required

        Number of nodes that responded successfully to the request.

      • failed number Required

        Number of nodes that rejected the request or failed to respond. If this value is not 0, a reason for the rejection or failure is included in the response.

    • cluster_name string Required
    • cluster_uuid string Required
    • indices object Required

      Additional properties are allowed.

      Hide indices attributes Show indices attributes object
    • nodes object Required

      Additional properties are allowed.

      Hide nodes attributes Show nodes attributes object
      • count object Required

        Additional properties are allowed.

        Hide count attributes Show count attributes object
      • discovery_types object Required

        Contains statistics about the discovery types used by selected nodes.

        Hide discovery_types attribute Show discovery_types attribute object
        • * number Additional properties
      • fs object Required

        Additional properties are allowed.

        Hide fs attributes Show fs attributes object
        • available_in_bytes number Required

          Total number of bytes available to JVM in file stores across all selected nodes. Depending on operating system or process-level restrictions, this number may be less than nodes.fs.free_in_byes. This is the actual amount of free disk space the selected Elasticsearch nodes can use.

        • free_in_bytes number Required

          Total number of unallocated bytes in file stores across all selected nodes.

        • total_in_bytes number Required

          Total size, in bytes, of all file stores across all selected nodes.

      • indexing_pressure object Required

        Additional properties are allowed.

        Hide indexing_pressure attribute Show indexing_pressure attribute object
      • ingest object Required

        Additional properties are allowed.

        Hide ingest attributes Show ingest attributes object
        • number_of_pipelines number Required
        • processor_stats object Required
          Hide processor_stats attribute Show processor_stats attribute object
          • * object Additional properties

            Additional properties are allowed.

            Hide * attributes Show * attributes object
            • count number Required
            • current number Required
            • failed number Required
            • time string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • jvm object Required

        Additional properties are allowed.

        Hide jvm attributes Show jvm attributes object
        • Time unit for milliseconds

        • mem object Required

          Additional properties are allowed.

          Hide mem attributes Show mem attributes object
          • heap_max_in_bytes number Required

            Maximum amount of memory, in bytes, available for use by the heap across all selected nodes.

          • heap_used_in_bytes number Required

            Memory, in bytes, currently in use by the heap across all selected nodes.

        • threads number Required

          Number of active threads in use by JVM across all selected nodes.

        • versions array[object] Required

          Contains statistics about the JVM versions used by selected nodes.

          Hide versions attributes Show versions attributes object
      • network_types object Required

        Additional properties are allowed.

        Hide network_types attributes Show network_types attributes object
        • http_types object Required

          Contains statistics about the HTTP network types used by selected nodes.

          Hide http_types attribute Show http_types attribute object
          • * number Additional properties
        • transport_types object Required

          Contains statistics about the transport network types used by selected nodes.

          Hide transport_types attribute Show transport_types attribute object
          • * number Additional properties
      • os object Required

        Additional properties are allowed.

        Hide os attributes Show os attributes object
        • allocated_processors number Required

          Number of processors used to calculate thread pool size across all selected nodes. This number can be set with the processors setting of a node and defaults to the number of processors reported by the operating system. In both cases, this number will never be larger than 32.

        • architectures array[object]

          Contains statistics about processor architectures (for example, x86_64 or aarch64) used by selected nodes.

          Hide architectures attributes Show architectures attributes object
          • arch string Required

            Name of an architecture used by one or more selected nodes.

          • count number Required

            Number of selected nodes using the architecture.

        • available_processors number Required

          Number of processors available to JVM across all selected nodes.

        • mem object Required

          Additional properties are allowed.

          Hide mem attributes Show mem attributes object
          • Total amount, in bytes, of memory across all selected nodes, but using the value specified using the es.total_memory_bytes system property instead of measured total memory for those nodes where that system property was set.

          • free_in_bytes number Required

            Amount, in bytes, of free physical memory across all selected nodes.

          • free_percent number Required

            Percentage of free physical memory across all selected nodes.

          • total_in_bytes number Required

            Total amount, in bytes, of physical memory across all selected nodes.

          • used_in_bytes number Required

            Amount, in bytes, of physical memory in use across all selected nodes.

          • used_percent number Required

            Percentage of physical memory in use across all selected nodes.

        • names array[object] Required

          Contains statistics about operating systems used by selected nodes.

          Hide names attributes Show names attributes object
          • count number Required

            Number of selected nodes using the operating system.

          • name string Required
        • pretty_names array[object] Required

          Contains statistics about operating systems used by selected nodes.

          Hide pretty_names attributes Show pretty_names attributes object
          • count number Required

            Number of selected nodes using the operating system.

          • pretty_name string Required
      • packaging_types array[object] Required

        Contains statistics about Elasticsearch distributions installed on selected nodes.

        Hide packaging_types attributes Show packaging_types attributes object
        • count number Required

          Number of selected nodes using the distribution flavor and file type.

        • flavor string Required

          Type of Elasticsearch distribution. This is always default.

        • type string Required

          File type (such as tar or zip) used for the distribution package.

      • plugins array[object] Required

        Contains statistics about installed plugins and modules by selected nodes. If no plugins or modules are installed, this array is empty.

        Hide plugins attributes Show plugins attributes object
      • process object Required

        Additional properties are allowed.

        Hide process attributes Show process attributes object
        • cpu object Required

          Additional properties are allowed.

          Hide cpu attribute Show cpu attribute object
          • percent number Required

            Percentage of CPU used across all selected nodes. Returns -1 if not supported.

        • open_file_descriptors object Required

          Additional properties are allowed.

          Hide open_file_descriptors attributes Show open_file_descriptors attributes object
          • avg number Required

            Average number of concurrently open file descriptors. Returns -1 if not supported.

          • max number Required

            Maximum number of concurrently open file descriptors allowed across all selected nodes. Returns -1 if not supported.

          • min number Required

            Minimum number of concurrently open file descriptors across all selected nodes. Returns -1 if not supported.

      • versions array[string] Required

        Array of Elasticsearch versions used on selected nodes.

    • status string Required

      Values are green, GREEN, yellow, YELLOW, red, or RED.

    • timestamp number Required

      Unix timestamp, in milliseconds, for the last time the cluster statistics were refreshed.

GET /_cluster/stats
curl \
 -X GET http://api.example.com/_cluster/stats
Response examples (200)
{
  "_nodes": {
    "failures": [
      {
        "type": "string",
        "reason": "string",
        "stack_trace": "string",
        "caused_by": {},
        "root_cause": [
          {}
        ],
        "suppressed": [
          {}
        ]
      }
    ],
    "total": 42.0,
    "successful": 42.0,
    "failed": 42.0
  },
  "cluster_name": "string",
  "cluster_uuid": "string",
  "indices": {
    "analysis": {
      "analyzer_types": [
        {
          "name": "string",
          "count": 42.0,
          "index_count": 42.0,
          "indexed_vector_count": 42.0,
          "indexed_vector_dim_max": 42.0,
          "indexed_vector_dim_min": 42.0,
          "script_count": 42.0
        }
      ],
      "built_in_analyzers": [
        {
          "name": "string",
          "count": 42.0,
          "index_count": 42.0,
          "indexed_vector_count": 42.0,
          "indexed_vector_dim_max": 42.0,
          "indexed_vector_dim_min": 42.0,
          "script_count": 42.0
        }
      ],
      "built_in_char_filters": [
        {
          "name": "string",
          "count": 42.0,
          "index_count": 42.0,
          "indexed_vector_count": 42.0,
          "indexed_vector_dim_max": 42.0,
          "indexed_vector_dim_min": 42.0,
          "script_count": 42.0
        }
      ],
      "built_in_filters": [
        {
          "name": "string",
          "count": 42.0,
          "index_count": 42.0,
          "indexed_vector_count": 42.0,
          "indexed_vector_dim_max": 42.0,
          "indexed_vector_dim_min": 42.0,
          "script_count": 42.0
        }
      ],
      "built_in_tokenizers": [
        {
          "name": "string",
          "count": 42.0,
          "index_count": 42.0,
          "indexed_vector_count": 42.0,
          "indexed_vector_dim_max": 42.0,
          "indexed_vector_dim_min": 42.0,
          "script_count": 42.0
        }
      ],
      "char_filter_types": [
        {
          "name": "string",
          "count": 42.0,
          "index_count": 42.0,
          "indexed_vector_count": 42.0,
          "indexed_vector_dim_max": 42.0,
          "indexed_vector_dim_min": 42.0,
          "script_count": 42.0
        }
      ],
      "filter_types": [
        {
          "name": "string",
          "count": 42.0,
          "index_count": 42.0,
          "indexed_vector_count": 42.0,
          "indexed_vector_dim_max": 42.0,
          "indexed_vector_dim_min": 42.0,
          "script_count": 42.0
        }
      ],
      "tokenizer_types": [
        {
          "name": "string",
          "count": 42.0,
          "index_count": 42.0,
          "indexed_vector_count": 42.0,
          "indexed_vector_dim_max": 42.0,
          "indexed_vector_dim_min": 42.0,
          "script_count": 42.0
        }
      ]
    },
    "completion": {
      "size_in_bytes": 42.0,
      "": 42.0,
      "fields": {
        "additionalProperty1": {
          "size_in_bytes": 42.0
        },
        "additionalProperty2": {
          "size_in_bytes": 42.0
        }
      }
    },
    "count": 42.0,
    "docs": {
      "count": 42.0,
      "deleted": 42.0
    },
    "fielddata": {
      "evictions": 42.0,
      "": 42.0,
      "memory_size_in_bytes": 42.0,
      "fields": {
        "additionalProperty1": {
          "memory_size_in_bytes": 42.0
        },
        "additionalProperty2": {
          "memory_size_in_bytes": 42.0
        }
      }
    },
    "query_cache": {
      "cache_count": 42.0,
      "cache_size": 42.0,
      "evictions": 42.0,
      "hit_count": 42.0,
      "": 42.0,
      "memory_size_in_bytes": 42.0,
      "miss_count": 42.0,
      "total_count": 42.0
    },
    "segments": {
      "count": 42.0,
      "": 42.0,
      "doc_values_memory_in_bytes": 42.0,
      "file_sizes": {
        "additionalProperty1": {
          "description": "string",
          "size_in_bytes": 42.0,
          "min_size_in_bytes": 42.0,
          "max_size_in_bytes": 42.0,
          "average_size_in_bytes": 42.0,
          "count": 42.0
        },
        "additionalProperty2": {
          "description": "string",
          "size_in_bytes": 42.0,
          "min_size_in_bytes": 42.0,
          "max_size_in_bytes": 42.0,
          "average_size_in_bytes": 42.0,
          "count": 42.0
        }
      },
      "fixed_bit_set_memory_in_bytes": 42.0,
      "index_writer_max_memory_in_bytes": 42.0,
      "index_writer_memory_in_bytes": 42.0,
      "max_unsafe_auto_id_timestamp": 42.0,
      "memory_in_bytes": 42.0,
      "norms_memory_in_bytes": 42.0,
      "points_memory_in_bytes": 42.0,
      "stored_fields_memory_in_bytes": 42.0,
      "terms_memory_in_bytes": 42.0,
      "term_vectors_memory_in_bytes": 42.0,
      "version_map_memory_in_bytes": 42.0
    },
    "shards": {
      "index": {
        "primaries": {
          "avg": 42.0,
          "max": 42.0,
          "min": 42.0
        },
        "replication": {
          "avg": 42.0,
          "max": 42.0,
          "min": 42.0
        },
        "shards": {
          "avg": 42.0,
          "max": 42.0,
          "min": 42.0
        }
      },
      "primaries": 42.0,
      "replication": 42.0,
      "total": 42.0
    },
    "store": {
      "": 42.0,
      "size_in_bytes": 42.0,
      "reserved_in_bytes": 42.0,
      "total_data_set_size_in_bytes": 42.0
    },
    "mappings": {
      "field_types": [
        {
          "name": "string",
          "count": 42.0,
          "index_count": 42.0,
          "indexed_vector_count": 42.0,
          "indexed_vector_dim_max": 42.0,
          "indexed_vector_dim_min": 42.0,
          "script_count": 42.0
        }
      ],
      "runtime_field_types": [
        {
          "chars_max": 42.0,
          "chars_total": 42.0,
          "count": 42.0,
          "doc_max": 42.0,
          "doc_total": 42.0,
          "index_count": 42.0,
          "lang": [
            "string"
          ],
          "lines_max": 42.0,
          "lines_total": 42.0,
          "name": "string",
          "scriptless_count": 42.0,
          "shadowed_count": 42.0,
          "source_max": 42.0,
          "source_total": 42.0
        }
      ],
      "total_field_count": 42.0,
      "total_deduplicated_field_count": 42.0,
      "": 42.0,
      "total_deduplicated_mapping_size_in_bytes": 42.0
    },
    "versions": [
      {
        "index_count": 42.0,
        "primary_shard_count": 42.0,
        "total_primary_bytes": 42.0,
        "version": "string"
      }
    ]
  },
  "nodes": {
    "count": {
      "coordinating_only": 42.0,
      "data": 42.0,
      "data_cold": 42.0,
      "data_content": 42.0,
      "data_frozen": 42.0,
      "data_hot": 42.0,
      "data_warm": 42.0,
      "ingest": 42.0,
      "master": 42.0,
      "ml": 42.0,
      "remote_cluster_client": 42.0,
      "total": 42.0,
      "transform": 42.0,
      "voting_only": 42.0
    },
    "discovery_types": {
      "additionalProperty1": 42.0,
      "additionalProperty2": 42.0
    },
    "fs": {
      "available_in_bytes": 42.0,
      "free_in_bytes": 42.0,
      "total_in_bytes": 42.0
    },
    "indexing_pressure": {
      "memory": {
        "current": {
          "all_in_bytes": 42.0,
          "combined_coordinating_and_primary_in_bytes": 42.0,
          "coordinating_in_bytes": 42.0,
          "coordinating_rejections": 42.0,
          "primary_in_bytes": 42.0,
          "primary_rejections": 42.0,
          "replica_in_bytes": 42.0,
          "replica_rejections": 42.0
        },
        "limit_in_bytes": 42.0,
        "total": {
          "all_in_bytes": 42.0,
          "combined_coordinating_and_primary_in_bytes": 42.0,
          "coordinating_in_bytes": 42.0,
          "coordinating_rejections": 42.0,
          "primary_in_bytes": 42.0,
          "primary_rejections": 42.0,
          "replica_in_bytes": 42.0,
          "replica_rejections": 42.0
        }
      }
    },
    "ingest": {
      "number_of_pipelines": 42.0,
      "processor_stats": {
        "additionalProperty1": {
          "count": 42.0,
          "current": 42.0,
          "failed": 42.0,
          "time": "string"
        },
        "additionalProperty2": {
          "count": 42.0,
          "current": 42.0,
          "failed": 42.0,
          "time": "string"
        }
      }
    },
    "jvm": {
      "": 42.0,
      "mem": {
        "heap_max_in_bytes": 42.0,
        "heap_used_in_bytes": 42.0
      },
      "threads": 42.0,
      "versions": [
        {
          "bundled_jdk": true,
          "count": 42.0,
          "using_bundled_jdk": true,
          "version": "string",
          "vm_name": "string",
          "vm_vendor": "string",
          "vm_version": "string"
        }
      ]
    },
    "network_types": {
      "http_types": {
        "additionalProperty1": 42.0,
        "additionalProperty2": 42.0
      },
      "transport_types": {
        "additionalProperty1": 42.0,
        "additionalProperty2": 42.0
      }
    },
    "os": {
      "allocated_processors": 42.0,
      "architectures": [
        {
          "arch": "string",
          "count": 42.0
        }
      ],
      "available_processors": 42.0,
      "mem": {
        "adjusted_total_in_bytes": 42.0,
        "free_in_bytes": 42.0,
        "free_percent": 42.0,
        "total_in_bytes": 42.0,
        "used_in_bytes": 42.0,
        "used_percent": 42.0
      },
      "names": [
        {
          "count": 42.0,
          "name": "string"
        }
      ],
      "pretty_names": [
        {
          "count": 42.0,
          "pretty_name": "string"
        }
      ]
    },
    "packaging_types": [
      {
        "count": 42.0,
        "flavor": "string",
        "type": "string"
      }
    ],
    "plugins": [
      {
        "classname": "string",
        "description": "string",
        "elasticsearch_version": "string",
        "extended_plugins": [
          "string"
        ],
        "has_native_controller": true,
        "java_version": "string",
        "name": "string",
        "version": "string",
        "licensed": true
      }
    ],
    "process": {
      "cpu": {
        "percent": 42.0
      },
      "open_file_descriptors": {
        "avg": 42.0,
        "max": 42.0,
        "min": 42.0
      }
    },
    "versions": [
      "string"
    ]
  },
  "status": "green",
  "timestamp": 42.0
}






















































































































































































Update the connector features Technical preview

PUT /_connector/{connector_id}/_features

Update the connector features in the connector document. This API can be used to control the following aspects of a connector:

  • document-level security
  • incremental syncs
  • advanced sync rules
  • basic sync rules

Normally, the running connector service automatically manages these features. However, you can use this API to override the default behavior.

To sync data using self-managed connectors, you need to deploy the Elastic connector service on your own infrastructure. This service runs automatically on Elastic Cloud for Elastic managed connectors.

Path parameters

  • connector_id string Required

    The unique identifier of the connector to be updated.

application/json

Body Required

  • features object Required

    Additional properties are allowed.

    Hide features attributes Show features attributes object
    • Additional properties are allowed.

      Hide document_level_security attribute Show document_level_security attribute object
    • Additional properties are allowed.

      Hide incremental_sync attribute Show incremental_sync attribute object
    • Additional properties are allowed.

      Hide native_connector_api_keys attribute Show native_connector_api_keys attribute object
    • Additional properties are allowed.

      Hide sync_rules attributes Show sync_rules attributes object
      • advanced object

        Additional properties are allowed.

        Hide advanced attribute Show advanced attribute object
      • basic object

        Additional properties are allowed.

        Hide basic attribute Show basic attribute object

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • result string Required

      Values are created, updated, deleted, not_found, or noop.

PUT /_connector/{connector_id}/_features
curl \
 -X PUT http://api.example.com/_connector/{connector_id}/_features \
 -H "Content-Type: application/json" \
 -d '{"features":{"document_level_security":{"enabled":true},"incremental_sync":{"enabled":true},"native_connector_api_keys":{"enabled":true},"sync_rules":{"advanced":{"enabled":true},"basic":{"enabled":true}}}}'
Request examples
{
  "features": {
    "document_level_security": {
      "enabled": true
    },
    "incremental_sync": {
      "enabled": true
    },
    "native_connector_api_keys": {
      "enabled": true
    },
    "sync_rules": {
      "advanced": {
        "enabled": true
      },
      "basic": {
        "enabled": true
      }
    }
  }
}
Response examples (200)
{
  "result": "created"
}

































































Get auto-follow patterns Added in 6.5.0

GET /_ccr/auto_follow

Get cross-cluster replication auto-follow patterns.

Responses

GET /_ccr/auto_follow
curl \
 -X GET http://api.example.com/_ccr/auto_follow
Response examples (200)
{
  "patterns": [
    {
      "name": "string",
      "pattern": {
        "active": true,
        "remote_cluster": "string",
        "follow_index_pattern": "string",
        "leader_index_patterns": [
          "string"
        ],
        "leader_index_exclusion_patterns": [
          "string"
        ],
        "max_outstanding_read_requests": 42.0
      }
    }
  ]
}


























































































































Check a document

HEAD /{index}/_doc/{id}

Verify that a document exists. For example, check to see if a document with the _id 0 exists:

HEAD my-index-000001/_doc/0

If the document exists, the API returns a status code of 200 - OK. If the document doesn’t exist, the API returns 404 - Not Found.

Versioning support

You can use the version parameter to check the document only if its current version is equal to the specified one.

Internally, Elasticsearch has marked the old document as deleted and added an entirely new document. The old version of the document doesn't disappear immediately, although you won't be able to access it. Elasticsearch cleans up deleted documents in the background as you continue to index more data.

Path parameters

  • index string Required

    A comma-separated list of data streams, indices, and aliases. It supports wildcards (*).

  • id string Required

    A unique document identifier.

Query parameters

  • The node or shard the operation should be performed on. By default, the operation is randomized between the shard replicas.

    If it is set to _local, the operation will prefer to be run on a local allocated shard when possible. If it is set to a custom value, the value is used to guarantee that the same shards will be used for the same custom value. This can help with "jumping values" when hitting different shards in different refresh states. A sample value can be something like the web session ID or the user name.

  • realtime boolean

    If true, the request is real-time as opposed to near-real-time.

  • refresh boolean

    If true, the request refreshes the relevant shards before retrieving the document. Setting it to true should be done after careful thought and verification that this does not cause a heavy load on the system (and slow down indexing).

  • routing string

    A custom value used to route operations to a specific shard.

  • _source boolean | string | array[string]

    Indicates whether to return the _source field (true or false) or lists the fields to return.

  • _source_excludes string | array[string]

    A comma-separated list of source fields to exclude from the response. You can also use this parameter to exclude fields from the subset specified in _source_includes query parameter. If the _source parameter is false, this parameter is ignored.

  • _source_includes string | array[string]

    A comma-separated list of source fields to include in the response. If this parameter is specified, only these source fields are returned. You can exclude fields from this subset using the _source_excludes query parameter. If the _source parameter is false, this parameter is ignored.

  • stored_fields string | array[string]

    A comma-separated list of stored fields to return as part of a hit. If no fields are specified, no stored fields are included in the response. If this field is specified, the _source parameter defaults to false.

  • version number

    Explicit version number for concurrency control. The specified version must match the current version of the document for the request to succeed.

  • The version type.

    Values are internal, external, external_gte, or force.

Responses

HEAD /{index}/_doc/{id}
curl \
 -X HEAD http://api.example.com/{index}/_doc/{id}

Delete documents Added in 5.0.0

POST /{index}/_delete_by_query

Deletes documents that match the specified query.

Path parameters

  • index string | array[string] Required

    Comma-separated list of data streams, indices, and aliases to search. Supports wildcards (*). To search all data streams or indices, omit this parameter or use * or _all.

Query parameters

  • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.

  • analyzer string

    Analyzer to use for the query string.

  • If true, wildcard and prefix queries are analyzed.

  • What to do if delete by query hits version conflicts: abort or proceed.

    Values are abort or proceed.

  • The default operator for query string query: AND or OR.

    Values are and, AND, or, or OR.

  • df string

    Field to use as default where no field prefix is given in the query string.

  • expand_wildcards string | array[string]

    Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as open,hidden. Valid values are: all, open, closed, hidden, none.

  • from number

    Starting offset (default: 0)

  • If false, the request returns an error if it targets a missing or closed index.

  • lenient boolean

    If true, format-based query failures (such as providing text to a numeric field) in the query string will be ignored.

  • max_docs number

    Maximum number of documents to process. Defaults to all documents.

  • Specifies the node or shard the operation should be performed on. Random by default.

  • refresh boolean

    If true, Elasticsearch refreshes all shards involved in the delete by query after the request completes.

  • If true, the request cache is used for this request. Defaults to the index-level setting.

  • The throttle for this request in sub-requests per second.

  • routing string

    Custom value used to route operations to a specific shard.

  • q string

    Query in the Lucene query string syntax.

  • scroll string

    Period to retain the search context for scrolling.

  • Size of the scroll request that powers the operation.

  • Explicit timeout for each search request. Defaults to no timeout.

  • The type of the search operation. Available options: query_then_fetch, dfs_query_then_fetch.

    Values are query_then_fetch or dfs_query_then_fetch.

  • slices number | string

    The number of slices this task should be divided into.

  • sort array[string]

    A comma-separated list of : pairs.

  • stats array[string]

    Specific tag of the request for logging and statistical purposes.

  • Maximum number of documents to collect for each shard. If a query reaches this limit, Elasticsearch terminates the query early. Elasticsearch collects documents before sorting. Use with caution. Elasticsearch applies this parameter to each shard handling the request. When possible, let Elasticsearch perform early termination automatically. Avoid specifying this parameter for requests that target data streams with backing indices across multiple data tiers.

  • timeout string

    Period each deletion request waits for active shards.

  • version boolean

    If true, returns the document version as part of a hit.

  • wait_for_active_shards number | string

    The number of shard copies that must be active before proceeding with the operation. Set to all or any positive integer up to the total number of shards in the index (number_of_replicas+1).

  • If true, the request blocks until the operation is complete.

application/json

Body Required

  • max_docs number

    The maximum number of documents to delete.

  • query object

    An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

    Additional properties are allowed.

  • slice object

    Additional properties are allowed.

    Hide slice attributes Show slice attributes object
    • field string

      Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

    • id string Required
    • max number Required

Responses

POST /{index}/_delete_by_query
curl \
 -X POST http://api.example.com/{index}/_delete_by_query \
 -H "Content-Type: application/json" \
 -d '{"max_docs":42.0,"query":{},"slice":{"field":"string","id":"string","max":42.0}}'
Request examples
{
  "max_docs": 42.0,
  "query": {},
  "slice": {
    "field": "string",
    "id": "string",
    "max": 42.0
  }
}
Response examples (200)
{
  "batches": 42.0,
  "deleted": 42.0,
  "failures": [
    {
      "cause": {
        "type": "string",
        "reason": "string",
        "stack_trace": "string",
        "caused_by": {},
        "root_cause": [
          {}
        ],
        "suppressed": [
          {}
        ]
      },
      "id": "string",
      "index": "string",
      "status": 42.0,
      "type": "string"
    }
  ],
  "noops": 42.0,
  "requests_per_second": 42.0,
  "retries": {
    "bulk": 42.0,
    "search": 42.0
  },
  "slice_id": 42.0,
  "": 42.0,
  "throttled": "string",
  "throttled_until": "string",
  "timed_out": true,
  "total": 42.0,
  "version_conflicts": 42.0
}








Check for a document source Added in 5.4.0

HEAD /{index}/_source/{id}

Check whether a document source exists in an index. For example:

HEAD my-index-000001/_source/1

A document's source is not available if it is disabled in the mapping.

Path parameters

  • index string Required

    A comma-separated list of data streams, indices, and aliases. It supports wildcards (*).

  • id string Required

    A unique identifier for the document.

Query parameters

  • The node or shard the operation should be performed on. By default, the operation is randomized between the shard replicas.

  • realtime boolean

    If true, the request is real-time as opposed to near-real-time.

  • refresh boolean

    If true, the request refreshes the relevant shards before retrieving the document. Setting it to true should be done after careful thought and verification that this does not cause a heavy load on the system (and slow down indexing).

  • routing string

    A custom value used to route operations to a specific shard.

  • _source boolean | string | array[string]

    Indicates whether to return the _source field (true or false) or lists the fields to return.

  • _source_excludes string | array[string]

    A comma-separated list of source fields to exclude in the response.

  • _source_includes string | array[string]

    A comma-separated list of source fields to include in the response.

  • version number

    The version number for concurrency control. It must match the current version of the document for the request to succeed.

  • The version type.

    Values are internal, external, external_gte, or force.

Responses

HEAD /{index}/_source/{id}
curl \
 -X HEAD http://api.example.com/{index}/_source/{id}




















Get multiple term vectors

GET /_mtermvectors

You can specify existing documents by index and ID or provide artificial documents in the body of the request. You can specify the index in the request body or request URI. The response contains a docs array with all the fetched termvectors. Each element has the structure provided by the termvectors API.

Query parameters

  • ids array[string]

    A comma-separated list of documents ids. You must define ids as parameter or set "ids" or "docs" in the request body

  • fields string | array[string]

    Comma-separated list or wildcard expressions of fields to include in the statistics. Used as the default list unless a specific field list is provided in the completion_fields or fielddata_fields parameters.

  • If true, the response includes the document count, sum of document frequencies, and sum of total term frequencies.

  • offsets boolean

    If true, the response includes term offsets.

  • payloads boolean

    If true, the response includes term payloads.

  • positions boolean

    If true, the response includes term positions.

  • Specifies the node or shard the operation should be performed on. Random by default.

  • realtime boolean

    If true, the request is real-time as opposed to near-real-time.

  • routing string

    Custom value used to route operations to a specific shard.

  • If true, the response includes term frequency and document frequency.

  • version number

    If true, returns the document version as part of a hit.

  • Specific version type.

    Values are internal, external, external_gte, or force.

application/json

Body

  • docs array[object]

    Array of existing or artificial documents.

    Hide docs attributes Show docs attributes object
    • _id string
    • _index string
    • doc object

      An artificial document (a document not present in the index) for which you want to retrieve term vectors.

      Additional properties are allowed.

    • fields string | array[string]
    • If true, the response includes the document count, sum of document frequencies, and sum of total term frequencies.

    • filter object

      Additional properties are allowed.

      Hide filter attributes Show filter attributes object
      • Ignore words which occur in more than this many docs. Defaults to unbounded.

      • Maximum number of terms that must be returned per field.

      • Ignore words with more than this frequency in the source doc. Defaults to unbounded.

      • The maximum word length above which words will be ignored. Defaults to unbounded.

      • Ignore terms which do not occur in at least this many docs.

      • Ignore words with less than this frequency in the source doc.

      • The minimum word length below which words will be ignored.

    • offsets boolean

      If true, the response includes term offsets.

    • payloads boolean

      If true, the response includes term payloads.

    • positions boolean

      If true, the response includes term positions.

    • routing string
    • If true, the response includes term frequency and document frequency.

    • version number
    • Values are internal, external, external_gte, or force.

  • ids array[string]

    Simplified syntax to specify documents by their ID if they're in the same index.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • docs array[object] Required
      Hide docs attributes Show docs attributes object
      • _id string
      • _index string Required
      • _version number
      • took number
      • found boolean
      • Hide term_vectors attribute Show term_vectors attribute object
        • * object Additional properties

          Additional properties are allowed.

          Hide * attributes Show * attributes object
          • Additional properties are allowed.

            Hide field_statistics attributes Show field_statistics attributes object
          • terms object Required
            Hide terms attribute Show terms attribute object
            • * object Additional properties

              Additional properties are allowed.

      • error object

        Additional properties are allowed.

        Hide error attributes Show error attributes object
        • type string Required

          The type of error

        • reason string

          A human-readable explanation of the error, in English.

        • The server stack trace. Present only if the error_trace=true parameter was sent with the request.

        • Additional properties are allowed.

        • root_cause array[object]

          Additional properties are allowed.

        • suppressed array[object]

          Additional properties are allowed.

GET /_mtermvectors
curl \
 -X GET http://api.example.com/_mtermvectors \
 -H "Content-Type: application/json" \
 -d '{"docs":[{"_id":"string","_index":"string","doc":{},"fields":"string","field_statistics":true,"filter":{"max_doc_freq":42.0,"max_num_terms":42.0,"max_term_freq":42.0,"max_word_length":42.0,"min_doc_freq":42.0,"min_term_freq":42.0,"min_word_length":42.0},"offsets":true,"payloads":true,"positions":true,"routing":"string","term_statistics":true,"version":42.0,"version_type":"internal"}],"ids":["string"]}'
Request examples
{
  "docs": [
    {
      "_id": "string",
      "_index": "string",
      "doc": {},
      "fields": "string",
      "field_statistics": true,
      "filter": {
        "max_doc_freq": 42.0,
        "max_num_terms": 42.0,
        "max_term_freq": 42.0,
        "max_word_length": 42.0,
        "min_doc_freq": 42.0,
        "min_term_freq": 42.0,
        "min_word_length": 42.0
      },
      "offsets": true,
      "payloads": true,
      "positions": true,
      "routing": "string",
      "term_statistics": true,
      "version": 42.0,
      "version_type": "internal"
    }
  ],
  "ids": [
    "string"
  ]
}
Response examples (200)
{
  "docs": [
    {
      "_id": "string",
      "_index": "string",
      "_version": 42.0,
      "took": 42.0,
      "found": true,
      "term_vectors": {
        "additionalProperty1": {
          "field_statistics": {
            "doc_count": 42.0,
            "sum_doc_freq": 42.0,
            "sum_ttf": 42.0
          },
          "terms": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          }
        },
        "additionalProperty2": {
          "field_statistics": {
            "doc_count": 42.0,
            "sum_doc_freq": 42.0,
            "sum_ttf": 42.0
          },
          "terms": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          }
        }
      },
      "error": {
        "type": "string",
        "reason": "string",
        "stack_trace": "string",
        "caused_by": {},
        "root_cause": [
          {}
        ],
        "suppressed": [
          {}
        ]
      }
    }
  ]
}

























































Delete an enrich policy Added in 7.5.0

DELETE /_enrich/policy/{name}

Deletes an existing enrich policy and its enrich index.

Path parameters

  • name string Required

    Enrich policy to delete.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • acknowledged boolean Required

      For a successful response, this value is always true. On failure, an exception is returned instead.

DELETE /_enrich/policy/{name}
curl \
 -X DELETE http://api.example.com/_enrich/policy/{name}
Response examples (200)
{
  "acknowledged": true
}












EQL

Event Query Language (EQL) is a query language for event-based time series data, such as logs, metrics, and traces.






















































































Get component templates Added in 7.8.0

GET /_component_template/{name}

Get information about component templates.

Path parameters

  • name string Required

    Comma-separated list of component template names used to limit the request. Wildcard (*) expressions are supported.

Query parameters

  • If true, returns settings in flat format.

  • Return all default configurations for the component template (default: false)

  • local boolean

    If true, the request retrieves information from the local node only. If false, information is retrieved from the master node.

  • Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • component_templates array[object] Required
      Hide component_templates attributes Show component_templates attributes object
      • name string Required
      • component_template object Required

        Additional properties are allowed.

        Hide component_template attributes Show component_template attributes object
        • template object Required

          Additional properties are allowed.

          Hide template attributes Show template attributes object
        • version number
        • _meta object
          Hide _meta attribute Show _meta attribute object
          • * object Additional properties

            Additional properties are allowed.

GET /_component_template/{name}
curl \
 -X GET http://api.example.com/_component_template/{name}
Response examples (200)
{
  "component_templates": [
    {
      "name": "string",
      "component_template": {
        "template": {
          "_meta": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "version": 42.0,
          "settings": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "mappings": {
            "all_field": {},
            "date_detection": true,
            "dynamic": "strict",
            "dynamic_date_formats": [
              "string"
            ],
            "dynamic_templates": [
              {}
            ],
            "_field_names": {},
            "index_field": {},
            "_meta": {},
            "numeric_detection": true,
            "properties": {},
            "_routing": {},
            "_size": {},
            "_source": {},
            "runtime": {},
            "enabled": true,
            "subobjects": true,
            "_data_stream_timestamp": {}
          },
          "aliases": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "lifecycle": {
            "data_retention": "string",
            "downsampling": {},
            "rollover": {}
          }
        },
        "version": 42.0,
        "_meta": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        }
      }
    }
  ]
}








































Get tokens from text analysis

POST /_analyze

The analyze API performs analysis on a text string and returns the resulting tokens.

Generating excessive amount of tokens may cause a node to run out of memory. The index.analyze.max_token_count setting enables you to limit the number of tokens that can be produced. If more than this limit of tokens gets generated, an error occurs. The _analyze endpoint without a specified index will always use 10000 as its limit.

application/json

Body

Responses

POST /_analyze
curl \
 -X POST http://api.example.com/_analyze \
 -H "Content-Type: application/json" \
 -d '{"text":"this is a test","analyzer":"standard"}'
Request example
You can apply any of the built-in analyzers to the text string without specifying an index.
{
  "text": "this is a test",
  "analyzer": "standard"
}
Response examples (200)
{
  "detail": {
    "analyzer": {
      "name": "string",
      "tokens": [
        {
          "bytes": "string",
          "end_offset": 42.0,
          "keyword": true,
          "position": 42.0,
          "positionLength": 42.0,
          "start_offset": 42.0,
          "termFrequency": 42.0,
          "token": "string",
          "type": "string"
        }
      ]
    },
    "charfilters": [
      {
        "filtered_text": [
          "string"
        ],
        "name": "string"
      }
    ],
    "custom_analyzer": true,
    "tokenfilters": [
      {
        "name": "string",
        "tokens": [
          {
            "bytes": "string",
            "end_offset": 42.0,
            "keyword": true,
            "position": 42.0,
            "positionLength": 42.0,
            "start_offset": 42.0,
            "termFrequency": 42.0,
            "token": "string",
            "type": "string"
          }
        ]
      }
    ],
    "tokenizer": {
      "name": "string",
      "tokens": [
        {
          "bytes": "string",
          "end_offset": 42.0,
          "keyword": true,
          "position": 42.0,
          "positionLength": 42.0,
          "start_offset": 42.0,
          "termFrequency": 42.0,
          "token": "string",
          "type": "string"
        }
      ]
    }
  },
  "tokens": [
    {
      "end_offset": 42.0,
      "position": 42.0,
      "positionLength": 42.0,
      "start_offset": 42.0,
      "token": "string",
      "type": "string"
    }
  ]
}




Get tokens from text analysis

POST /{index}/_analyze

The analyze API performs analysis on a text string and returns the resulting tokens.

Generating excessive amount of tokens may cause a node to run out of memory. The index.analyze.max_token_count setting enables you to limit the number of tokens that can be produced. If more than this limit of tokens gets generated, an error occurs. The _analyze endpoint without a specified index will always use 10000 as its limit.

Path parameters

  • index string Required

    Index used to derive the analyzer. If specified, the analyzer or field parameter overrides this value. If no index is specified or the index does not have a default analyzer, the analyze API uses the standard analyzer.

application/json

Body

Responses

POST /{index}/_analyze
curl \
 -X POST http://api.example.com/{index}/_analyze \
 -H "Content-Type: application/json" \
 -d '{"text":"this is a test","analyzer":"standard"}'
Request example
You can apply any of the built-in analyzers to the text string without specifying an index.
{
  "text": "this is a test",
  "analyzer": "standard"
}
Response examples (200)
{
  "detail": {
    "analyzer": {
      "name": "string",
      "tokens": [
        {
          "bytes": "string",
          "end_offset": 42.0,
          "keyword": true,
          "position": 42.0,
          "positionLength": 42.0,
          "start_offset": 42.0,
          "termFrequency": 42.0,
          "token": "string",
          "type": "string"
        }
      ]
    },
    "charfilters": [
      {
        "filtered_text": [
          "string"
        ],
        "name": "string"
      }
    ],
    "custom_analyzer": true,
    "tokenfilters": [
      {
        "name": "string",
        "tokens": [
          {
            "bytes": "string",
            "end_offset": 42.0,
            "keyword": true,
            "position": 42.0,
            "positionLength": 42.0,
            "start_offset": 42.0,
            "termFrequency": 42.0,
            "token": "string",
            "type": "string"
          }
        ]
      }
    ],
    "tokenizer": {
      "name": "string",
      "tokens": [
        {
          "bytes": "string",
          "end_offset": 42.0,
          "keyword": true,
          "position": 42.0,
          "positionLength": 42.0,
          "start_offset": 42.0,
          "termFrequency": 42.0,
          "token": "string",
          "type": "string"
        }
      ]
    }
  },
  "tokens": [
    {
      "end_offset": 42.0,
      "position": 42.0,
      "positionLength": 42.0,
      "start_offset": 42.0,
      "token": "string",
      "type": "string"
    }
  ]
}




























Delete indices

DELETE /{index}

Deleting an index deletes its documents, shards, and metadata. It does not delete related Kibana components, such as data views, visualizations, or dashboards.

You cannot delete the current write index of a data stream. To delete the index, you must roll over the data stream so a new write index is created. You can then use the delete index API to delete the previous write index.

Path parameters

  • index string | array[string] Required

    Comma-separated list of indices to delete. You cannot specify index aliases. By default, this parameter does not support wildcards (*) or _all. To use wildcards or _all, set the action.destructive_requires_name cluster setting to false.

Query parameters

  • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices.

  • expand_wildcards string | array[string]

    Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as open,hidden. Valid values are: all, open, closed, hidden, none.

  • If false, the request returns an error if it targets a missing or closed index.

  • Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

  • timeout string

    Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • acknowledged boolean Required

      For a successful response, this value is always true. On failure, an exception is returned instead.

    • _shards object

      Additional properties are allowed.

      Hide _shards attributes Show _shards attributes object
DELETE /{index}
curl \
 -X DELETE http://api.example.com/{index}
Response examples (200)
{
  "acknowledged": true,
  "_shards": {
    "failed": 42.0,
    "successful": 42.0,
    "total": 42.0,
    "failures": [
      {
        "index": "string",
        "node": "string",
        "reason": {
          "type": "string",
          "reason": "string",
          "stack_trace": "string",
          "caused_by": {},
          "root_cause": [
            {}
          ],
          "suppressed": [
            {}
          ]
        },
        "shard": 42.0,
        "status": "string"
      }
    ],
    "skipped": 42.0
  }
}
























Create or update an alias

PUT /{index}/_aliases/{name}

Adds a data stream or index to an alias.

Path parameters

  • index string | array[string] Required

    Comma-separated list of data streams or indices to add. Supports wildcards (*). Wildcard patterns that match both data streams and indices return an error.

  • name string Required

    Alias to update. If the alias doesn’t exist, the request creates it. Index alias names support date math.

Query parameters

  • Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

  • timeout string

    Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

application/json

Body

  • filter object

    An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

    Additional properties are allowed.

  • If true, sets the write index or data stream for the alias. If an alias points to multiple indices or data streams and is_write_index isn’t set, the alias rejects write requests. If an index alias points to one index and is_write_index isn’t set, the index automatically acts as the write index. Data stream aliases don’t automatically set a write data stream, even if the alias points to one data stream.

  • routing string

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • acknowledged boolean Required

      For a successful response, this value is always true. On failure, an exception is returned instead.

PUT /{index}/_aliases/{name}
curl \
 -X PUT http://api.example.com/{index}/_aliases/{name} \
 -H "Content-Type: application/json" \
 -d '{"filter":{},"index_routing":"string","is_write_index":true,"routing":"string","search_routing":"string"}'
Request examples
{
  "filter": {},
  "index_routing": "string",
  "is_write_index": true,
  "routing": "string",
  "search_routing": "string"
}
Response examples (200)
{
  "acknowledged": true
}

Create or update an alias

POST /{index}/_aliases/{name}

Adds a data stream or index to an alias.

Path parameters

  • index string | array[string] Required

    Comma-separated list of data streams or indices to add. Supports wildcards (*). Wildcard patterns that match both data streams and indices return an error.

  • name string Required

    Alias to update. If the alias doesn’t exist, the request creates it. Index alias names support date math.

Query parameters

  • Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

  • timeout string

    Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

application/json

Body

  • filter object

    An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

    Additional properties are allowed.

  • If true, sets the write index or data stream for the alias. If an alias points to multiple indices or data streams and is_write_index isn’t set, the alias rejects write requests. If an index alias points to one index and is_write_index isn’t set, the index automatically acts as the write index. Data stream aliases don’t automatically set a write data stream, even if the alias points to one data stream.

  • routing string

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • acknowledged boolean Required

      For a successful response, this value is always true. On failure, an exception is returned instead.

POST /{index}/_aliases/{name}
curl \
 -X POST http://api.example.com/{index}/_aliases/{name} \
 -H "Content-Type: application/json" \
 -d '{"filter":{},"index_routing":"string","is_write_index":true,"routing":"string","search_routing":"string"}'
Request examples
{
  "filter": {},
  "index_routing": "string",
  "is_write_index": true,
  "routing": "string",
  "search_routing": "string"
}
Response examples (200)
{
  "acknowledged": true
}












































Check existence of index templates

HEAD /_template/{name}

Get information about whether index templates exist. Index templates define settings, mappings, and aliases that can be applied automatically to new indices.

IMPORTANT: This documentation is about legacy index templates, which are deprecated and will be replaced by the composable templates introduced in Elasticsearch 7.8.

Path parameters

  • name string | array[string] Required

    A comma-separated list of index template names used to limit the request. Wildcard (*) expressions are supported.

Query parameters

  • Indicates whether to use a flat format for the response.

  • local boolean

    Indicates whether to get information from the local node only.

  • The period to wait for the master node. If the master node is not available before the timeout expires, the request fails and returns an error. To indicate that the request should never timeout, set it to -1.

Responses

HEAD /_template/{name}
curl \
 -X HEAD http://api.example.com/_template/{name}








































Get aliases

GET /_alias

Retrieves information for one or more data stream or index aliases.

Query parameters

  • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices.

  • expand_wildcards string | array[string]

    Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as open,hidden. Valid values are: all, open, closed, hidden, none.

  • If false, the request returns an error if it targets a missing or closed index.

  • local boolean Deprecated

    If true, the request retrieves information from the local node only.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • * object Additional properties

      Additional properties are allowed.

      Hide * attribute Show * attribute object
      • aliases object Required
        Hide aliases attribute Show aliases attribute object
        • * object Additional properties

          Additional properties are allowed.

          Hide * attributes Show * attributes object
          • filter object

            An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

            Additional properties are allowed.

          • Value used to route indexing operations to a specific shard. If specified, this overwrites the routing value for indexing operations.

          • If true, the index is the write index for the alias.

          • routing string

            Value used to route indexing and search operations to a specific shard.

          • Value used to route search operations to a specific shard. If specified, this overwrites the routing value for search operations.

          • is_hidden boolean

            If true, the alias is hidden. All indices for the alias must have the same is_hidden value.

GET /_alias
curl \
 -X GET http://api.example.com/_alias
Response examples (200)
{
  "additionalProperty1": {
    "aliases": {
      "additionalProperty1": {
        "filter": {},
        "index_routing": "string",
        "is_write_index": true,
        "routing": "string",
        "search_routing": "string",
        "is_hidden": true
      },
      "additionalProperty2": {
        "filter": {},
        "index_routing": "string",
        "is_write_index": true,
        "routing": "string",
        "search_routing": "string",
        "is_hidden": true
      }
    }
  },
  "additionalProperty2": {
    "aliases": {
      "additionalProperty1": {
        "filter": {},
        "index_routing": "string",
        "is_write_index": true,
        "routing": "string",
        "search_routing": "string",
        "is_hidden": true
      },
      "additionalProperty2": {
        "filter": {},
        "index_routing": "string",
        "is_write_index": true,
        "routing": "string",
        "search_routing": "string",
        "is_hidden": true
      }
    }
  }
}
















Get mapping definitions

GET /_mapping

For data streams, the API retrieves mappings for the stream’s backing indices.

Query parameters

  • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices.

  • expand_wildcards string | array[string]

    Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as open,hidden. Valid values are: all, open, closed, hidden, none.

  • If false, the request returns an error if it targets a missing or closed index.

  • local boolean

    If true, the request retrieves information from the local node only.

  • Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

Responses

GET /_mapping
curl \
 -X GET http://api.example.com/_mapping
Response examples (200)
{
  "additionalProperty1": {
    "item": {
      "all_field": {
        "analyzer": "string",
        "enabled": true,
        "omit_norms": true,
        "search_analyzer": "string",
        "similarity": "string",
        "store": true,
        "store_term_vector_offsets": true,
        "store_term_vector_payloads": true,
        "store_term_vector_positions": true,
        "store_term_vectors": true
      },
      "date_detection": true,
      "dynamic": "strict",
      "dynamic_date_formats": [
        "string"
      ],
      "dynamic_templates": [
        {}
      ],
      "_field_names": {
        "enabled": true
      },
      "index_field": {
        "enabled": true
      },
      "_meta": {
        "additionalProperty1": {},
        "additionalProperty2": {}
      },
      "numeric_detection": true,
      "properties": {},
      "_routing": {
        "required": true
      },
      "_size": {
        "enabled": true
      },
      "_source": {
        "compress": true,
        "compress_threshold": "string",
        "enabled": true,
        "excludes": [
          "string"
        ],
        "includes": [
          "string"
        ],
        "mode": "disabled"
      },
      "runtime": {
        "additionalProperty1": {
          "fields": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "fetch_fields": [
            {}
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {},
            "options": {}
          },
          "type": "boolean"
        },
        "additionalProperty2": {
          "fields": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "fetch_fields": [
            {}
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {},
            "options": {}
          },
          "type": "boolean"
        }
      },
      "enabled": true,
      "subobjects": true,
      "_data_stream_timestamp": {
        "enabled": true
      }
    },
    "mappings": {
      "all_field": {
        "analyzer": "string",
        "enabled": true,
        "omit_norms": true,
        "search_analyzer": "string",
        "similarity": "string",
        "store": true,
        "store_term_vector_offsets": true,
        "store_term_vector_payloads": true,
        "store_term_vector_positions": true,
        "store_term_vectors": true
      },
      "date_detection": true,
      "dynamic": "strict",
      "dynamic_date_formats": [
        "string"
      ],
      "dynamic_templates": [
        {}
      ],
      "_field_names": {
        "enabled": true
      },
      "index_field": {
        "enabled": true
      },
      "_meta": {
        "additionalProperty1": {},
        "additionalProperty2": {}
      },
      "numeric_detection": true,
      "properties": {},
      "_routing": {
        "required": true
      },
      "_size": {
        "enabled": true
      },
      "_source": {
        "compress": true,
        "compress_threshold": "string",
        "enabled": true,
        "excludes": [
          "string"
        ],
        "includes": [
          "string"
        ],
        "mode": "disabled"
      },
      "runtime": {
        "additionalProperty1": {
          "fields": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "fetch_fields": [
            {}
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {},
            "options": {}
          },
          "type": "boolean"
        },
        "additionalProperty2": {
          "fields": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "fetch_fields": [
            {}
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {},
            "options": {}
          },
          "type": "boolean"
        }
      },
      "enabled": true,
      "subobjects": true,
      "_data_stream_timestamp": {
        "enabled": true
      }
    }
  },
  "additionalProperty2": {
    "item": {
      "all_field": {
        "analyzer": "string",
        "enabled": true,
        "omit_norms": true,
        "search_analyzer": "string",
        "similarity": "string",
        "store": true,
        "store_term_vector_offsets": true,
        "store_term_vector_payloads": true,
        "store_term_vector_positions": true,
        "store_term_vectors": true
      },
      "date_detection": true,
      "dynamic": "strict",
      "dynamic_date_formats": [
        "string"
      ],
      "dynamic_templates": [
        {}
      ],
      "_field_names": {
        "enabled": true
      },
      "index_field": {
        "enabled": true
      },
      "_meta": {
        "additionalProperty1": {},
        "additionalProperty2": {}
      },
      "numeric_detection": true,
      "properties": {},
      "_routing": {
        "required": true
      },
      "_size": {
        "enabled": true
      },
      "_source": {
        "compress": true,
        "compress_threshold": "string",
        "enabled": true,
        "excludes": [
          "string"
        ],
        "includes": [
          "string"
        ],
        "mode": "disabled"
      },
      "runtime": {
        "additionalProperty1": {
          "fields": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "fetch_fields": [
            {}
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {},
            "options": {}
          },
          "type": "boolean"
        },
        "additionalProperty2": {
          "fields": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "fetch_fields": [
            {}
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {},
            "options": {}
          },
          "type": "boolean"
        }
      },
      "enabled": true,
      "subobjects": true,
      "_data_stream_timestamp": {
        "enabled": true
      }
    },
    "mappings": {
      "all_field": {
        "analyzer": "string",
        "enabled": true,
        "omit_norms": true,
        "search_analyzer": "string",
        "similarity": "string",
        "store": true,
        "store_term_vector_offsets": true,
        "store_term_vector_payloads": true,
        "store_term_vector_positions": true,
        "store_term_vectors": true
      },
      "date_detection": true,
      "dynamic": "strict",
      "dynamic_date_formats": [
        "string"
      ],
      "dynamic_templates": [
        {}
      ],
      "_field_names": {
        "enabled": true
      },
      "index_field": {
        "enabled": true
      },
      "_meta": {
        "additionalProperty1": {},
        "additionalProperty2": {}
      },
      "numeric_detection": true,
      "properties": {},
      "_routing": {
        "required": true
      },
      "_size": {
        "enabled": true
      },
      "_source": {
        "compress": true,
        "compress_threshold": "string",
        "enabled": true,
        "excludes": [
          "string"
        ],
        "includes": [
          "string"
        ],
        "mode": "disabled"
      },
      "runtime": {
        "additionalProperty1": {
          "fields": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "fetch_fields": [
            {}
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {},
            "options": {}
          },
          "type": "boolean"
        },
        "additionalProperty2": {
          "fields": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "fetch_fields": [
            {}
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {},
            "options": {}
          },
          "type": "boolean"
        }
      },
      "enabled": true,
      "subobjects": true,
      "_data_stream_timestamp": {
        "enabled": true
      }
    }
  }
}




Update field mappings

PUT /{index}/_mapping

Add new fields to an existing data stream or index. You can also use this API to change the search settings of existing fields and add new properties to existing object fields. For data streams, these changes are applied to all backing indices by default.

Add multi-fields to an existing field

Multi-fields let you index the same field in different ways. You can use this API to update the fields mapping parameter and enable multi-fields for an existing field. WARNING: If an index (or data stream) contains documents when you add a multi-field, those documents will not have values for the new multi-field. You can populate the new multi-field with the update by query API.

Change supported mapping parameters for an existing field

The documentation for each mapping parameter indicates whether you can update it for an existing field using this API. For example, you can use the update mapping API to update the ignore_above parameter.

Change the mapping of an existing field

Except for supported mapping parameters, you can't change the mapping or field type of an existing field. Changing an existing field could invalidate data that's already indexed.

If you need to change the mapping of a field in a data stream's backing indices, refer to documentation about modifying data streams. If you need to change the mapping of a field in other indices, create a new index with the correct mapping and reindex your data into that index.

Rename a field

Renaming a field would invalidate data already indexed under the old field name. Instead, add an alias field to create an alternate field name.

Path parameters

  • index string | array[string] Required

    A comma-separated list of index names the mapping should be added to (supports wildcards); use _all or omit to add the mapping on all indices.

Query parameters

  • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices.

  • expand_wildcards string | array[string]

    Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as open,hidden. Valid values are: all, open, closed, hidden, none.

  • If false, the request returns an error if it targets a missing or closed index.

  • Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

  • timeout string

    Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

  • If true, the mappings are applied only to the current write index for the target.

application/json

Body Required

  • Controls whether dynamic date detection is enabled.

  • dynamic string

    Values are strict, runtime, true, or false.

  • If date detection is enabled then new string fields are checked against 'dynamic_date_formats' and if the value matches then a new date field is added instead of string.

  • dynamic_templates object | array[object]

    Specify dynamic templates for the mapping.

  • Additional properties are allowed.

    Hide _field_names attribute Show _field_names attribute object
  • _meta object
    Hide _meta attribute Show _meta attribute object
    • * object Additional properties

      Additional properties are allowed.

  • Automatically map strings into numeric data types for all fields.

  • Mapping for a field. For new fields, this mapping can include:

    • Field name
    • Field data type
    • Mapping parameters
  • _routing object

    Additional properties are allowed.

    Hide _routing attribute Show _routing attribute object
  • _source object

    Additional properties are allowed.

    Hide _source attributes Show _source attributes object
  • runtime object
    Hide runtime attribute Show runtime attribute object
    • * object Additional properties

      Additional properties are allowed.

      Hide * attributes Show * attributes object
      • fields object

        For type composite

        Hide fields attribute Show fields attribute object
        • * object Additional properties

          Additional properties are allowed.

          Hide * attribute Show * attribute object
          • type string Required

            Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

      • fetch_fields array[object]

        For type lookup

        Hide fetch_fields attributes Show fetch_fields attributes object
        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • format string
      • format string

        A custom format for date type runtime fields.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • script object

        Additional properties are allowed.

        Hide script attributes Show script attributes object
        • source string

          The script source.

        • id string
        • params object

          Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

          Hide params attribute Show params attribute object
          • * object Additional properties

            Additional properties are allowed.

        • lang string

          Any of:

          Values are painless, expression, mustache, or java.

        • options object
          Hide options attribute Show options attribute object
          • * string Additional properties
      • type string Required

        Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • acknowledged boolean Required

      For a successful response, this value is always true. On failure, an exception is returned instead.

    • _shards object

      Additional properties are allowed.

      Hide _shards attributes Show _shards attributes object
PUT /{index}/_mapping
curl \
 -X PUT http://api.example.com/{index}/_mapping \
 -H "Content-Type: application/json" \
 -d '{"properties":{"user":{"properties":{"name":{"type":"keyword"}}}}}'
Request example
The update mapping API can be applied to multiple data streams or indices with a single request. For example, run `PUT /my-index-000001,my-index-000002/_mapping` to update mappings for the `my-index-000001` and `my-index-000002` indices at the same time.
{
  "properties": {
    "user": {
      "properties": {
        "name": {
          "type": "keyword"
        }
      }
    }
  }
}
Response examples (200)
{
  "acknowledged": true,
  "_shards": {
    "failed": 42.0,
    "successful": 42.0,
    "total": 42.0,
    "failures": [
      {
        "index": "string",
        "node": "string",
        "reason": {
          "type": "string",
          "reason": "string",
          "stack_trace": "string",
          "caused_by": {},
          "root_cause": [
            {}
          ],
          "suppressed": [
            {}
          ]
        },
        "shard": 42.0,
        "status": "string"
      }
    ],
    "skipped": 42.0
  }
}

Update field mappings

POST /{index}/_mapping

Add new fields to an existing data stream or index. You can also use this API to change the search settings of existing fields and add new properties to existing object fields. For data streams, these changes are applied to all backing indices by default.

Add multi-fields to an existing field

Multi-fields let you index the same field in different ways. You can use this API to update the fields mapping parameter and enable multi-fields for an existing field. WARNING: If an index (or data stream) contains documents when you add a multi-field, those documents will not have values for the new multi-field. You can populate the new multi-field with the update by query API.

Change supported mapping parameters for an existing field

The documentation for each mapping parameter indicates whether you can update it for an existing field using this API. For example, you can use the update mapping API to update the ignore_above parameter.

Change the mapping of an existing field

Except for supported mapping parameters, you can't change the mapping or field type of an existing field. Changing an existing field could invalidate data that's already indexed.

If you need to change the mapping of a field in a data stream's backing indices, refer to documentation about modifying data streams. If you need to change the mapping of a field in other indices, create a new index with the correct mapping and reindex your data into that index.

Rename a field

Renaming a field would invalidate data already indexed under the old field name. Instead, add an alias field to create an alternate field name.

Path parameters

  • index string | array[string] Required

    A comma-separated list of index names the mapping should be added to (supports wildcards); use _all or omit to add the mapping on all indices.

Query parameters

  • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices.

  • expand_wildcards string | array[string]

    Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as open,hidden. Valid values are: all, open, closed, hidden, none.

  • If false, the request returns an error if it targets a missing or closed index.

  • Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

  • timeout string

    Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

  • If true, the mappings are applied only to the current write index for the target.

application/json

Body Required

  • Controls whether dynamic date detection is enabled.

  • dynamic string

    Values are strict, runtime, true, or false.

  • If date detection is enabled then new string fields are checked against 'dynamic_date_formats' and if the value matches then a new date field is added instead of string.

  • dynamic_templates object | array[object]

    Specify dynamic templates for the mapping.

  • Additional properties are allowed.

    Hide _field_names attribute Show _field_names attribute object
  • _meta object
    Hide _meta attribute Show _meta attribute object
    • * object Additional properties

      Additional properties are allowed.

  • Automatically map strings into numeric data types for all fields.

  • Mapping for a field. For new fields, this mapping can include:

    • Field name
    • Field data type
    • Mapping parameters
  • _routing object

    Additional properties are allowed.

    Hide _routing attribute Show _routing attribute object
  • _source object

    Additional properties are allowed.

    Hide _source attributes Show _source attributes object
  • runtime object
    Hide runtime attribute Show runtime attribute object
    • * object Additional properties

      Additional properties are allowed.

      Hide * attributes Show * attributes object
      • fields object

        For type composite

        Hide fields attribute Show fields attribute object
        • * object Additional properties

          Additional properties are allowed.

          Hide * attribute Show * attribute object
          • type string Required

            Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

      • fetch_fields array[object]

        For type lookup

        Hide fetch_fields attributes Show fetch_fields attributes object
        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • format string
      • format string

        A custom format for date type runtime fields.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • script object

        Additional properties are allowed.

        Hide script attributes Show script attributes object
        • source string

          The script source.

        • id string
        • params object

          Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

          Hide params attribute Show params attribute object
          • * object Additional properties

            Additional properties are allowed.

        • lang string

          Any of:

          Values are painless, expression, mustache, or java.

        • options object
          Hide options attribute Show options attribute object
          • * string Additional properties
      • type string Required

        Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • acknowledged boolean Required

      For a successful response, this value is always true. On failure, an exception is returned instead.

    • _shards object

      Additional properties are allowed.

      Hide _shards attributes Show _shards attributes object
POST /{index}/_mapping
curl \
 -X POST http://api.example.com/{index}/_mapping \
 -H "Content-Type: application/json" \
 -d '{"properties":{"user":{"properties":{"name":{"type":"keyword"}}}}}'
Request example
The update mapping API can be applied to multiple data streams or indices with a single request. For example, run `PUT /my-index-000001,my-index-000002/_mapping` to update mappings for the `my-index-000001` and `my-index-000002` indices at the same time.
{
  "properties": {
    "user": {
      "properties": {
        "name": {
          "type": "keyword"
        }
      }
    }
  }
}
Response examples (200)
{
  "acknowledged": true,
  "_shards": {
    "failed": 42.0,
    "successful": 42.0,
    "total": 42.0,
    "failures": [
      {
        "index": "string",
        "node": "string",
        "reason": {
          "type": "string",
          "reason": "string",
          "stack_trace": "string",
          "caused_by": {},
          "root_cause": [
            {}
          ],
          "suppressed": [
            {}
          ]
        },
        "shard": 42.0,
        "status": "string"
      }
    ],
    "skipped": 42.0
  }
}
















































Refresh an index

GET /{index}/_refresh

A refresh makes recent operations performed on one or more indices available for search. For data streams, the API runs the refresh operation on the stream’s backing indices.

By default, Elasticsearch periodically refreshes indices every second, but only on indices that have received one search request or more in the last 30 seconds. You can change this default interval with the index.refresh_interval setting.

Refresh requests are synchronous and do not return a response until the refresh operation completes.

Refreshes are resource-intensive. To ensure good cluster performance, it's recommended to wait for Elasticsearch's periodic refresh rather than performing an explicit refresh when possible.

If your application workflow indexes documents and then runs a search to retrieve the indexed document, it's recommended to use the index API's refresh=wait_for query parameter option. This option ensures the indexing operation waits for a periodic refresh before running the search.

Path parameters

  • index string | array[string] Required

    Comma-separated list of data streams, indices, and aliases used to limit the request. Supports wildcards (*). To target all data streams and indices, omit this parameter or use * or _all.

Query parameters

  • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices.

  • expand_wildcards string | array[string]

    Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as open,hidden. Valid values are: all, open, closed, hidden, none.

  • If false, the request returns an error if it targets a missing or closed index.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • _shards object

      Additional properties are allowed.

      Hide _shards attributes Show _shards attributes object
      • failed number Required
      • successful number Required
      • total number Required
      • failures array[object]
        Hide failures attributes Show failures attributes object
        • index string
        • node string
        • reason object Required

          Additional properties are allowed.

          Hide reason attributes Show reason attributes object
          • type string Required

            The type of error

          • reason string

            A human-readable explanation of the error, in English.

          • The server stack trace. Present only if the error_trace=true parameter was sent with the request.

          • Additional properties are allowed.

          • root_cause array[object]

            Additional properties are allowed.

          • suppressed array[object]

            Additional properties are allowed.

        • shard number Required
        • status string
      • skipped number
GET /{index}/_refresh
curl \
 -X GET http://api.example.com/{index}/_refresh
Response examples (200)
{
  "_shards": {
    "failed": 42.0,
    "successful": 42.0,
    "total": 42.0,
    "failures": [
      {
        "index": "string",
        "node": "string",
        "reason": {
          "type": "string",
          "reason": "string",
          "stack_trace": "string",
          "caused_by": {},
          "root_cause": [
            {}
          ],
          "suppressed": [
            {}
          ]
        },
        "shard": 42.0,
        "status": "string"
      }
    ],
    "skipped": 42.0
  }
}




































Get index shard stores

GET /_shard_stores

Get store information about replica shards in one or more indices. For data streams, the API retrieves store information for the stream's backing indices.

The index shard stores API returns the following information:

  • The node on which each replica shard exists.
  • The allocation ID for each replica shard.
  • A unique ID for each replica shard.
  • Any errors encountered while opening the shard index or from an earlier failure.

By default, the API returns store information only for primary shards that are unassigned or have one or more unassigned replica shards.

Query parameters

  • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices.

  • expand_wildcards string | array[string]

    Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams.

  • If true, missing or closed indices are not included in the response.

  • status string | array[string]

    List of shard health statuses used to limit the request.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • indices object Required
      Hide indices attribute Show indices attribute object
      • * object Additional properties

        Additional properties are allowed.

        Hide * attribute Show * attribute object
        • shards object Required
          Hide shards attribute Show shards attribute object
          • * object Additional properties

            Additional properties are allowed.

            Hide * attribute Show * attribute object
            • stores array[object] Required

              Additional properties are allowed.

GET /_shard_stores
curl \
 -X GET http://api.example.com/_shard_stores
Response examples (200)
{
  "indices": {
    "additionalProperty1": {
      "shards": {
        "additionalProperty1": {
          "stores": [
            {}
          ]
        },
        "additionalProperty2": {
          "stores": [
            {}
          ]
        }
      }
    },
    "additionalProperty2": {
      "shards": {
        "additionalProperty1": {
          "stores": [
            {}
          ]
        },
        "additionalProperty2": {
          "stores": [
            {}
          ]
        }
      }
    }
  }
}




Shrink an index Added in 5.0.0

PUT /{index}/_shrink/{target}

Shrink an index into a new index with fewer primary shards.

Before you can shrink an index:

  • The index must be read-only.
  • A copy of every shard in the index must reside on the same node.
  • The index must have a green health status.

To make shard allocation easier, we recommend you also remove the index's replica shards. You can later re-add replica shards as part of the shrink operation.

The requested number of primary shards in the target index must be a factor of the number of shards in the source index. For example an index with 8 primary shards can be shrunk into 4, 2 or 1 primary shards or an index with 15 primary shards can be shrunk into 5, 3 or 1. If the number of shards in the index is a prime number it can only be shrunk into a single primary shard Before shrinking, a (primary or replica) copy of every shard in the index must be present on the same node.

The current write index on a data stream cannot be shrunk. In order to shrink the current write index, the data stream must first be rolled over so that a new write index is created and then the previous write index can be shrunk.

A shrink operation:

  • Creates a new target index with the same definition as the source index, but with a smaller number of primary shards.
  • Hard-links segments from the source index into the target index. If the file system does not support hard-linking, then all segments are copied into the new index, which is a much more time consuming process. Also if using multiple data paths, shards on different data paths require a full copy of segment files if they are not on the same disk since hardlinks do not work across disks.
  • Recovers the target index as though it were a closed index which had just been re-opened. Recovers shards to the .routing.allocation.initial_recovery._id index setting.

IMPORTANT: Indices can only be shrunk if they satisfy the following requirements:

  • The target index must not exist.
  • The source index must have more primary shards than the target index.
  • The number of primary shards in the target index must be a factor of the number of primary shards in the source index. The source index must have more primary shards than the target index.
  • The index must not contain more than 2,147,483,519 documents in total across all shards that will be shrunk into a single shard on the target index as this is the maximum number of docs that can fit into a single shard.
  • The node handling the shrink process must have sufficient free disk space to accommodate a second copy of the existing index.

Path parameters

  • index string Required

    Name of the source index to shrink.

  • target string Required

    Name of the target index to create.

Query parameters

  • Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

  • timeout string

    Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

  • wait_for_active_shards number | string

    The number of shard copies that must be active before proceeding with the operation. Set to all or any positive integer up to the total number of shards in the index (number_of_replicas+1).

application/json

Body

  • aliases object

    The key is the alias name. Index alias names support date math.

    Hide aliases attribute Show aliases attribute object
  • settings object

    Configuration options for the target index.

    Hide settings attribute Show settings attribute object
    • * object Additional properties

      Additional properties are allowed.

Responses

PUT /{index}/_shrink/{target}
curl \
 -X PUT http://api.example.com/{index}/_shrink/{target} \
 -H "Content-Type: application/json" \
 -d '"{\n  \"settings\": {\n    \"index.routing.allocation.require._name\": null,\n    \"index.blocks.write\": null\n  }\n}"'
Request example
{
  "settings": {
    "index.routing.allocation.require._name": null,
    "index.blocks.write": null
  }
}
Response examples (200)
{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "string"
}




Simulate an index Added in 7.9.0

POST /_index_template/_simulate_index/{name}

Get the index configuration that would be applied to the specified index from an existing index template.

Path parameters

  • name string Required

    Name of the index to simulate

Query parameters

  • Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

  • If true, returns all relevant default configurations for the index template.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • overlapping array[object]
      Hide overlapping attributes Show overlapping attributes object
    • template object Required

      Additional properties are allowed.

      Hide template attributes Show template attributes object
      • aliases object Required
        Hide aliases attribute Show aliases attribute object
      • mappings object Required

        Additional properties are allowed.

        Hide mappings attributes Show mappings attributes object
      • settings object Required Additional properties

        Additional properties are allowed.

        Hide settings attributes Show settings attributes object
        • index object Additional properties

          Additional properties are allowed.

        • mode string
        • Additional properties are allowed.

          Hide soft_deletes attributes Show soft_deletes attributes object
          • enabled boolean

            Indicates whether soft deletes are enabled on the index.

          • Additional properties are allowed.

            Hide retention_lease attribute Show retention_lease attribute object
            • period string Required

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

        • sort object

          Additional properties are allowed.

          Hide sort attributes Show sort attributes object
        • Values are true, false, or checksum.

        • codec string
        • routing_partition_size number | string

          Some APIs will return values such as numbers also as a string (notably epoch timestamps). This behavior is used to capture this behavior while keeping the semantics of the field type.

          Depending on the target language, code generators can keep the union or remove it and leniently parse strings to the target type.

        • merge object

          Additional properties are allowed.

          Hide merge attribute Show merge attribute object
        • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

        • blocks object

          Additional properties are allowed.

          Hide blocks attributes Show blocks attributes object
        • analyze object

          Additional properties are allowed.

          Hide analyze attribute Show analyze attribute object
        • Additional properties are allowed.

          Hide highlight attribute Show highlight attribute object
        • routing object

          Additional properties are allowed.

          Hide routing attributes Show routing attributes object
          • Additional properties are allowed.

            Hide allocation attributes Show allocation attributes object
            • enable string

              Values are all, primaries, new_primaries, or none.

            • include object

              Additional properties are allowed.

              Hide include attributes Show include attributes object
            • Additional properties are allowed.

              Hide initial_recovery attribute Show initial_recovery attribute object
            • disk object

              Additional properties are allowed.

              Hide disk attribute Show disk attribute object
          • Additional properties are allowed.

            Hide rebalance attribute Show rebalance attribute object
            • enable string Required

              Values are all, primaries, replicas, or none.

        • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

        • Additional properties are allowed.

          Hide lifecycle attributes Show lifecycle attributes object
          • name string
          • indexing_complete boolean | string

            Some APIs will return values such as numbers also as a string (notably epoch timestamps). This behavior is used to capture this behavior while keeping the semantics of the field type.

            Depending on the target language, code generators can keep the union or remove it and leniently parse strings to the target type.

          • If specified, this is the timestamp used to calculate the index age for its phase transitions. Use this setting if you create a new index that contains old data and want to use the original creation date to calculate the index age. Specified as a Unix epoch value in milliseconds.

          • Set to true to parse the origination date from the index name. This origination date is used to calculate the index age for its phase transitions. The index name must match the pattern .*-{date_format}-\d+, where the date_format is yyyy.MM.dd and the trailing digits are optional. An index that was rolled over would normally match the full format, for example logs-2016.10.31-000002). If the index name doesn’t match the pattern, index creation fails.

          • step object

            Additional properties are allowed.

            Hide step attribute Show step attribute object
            • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

          • The index alias to update when the index rolls over. Specify when using a policy that contains a rollover action. When the index rolls over, the alias is updated to reflect that the index is no longer the write index. For more information about rolling indices, see Rollover.

        • creation_date number | string

          Some APIs will return values such as numbers also as a string (notably epoch timestamps). This behavior is used to capture this behavior while keeping the semantics of the field type.

          Depending on the target language, code generators can keep the union or remove it and leniently parse strings to the target type.

        • creation_date_string string | number

          A date and time, either as a string whose format can depend on the context (defaulting to ISO 8601), or a number of milliseconds since the Epoch. Elasticsearch accepts both as input, but will generally output a string representation.

        • uuid string
        • version object

          Additional properties are allowed.

          Hide version attributes Show version attributes object
        • translog object

          Additional properties are allowed.

          Hide translog attributes Show translog attributes object
        • Additional properties are allowed.

          Hide query_string attribute Show query_string attribute object
        • analysis object

          Additional properties are allowed.

          Hide analysis attributes Show analysis attributes object
        • settings object Additional properties

          Additional properties are allowed.

        • Additional properties are allowed.

          Hide time_series attributes Show time_series attributes object
        • queries object

          Additional properties are allowed.

          Hide queries attribute Show queries attribute object
          • cache object

            Additional properties are allowed.

            Hide cache attribute Show cache attribute object
        • Configure custom similarity settings to customize how search results are scored.

        • mapping object

          Additional properties are allowed.

          Hide mapping attributes Show mapping attributes object
          • coerce boolean
          • Additional properties are allowed.

            Hide total_fields attributes Show total_fields attributes object
            • limit number | string

              The maximum number of fields in an index. Field and object mappings, as well as field aliases count towards this limit. The limit is in place to prevent mappings and searches from becoming too large. Higher values can lead to performance degradations and memory issues, especially in clusters with a high load or few resources.

            • ignore_dynamic_beyond_limit boolean | string

              This setting determines what happens when a dynamically mapped field would exceed the total fields limit. When set to false (the default), the index request of the document that tries to add a dynamic field to the mapping will fail with the message Limit of total fields [X] has been exceeded. When set to true, the index request will not fail. Instead, fields that would exceed the limit are not added to the mapping, similar to dynamic: false. The fields that were not added to the mapping will be added to the _ignored field.

          • depth object

            Additional properties are allowed.

            Hide depth attribute Show depth attribute object
            • limit number

              The maximum depth for a field, which is measured as the number of inner objects. For instance, if all fields are defined at the root object level, then the depth is 1. If there is one object mapping, then the depth is 2, etc.

          • Additional properties are allowed.

            Hide nested_fields attribute Show nested_fields attribute object
            • limit number

              The maximum number of distinct nested mappings in an index. The nested type should only be used in special cases, when arrays of objects need to be queried independently of each other. To safeguard against poorly designed mappings, this setting limits the number of unique nested types per index.

          • Additional properties are allowed.

            Hide nested_objects attribute Show nested_objects attribute object
            • limit number

              The maximum number of nested JSON objects that a single document can contain across all nested types. This limit helps to prevent out of memory errors when a document contains too many nested objects.

          • Additional properties are allowed.

            Hide field_name_length attribute Show field_name_length attribute object
            • limit number

              Setting for the maximum length of a field name. This setting isn’t really something that addresses mappings explosion but might still be useful if you want to limit the field length. It usually shouldn’t be necessary to set this setting. The default is okay unless a user starts to add a huge number of fields with really long names. Default is Long.MAX_VALUE (no limit).

          • Additional properties are allowed.

            Hide dimension_fields attribute Show dimension_fields attribute object
            • limit number

              [preview] This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.

        • Additional properties are allowed.

          Hide indexing.slowlog attributes Show indexing.slowlog attributes object
          • level string
          • source number
          • reformat boolean
          • Additional properties are allowed.

            Hide threshold attribute Show threshold attribute object
            • index object

              Additional properties are allowed.

              Hide index attributes Show index attributes object
              • warn string

                A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

              • info string

                A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

              • debug string

                A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

              • trace string

                A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

        • Additional properties are allowed.

          Hide indexing_pressure attribute Show indexing_pressure attribute object
          • memory object Required

            Additional properties are allowed.

            Hide memory attribute Show memory attribute object
            • limit number

              Number of outstanding bytes that may be consumed by indexing requests. When this limit is reached or exceeded, the node will reject new coordinating and primary operations. When replica operations consume 1.5x this limit, the node will reject new replica operations. Defaults to 10% of the heap.

        • store object

          Additional properties are allowed.

          Hide store attributes Show store attributes object
          • type string Required

          • allow_mmap boolean

            You can restrict the use of the mmapfs and the related hybridfs store type via the setting node.store.allow_mmap. This is a boolean setting indicating whether or not memory-mapping is allowed. The default is to allow it. This setting is useful, for example, if you are in an environment where you can not control the ability to create a lot of memory maps so you need disable the ability to use memory-mapping.

POST /_index_template/_simulate_index/{name}
curl \
 -X POST http://api.example.com/_index_template/_simulate_index/{name}
Response examples (200)
{
  "overlapping": [
    {
      "name": "string",
      "index_patterns": [
        "string"
      ]
    }
  ],
  "template": {
    "aliases": {
      "additionalProperty1": {
        "filter": {},
        "index_routing": "string",
        "is_hidden": true,
        "is_write_index": true,
        "routing": "string",
        "search_routing": "string"
      },
      "additionalProperty2": {
        "filter": {},
        "index_routing": "string",
        "is_hidden": true,
        "is_write_index": true,
        "routing": "string",
        "search_routing": "string"
      }
    },
    "mappings": {
      "all_field": {
        "analyzer": "string",
        "enabled": true,
        "omit_norms": true,
        "search_analyzer": "string",
        "similarity": "string",
        "store": true,
        "store_term_vector_offsets": true,
        "store_term_vector_payloads": true,
        "store_term_vector_positions": true,
        "store_term_vectors": true
      },
      "date_detection": true,
      "dynamic": "strict",
      "dynamic_date_formats": [
        "string"
      ],
      "dynamic_templates": [
        {}
      ],
      "_field_names": {
        "enabled": true
      },
      "index_field": {
        "enabled": true
      },
      "_meta": {
        "additionalProperty1": {},
        "additionalProperty2": {}
      },
      "numeric_detection": true,
      "properties": {},
      "_routing": {
        "required": true
      },
      "_size": {
        "enabled": true
      },
      "_source": {
        "compress": true,
        "compress_threshold": "string",
        "enabled": true,
        "excludes": [
          "string"
        ],
        "includes": [
          "string"
        ],
        "mode": "disabled"
      },
      "runtime": {
        "additionalProperty1": {
          "fields": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "fetch_fields": [
            {}
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {},
            "options": {}
          },
          "type": "boolean"
        },
        "additionalProperty2": {
          "fields": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "fetch_fields": [
            {}
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {},
            "options": {}
          },
          "type": "boolean"
        }
      },
      "enabled": true,
      "subobjects": true,
      "_data_stream_timestamp": {
        "enabled": true
      }
    },
    "additionalProperty1": {
      "additionalProperty1": {},
      "additionalProperty2": {},
      "mode": "string",
      "routing_path": "string",
      "soft_deletes": {
        "enabled": true,
        "retention_lease": {
          "period": "string"
        }
      },
      "sort": {
        "field": "string",
        "order": "asc",
        "mode": "min",
        "missing": "_last"
      },
      "number_of_shards": 42.0,
      "number_of_replicas": 42.0,
      "number_of_routing_shards": 42.0,
      "check_on_startup": "true",
      "codec": "string",
      "": "string",
      "load_fixed_bitset_filters_eagerly": true,
      "hidden": true,
      "auto_expand_replicas": "string",
      "merge": {
        "scheduler": {
          "": 42.0
        }
      },
      "search": {
        "idle": {
          "after": "string"
        },
        "slowlog": {
          "level": "string",
          "source": 42.0,
          "reformat": true,
          "threshold": {
            "query": {},
            "fetch": {}
          }
        }
      },
      "refresh_interval": "string",
      "max_result_window": 42.0,
      "max_inner_result_window": 42.0,
      "max_rescore_window": 42.0,
      "max_docvalue_fields_search": 42.0,
      "max_script_fields": 42.0,
      "max_ngram_diff": 42.0,
      "max_shingle_diff": 42.0,
      "blocks": {
        "": true
      },
      "max_refresh_listeners": 42.0,
      "analyze": {
        "": 42.0
      },
      "highlight": {
        "max_analyzed_offset": 42.0
      },
      "max_terms_count": 42.0,
      "max_regex_length": 42.0,
      "routing": {
        "allocation": {
          "enable": "all",
          "include": {
            "_tier_preference": "string",
            "_id": "string"
          },
          "initial_recovery": {
            "_id": "string"
          },
          "disk": {}
        },
        "rebalance": {
          "enable": "all"
        }
      },
      "gc_deletes": "string",
      "default_pipeline": "string",
      "final_pipeline": "string",
      "lifecycle": {
        "name": "string",
        "": true,
        "origination_date": 42.0,
        "parse_origination_date": true,
        "step": {
          "wait_time_threshold": "string"
        },
        "rollover_alias": "string"
      },
      "provided_name": "string",
      "uuid": "string",
      "version": {
        "created": "string",
        "created_string": "string"
      },
      "verified_before_close": true,
      "format": "string",
      "max_slices_per_scroll": 42.0,
      "translog": {
        "sync_interval": "string",
        "durability": "request",
        "": 42.0,
        "retention": {
          "": 42.0,
          "age": "string"
        }
      },
      "query_string": {
        "": true
      },
      "priority": 42.0,
      "top_metrics_max_size": 42.0,
      "analysis": {
        "analyzer": {},
        "char_filter": {},
        "filter": {},
        "normalizer": {},
        "tokenizer": {}
      },
      "time_series": {
        "": "string"
      },
      "queries": {
        "cache": {
          "enabled": true
        }
      },
      "similarity": {},
      "mapping": {
        "coerce": true,
        "total_fields": {
          "limit": 42.0,
          "ignore_dynamic_beyond_limit": true
        },
        "depth": {
          "limit": 42.0
        },
        "nested_fields": {
          "limit": 42.0
        },
        "nested_objects": {
          "limit": 42.0
        },
        "field_name_length": {
          "limit": 42.0
        },
        "dimension_fields": {
          "limit": 42.0
        },
        "ignore_malformed": true
      },
      "indexing.slowlog": {
        "level": "string",
        "source": 42.0,
        "reformat": true,
        "threshold": {
          "index": {
            "warn": "string",
            "info": "string",
            "debug": "string",
            "trace": "string"
          }
        }
      },
      "indexing_pressure": {
        "memory": {
          "limit": 42.0
        }
      },
      "store": {
        "": "fs",
        "allow_mmap": true
      }
    },
    "additionalProperty2": {
      "additionalProperty1": {},
      "additionalProperty2": {},
      "mode": "string",
      "routing_path": "string",
      "soft_deletes": {
        "enabled": true,
        "retention_lease": {
          "period": "string"
        }
      },
      "sort": {
        "field": "string",
        "order": "asc",
        "mode": "min",
        "missing": "_last"
      },
      "number_of_shards": 42.0,
      "number_of_replicas": 42.0,
      "number_of_routing_shards": 42.0,
      "check_on_startup": "true",
      "codec": "string",
      "": "string",
      "load_fixed_bitset_filters_eagerly": true,
      "hidden": true,
      "auto_expand_replicas": "string",
      "merge": {
        "scheduler": {
          "": 42.0
        }
      },
      "search": {
        "idle": {
          "after": "string"
        },
        "slowlog": {
          "level": "string",
          "source": 42.0,
          "reformat": true,
          "threshold": {
            "query": {},
            "fetch": {}
          }
        }
      },
      "refresh_interval": "string",
      "max_result_window": 42.0,
      "max_inner_result_window": 42.0,
      "max_rescore_window": 42.0,
      "max_docvalue_fields_search": 42.0,
      "max_script_fields": 42.0,
      "max_ngram_diff": 42.0,
      "max_shingle_diff": 42.0,
      "blocks": {
        "": true
      },
      "max_refresh_listeners": 42.0,
      "analyze": {
        "": 42.0
      },
      "highlight": {
        "max_analyzed_offset": 42.0
      },
      "max_terms_count": 42.0,
      "max_regex_length": 42.0,
      "routing": {
        "allocation": {
          "enable": "all",
          "include": {
            "_tier_preference": "string",
            "_id": "string"
          },
          "initial_recovery": {
            "_id": "string"
          },
          "disk": {}
        },
        "rebalance": {
          "enable": "all"
        }
      },
      "gc_deletes": "string",
      "default_pipeline": "string",
      "final_pipeline": "string",
      "lifecycle": {
        "name": "string",
        "": true,
        "origination_date": 42.0,
        "parse_origination_date": true,
        "step": {
          "wait_time_threshold": "string"
        },
        "rollover_alias": "string"
      },
      "provided_name": "string",
      "uuid": "string",
      "version": {
        "created": "string",
        "created_string": "string"
      },
      "verified_before_close": true,
      "format": "string",
      "max_slices_per_scroll": 42.0,
      "translog": {
        "sync_interval": "string",
        "durability": "request",
        "": 42.0,
        "retention": {
          "": 42.0,
          "age": "string"
        }
      },
      "query_string": {
        "": true
      },
      "priority": 42.0,
      "top_metrics_max_size": 42.0,
      "analysis": {
        "analyzer": {},
        "char_filter": {},
        "filter": {},
        "normalizer": {},
        "tokenizer": {}
      },
      "time_series": {
        "": "string"
      },
      "queries": {
        "cache": {
          "enabled": true
        }
      },
      "similarity": {},
      "mapping": {
        "coerce": true,
        "total_fields": {
          "limit": 42.0,
          "ignore_dynamic_beyond_limit": true
        },
        "depth": {
          "limit": 42.0
        },
        "nested_fields": {
          "limit": 42.0
        },
        "nested_objects": {
          "limit": 42.0
        },
        "field_name_length": {
          "limit": 42.0
        },
        "dimension_fields": {
          "limit": 42.0
        },
        "ignore_malformed": true
      },
      "indexing.slowlog": {
        "level": "string",
        "source": 42.0,
        "reformat": true,
        "threshold": {
          "index": {
            "warn": "string",
            "info": "string",
            "debug": "string",
            "trace": "string"
          }
        }
      },
      "indexing_pressure": {
        "memory": {
          "limit": 42.0
        }
      },
      "store": {
        "": "fs",
        "allow_mmap": true
      }
    }
  }
}




















































Validate a query Added in 1.3.0

POST /{index}/_validate/query

Validates a query without running it.

Path parameters

  • index string | array[string] Required

    Comma-separated list of data streams, indices, and aliases to search. Supports wildcards (*). To search all data streams or indices, omit this parameter or use * or _all.

Query parameters

  • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices.

  • all_shards boolean

    If true, the validation is executed on all shards instead of one random shard per index.

  • analyzer string

    Analyzer to use for the query string. This parameter can only be used when the q query string parameter is specified.

  • If true, wildcard and prefix queries are analyzed.

  • The default operator for query string query: AND or OR.

    Values are and, AND, or, or OR.

  • df string

    Field to use as default where no field prefix is given in the query string. This parameter can only be used when the q query string parameter is specified.

  • expand_wildcards string | array[string]

    Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as open,hidden. Valid values are: all, open, closed, hidden, none.

  • explain boolean

    If true, the response returns detailed information if an error has occurred.

  • If false, the request returns an error if it targets a missing or closed index.

  • lenient boolean

    If true, format-based query failures (such as providing text to a numeric field) in the query string will be ignored.

  • rewrite boolean

    If true, returns a more detailed explanation showing the actual Lucene query that will be executed.

  • q string

    Query in the Lucene query string syntax.

application/json

Body

  • query object

    An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

    Additional properties are allowed.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
POST /{index}/_validate/query
curl \
 -X POST http://api.example.com/{index}/_validate/query \
 -H "Content-Type: application/json" \
 -d '{"query":{}}'
Request examples
{
  "query": {}
}
Response examples (200)
{
  "explanations": [
    {
      "error": "string",
      "explanation": "string",
      "index": "string",
      "valid": true
    }
  ],
  "_shards": {
    "failed": 42.0,
    "successful": 42.0,
    "total": 42.0,
    "failures": [
      {
        "index": "string",
        "node": "string",
        "reason": {
          "type": "string",
          "reason": "string",
          "stack_trace": "string",
          "caused_by": {},
          "root_cause": [
            {}
          ],
          "suppressed": [
            {}
          ]
        },
        "shard": 42.0,
        "status": "string"
      }
    ],
    "skipped": 42.0
  },
  "valid": true,
  "error": "string"
}






















































Create an inference endpoint Added in 8.11.0

PUT /_inference/{inference_id}

When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running. After creating the endpoint, wait for the model deployment to complete before using it. To verify the deployment status, use the get trained model statistics API. Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count". Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.

IMPORTANT: The inference APIs enable you to use certain services, such as built-in machine learning models (ELSER, E5), models uploaded through Eland, Cohere, OpenAI, Mistral, Azure OpenAI, Google AI Studio, Google Vertex AI, Anthropic, Watsonx.ai, or Hugging Face. For built-in models and models uploaded through Eland, the inference APIs offer an alternative way to use and manage trained models. However, if you do not plan to use the inference APIs to use these models or if you want to use non-NLP models, use the machine learning trained model APIs.

Path parameters

application/json

Body Required

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
PUT /_inference/{inference_id}
curl \
 -X PUT http://api.example.com/_inference/{inference_id} \
 -H "Content-Type: application/json" \
 -d '{"service":"string","service_settings":{},"task_settings":{}}'
Request examples
{
  "service": "string",
  "service_settings": {},
  "task_settings": {}
}
Response examples (200)
{
  "service": "string",
  "service_settings": {},
  "task_settings": {},
  "inference_id": "string",
  "task_type": "sparse_embedding"
}


















































































Delete pipelines Added in 5.0.0

DELETE /_ingest/pipeline/{id}

Delete one or more ingest pipelines.

Path parameters

  • id string Required

    Pipeline ID or wildcard expression of pipeline IDs used to limit the request. To delete all ingest pipelines in a cluster, use a value of *.

Query parameters

  • Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

  • timeout string

    Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • acknowledged boolean Required

      For a successful response, this value is always true. On failure, an exception is returned instead.

DELETE /_ingest/pipeline/{id}
curl \
 -X DELETE http://api.example.com/_ingest/pipeline/{id}
Response examples (200)
{
  "acknowledged": true
}
































Simulate a pipeline Added in 5.0.0

POST /_ingest/pipeline/{id}/_simulate

Run an ingest pipeline against a set of provided documents. You can either specify an existing pipeline to use with the provided documents or supply a pipeline definition in the body of the request.

Path parameters

  • id string Required

    Pipeline to test. If you don’t specify a pipeline in the request body, this parameter is required.

Query parameters

  • verbose boolean

    If true, the response includes output data for each processor in the executed pipeline.

application/json

Body Required

  • docs array[object] Required

    Sample documents to test in the pipeline.

    Hide docs attributes Show docs attributes object
    • _id string
    • _index string
    • _source object Required

      JSON body for the document.

      Additional properties are allowed.

  • pipeline object Additional properties

    Additional properties are allowed.

    Hide pipeline attributes Show pipeline attributes object
    • Description of the ingest pipeline.

    • on_failure array[object]

      Processors to run immediately after a processor failure.

      Hide on_failure attributes Show on_failure attributes object
      • append object

        Additional properties are allowed.

        Hide append attributes Show append attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • value object | array[object] Required

          The value to be appended. Supports template snippets.

          One of:

          Additional properties are allowed.

        • If false, the processor does not append values already present in the field.

      • Additional properties are allowed.

        Hide attachment attributes Show attachment attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • The number of chars being used for extraction to prevent huge fields. Use -1 for no limit.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • properties array[string]

          Array of properties to select to be stored. Can be content, title, name, author, keywords, date, content_type, content_length, language.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true, the binary field will be removed from the document

        • Field containing the name of the resource to decode. If specified, the processor passes this resource name to the underlying Tika library to enable Resource Name Based Detection.

      • bytes object

        Additional properties are allowed.

        Hide bytes attributes Show bytes attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • circle object

        Additional properties are allowed.

        Hide circle attributes Show circle attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • error_distance number Required

          The difference between the resulting inscribed distance from center to side and the circle’s radius (measured in meters for geo_shape, unit-less for shape).

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • shape_type string Required

          Values are geo_shape or shape.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Additional properties are allowed.

        Hide community_id attributes Show community_id attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • seed number

          Seed for the community ID hash. Must be between 0 and 65535 (inclusive). The seed can prevent hash collisions between network domains, such as a staging and production network that use the same addressing scheme.

        • If true and any required fields are missing, the processor quietly exits without modifying the document.

      • convert object

        Additional properties are allowed.

        Hide convert attributes Show convert attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • type string Required

          Values are integer, long, double, float, boolean, ip, string, or auto.

      • csv object

        Additional properties are allowed.

        Hide csv attributes Show csv attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • Value used to fill empty fields. Empty fields are skipped if this is not provided. An empty field is one with no value (2 consecutive separators) or empty quotes ("").

          Additional properties are allowed.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • quote string

          Quote used in CSV, has to be single character string.

        • Separator used in CSV, has to be single character string.

        • target_fields string | array[string] Required
        • trim boolean

          Trim whitespaces in unquoted fields.

      • date object

        Additional properties are allowed.

        Hide date attributes Show date attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • formats array[string] Required

          An array of the expected date formats. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

        • locale string

          The locale to use when parsing the date, relevant when parsing month names or week days. Supports template snippets.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • timezone string

          The timezone to use when parsing the date. Supports template snippets.

        • The format to use when writing the date to target_field. Must be a valid java time pattern.

      • Additional properties are allowed.

        Hide date_index_name attributes Show date_index_name attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • date_formats array[string] Required

          An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

        • date_rounding string Required

          How to round the date when formatting the date into the index name. Valid values are: y (year), M (month), w (week), d (day), h (hour), m (minute) and s (second). Supports template snippets.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • The format to be used when printing the parsed date into the index name. A valid java time pattern is expected here. Supports template snippets.

        • A prefix of the index name to be prepended before the printed date. Supports template snippets.

        • locale string

          The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.

        • timezone string

          The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.

      • dissect object

        Additional properties are allowed.

        Hide dissect attributes Show dissect attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • The character(s) that separate the appended fields.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • pattern string Required

          The pattern to apply to the field.

      • Additional properties are allowed.

        Hide dot_expander attributes Show dot_expander attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • override boolean

          Controls the behavior when there is already an existing nested object that conflicts with the expanded field. When false, the processor will merge conflicts by combining the old and the new values into an array. When true, the value from the expanded field will overwrite the existing value.

        • path string

          The field that contains the field to expand. Only required if the field to expand is part another object field, because the field option can only understand leaf fields.

      • drop object

        Additional properties are allowed.

        Hide drop attributes Show drop attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

      • enrich object

        Additional properties are allowed.

        Hide enrich attributes Show enrich attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • The maximum number of matched documents to include under the configured target field. The target_field will be turned into a json array if max_matches is higher than 1, otherwise target_field will become a json object. In order to avoid documents getting too large, the maximum allowed value is 128.

        • override boolean

          If processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

        • policy_name string Required

          The name of the enrich policy to use.

        • Values are intersects, disjoint, within, or contains.

        • target_field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • fail object

        Additional properties are allowed.

        Hide fail attributes Show fail attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • message string Required

          The error message thrown by the processor. Supports template snippets.

      • Additional properties are allowed.

        Hide fingerprint attributes Show fingerprint attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • fields string | array[string] Required
        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • salt string

          Salt value for the hash function.

        • method string

          Values are MD5, SHA-1, SHA-256, SHA-512, or MurmurHash3.

        • If true, the processor ignores any missing fields. If all fields are missing, the processor silently exits without modifying the document.

      • foreach object

        Additional properties are allowed.

        Hide foreach attributes Show foreach attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true, the processor silently exits without changing the document if the field is null or missing.

        • processor object Required

          Additional properties are allowed.

      • Additional properties are allowed.

        Hide ip_location attributes Show ip_location attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • first_only boolean

          If true, only the first found IP location data will be returned, even if the field contains an array.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • properties array[string]

          Controls what properties are added to the target_field based on the IP location lookup.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

      • geo_grid object

        Additional properties are allowed.

        Hide geo_grid attributes Show geo_grid attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          The field to interpret as a geo-tile.= The field format is determined by the tile_type.

        • tile_type string Required

          Values are geotile, geohex, or geohash.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • Values are geojson or wkt.

      • geoip object

        Additional properties are allowed.

        Hide geoip attributes Show geoip attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • first_only boolean

          If true, only the first found geoip data will be returned, even if the field contains an array.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • properties array[string]

          Controls what properties are added to the target_field based on the geoip lookup.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

      • grok object

        Additional properties are allowed.

        Hide grok attributes Show grok attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • Must be disabled or v1. If v1, the processor uses patterns with Elastic Common Schema (ECS) field names.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • A map of pattern-name and pattern tuples defining custom patterns to be used by the current processor. Patterns matching existing names will override the pre-existing definition.

          Hide pattern_definitions attribute Show pattern_definitions attribute object
          • * string Additional properties
        • patterns array[string] Required

          An ordered list of grok expression to match and extract named captures with. Returns on the first expression in the list that matches.

        • When true, _ingest._grok_match_index will be inserted into your matched document’s metadata with the index into the pattern found in patterns that matched.

      • gsub object

        Additional properties are allowed.

        Hide gsub attributes Show gsub attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • pattern string Required

          The pattern to be replaced.

        • replacement string Required

          The string to replace the matching patterns with.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Additional properties are allowed.

        Hide html_strip attributes Show html_strip attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist or is null, the processor quietly exits without modifying the document,

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Additional properties are allowed.

        Hide inference attributes Show inference attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • model_id string Required
        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Maps the document field names to the known field names of the model. This mapping takes precedence over any default mappings provided in the model configuration.

          Hide field_map attribute Show field_map attribute object
          • * object Additional properties

            Additional properties are allowed.

        • Additional properties are allowed.

          Hide inference_config attributes Show inference_config attributes object
      • join object

        Additional properties are allowed.

        Hide join attributes Show join attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • separator string Required

          The separator character.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • json object

        Additional properties are allowed.

        Hide json attributes Show json attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • Flag that forces the parsed JSON to be added at the top level of the document. target_field must not be set when this option is chosen.

        • Values are replace or merge.

        • When set to true, the JSON parser will not fail if the JSON contains duplicate keys. Instead, the last encountered value for any duplicate key wins.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • kv object

        Additional properties are allowed.

        Hide kv attributes Show kv attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • exclude_keys array[string]

          List of keys to exclude from document.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • field_split string Required

          Regex pattern to use for splitting key-value pairs.

        • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • include_keys array[string]

          List of keys to filter and insert into document. Defaults to including all keys.

        • prefix string

          Prefix to be added to extracted keys.

        • If true. strip brackets (), <>, [] as well as quotes ' and " from extracted values.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • trim_key string

          String of characters to trim from extracted keys.

        • String of characters to trim from extracted values.

        • value_split string Required

          Regex pattern to use for splitting the key from the value within a key-value pair.

      • Additional properties are allowed.

        Hide lowercase attributes Show lowercase attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Additional properties are allowed.

        Hide network_direction attributes Show network_direction attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • internal_networks array[string]

          List of internal networks. Supports IPv4 and IPv6 addresses and ranges in CIDR notation. Also supports the named ranges listed below. These may be constructed with template snippets. Must specify only one of internal_networks or internal_networks_field.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and any required fields are missing, the processor quietly exits without modifying the document.

      • pipeline object

        Additional properties are allowed.

        Hide pipeline attributes Show pipeline attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • name string Required
        • Whether to ignore missing pipelines instead of failing.

      • redact object

        Additional properties are allowed.

        Hide redact attributes Show redact attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • patterns array[string] Required

          A list of grok expressions to match and redact named captures with

        • Hide pattern_definitions attribute Show pattern_definitions attribute object
          • * string Additional properties
        • prefix string

          Start a redacted section with this token

        • suffix string

          End a redacted section with this token

        • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • If true and the current license does not support running redact processors, then the processor quietly exits without modifying the document

        • If true then ingest metadata _ingest._redact._is_redacted is set to true if the document has been redacted

      • Additional properties are allowed.

        Hide registered_domain attributes Show registered_domain attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and any required fields are missing, the processor quietly exits without modifying the document.

      • remove object

        Additional properties are allowed.

        Hide remove attributes Show remove attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string | array[string] Required
        • keep string | array[string]
        • If true and field does not exist or is null, the processor quietly exits without modifying the document.

      • rename object

        Additional properties are allowed.

        Hide rename attributes Show rename attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • target_field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • reroute object

        Additional properties are allowed.

        Hide reroute attributes Show reroute attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • A static value for the target. Can’t be set when the dataset or namespace option is set.

        • dataset string | array[string]

          Field references or a static value for the dataset part of the data stream name. In addition to the criteria for index names, cannot contain - and must be no longer than 100 characters. Example values are nginx.access and nginx.error.

          Supports field references with a mustache-like syntax (denoted as {{double}} or {{{triple}}} curly braces). When resolving field references, the processor replaces invalid characters with _. Uses the part of the index name as a fallback if all field references resolve to a null, missing, or non-string value.

          default {{data_stream.dataset}}

        • namespace string | array[string]

          Field references or a static value for the namespace part of the data stream name. See the criteria for index names for allowed characters. Must be no longer than 100 characters.

          Supports field references with a mustache-like syntax (denoted as {{double}} or {{{triple}}} curly braces). When resolving field references, the processor replaces invalid characters with _. Uses the part of the index name as a fallback if all field references resolve to a null, missing, or non-string value.

          default {{data_stream.namespace}}

      • script object

        Additional properties are allowed.

        Hide script attributes Show script attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • id string
        • lang string

          Script language.

        • params object

          Object containing parameters for the script.

          Hide params attribute Show params attribute object
          • * object Additional properties

            Additional properties are allowed.

        • source string

          Inline script. If no id is specified, this parameter is required.

      • set object

        Additional properties are allowed.

        Hide set attributes Show set attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and value is a template snippet that evaluates to null or the empty string, the processor quietly exits without modifying the document.

        • The media type for encoding value. Applies only when value is a template snippet. Must be one of application/json, text/plain, or application/x-www-form-urlencoded.

        • override boolean

          If true processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

        • value object

          The value to be set for the field. Supports template snippets. May specify only one of value or copy_from.

          Additional properties are allowed.

      • Additional properties are allowed.

        Hide set_security_user attributes Show set_security_user attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • properties array[string]

          Controls what user related properties are added to the field.

      • sort object

        Additional properties are allowed.

        Hide sort attributes Show sort attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • order string

          Values are asc or desc.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • split object

        Additional properties are allowed.

        Hide split attributes Show split attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • Preserves empty trailing fields, if any.

        • separator string Required

          A regex which matches the separator, for example, , or \s+.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Additional properties are allowed.

        Hide terminate attributes Show terminate attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

      • trim object

        Additional properties are allowed.

        Hide trim attributes Show trim attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Additional properties are allowed.

        Hide uppercase attributes Show uppercase attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Additional properties are allowed.

        Hide urldecode attributes Show urldecode attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Additional properties are allowed.

        Hide uri_parts attributes Show uri_parts attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • If true, the processor copies the unparsed URI to <target_field>.original.

        • If true, the processor removes the field after parsing the URI string. If parsing fails, the processor does not remove the field.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Additional properties are allowed.

        Hide user_agent attributes Show user_agent attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • The name of the file in the config/ingest-user-agent directory containing the regular expressions for parsing the user agent string. Both the directory and the file have to be created before starting Elasticsearch. If not specified, ingest-user-agent will use the regexes.yaml from uap-core it ships with.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • properties array[string]

          Controls what properties are added to target_field.

          Values are name, os, device, original, or version.

        • Extracts device type from the user agent string on a best-effort basis.

    • processors array[object]

      Processors used to perform transformations on documents before indexing. Processors run sequentially in the order specified.

      Hide processors attributes Show processors attributes object
      • append object

        Additional properties are allowed.

        Hide append attributes Show append attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • value object | array[object] Required

          The value to be appended. Supports template snippets.

          One of:

          Additional properties are allowed.

        • If false, the processor does not append values already present in the field.

      • Additional properties are allowed.

        Hide attachment attributes Show attachment attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • The number of chars being used for extraction to prevent huge fields. Use -1 for no limit.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • properties array[string]

          Array of properties to select to be stored. Can be content, title, name, author, keywords, date, content_type, content_length, language.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true, the binary field will be removed from the document

        • Field containing the name of the resource to decode. If specified, the processor passes this resource name to the underlying Tika library to enable Resource Name Based Detection.

      • bytes object

        Additional properties are allowed.

        Hide bytes attributes Show bytes attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • circle object

        Additional properties are allowed.

        Hide circle attributes Show circle attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • error_distance number Required

          The difference between the resulting inscribed distance from center to side and the circle’s radius (measured in meters for geo_shape, unit-less for shape).

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • shape_type string Required

          Values are geo_shape or shape.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Additional properties are allowed.

        Hide community_id attributes Show community_id attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • seed number

          Seed for the community ID hash. Must be between 0 and 65535 (inclusive). The seed can prevent hash collisions between network domains, such as a staging and production network that use the same addressing scheme.

        • If true and any required fields are missing, the processor quietly exits without modifying the document.

      • convert object

        Additional properties are allowed.

        Hide convert attributes Show convert attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • type string Required

          Values are integer, long, double, float, boolean, ip, string, or auto.

      • csv object

        Additional properties are allowed.

        Hide csv attributes Show csv attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • Value used to fill empty fields. Empty fields are skipped if this is not provided. An empty field is one with no value (2 consecutive separators) or empty quotes ("").

          Additional properties are allowed.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • quote string

          Quote used in CSV, has to be single character string.

        • Separator used in CSV, has to be single character string.

        • target_fields string | array[string] Required
        • trim boolean

          Trim whitespaces in unquoted fields.

      • date object

        Additional properties are allowed.

        Hide date attributes Show date attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • formats array[string] Required

          An array of the expected date formats. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

        • locale string

          The locale to use when parsing the date, relevant when parsing month names or week days. Supports template snippets.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • timezone string

          The timezone to use when parsing the date. Supports template snippets.

        • The format to use when writing the date to target_field. Must be a valid java time pattern.

      • Additional properties are allowed.

        Hide date_index_name attributes Show date_index_name attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • date_formats array[string] Required

          An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

        • date_rounding string Required

          How to round the date when formatting the date into the index name. Valid values are: y (year), M (month), w (week), d (day), h (hour), m (minute) and s (second). Supports template snippets.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • The format to be used when printing the parsed date into the index name. A valid java time pattern is expected here. Supports template snippets.

        • A prefix of the index name to be prepended before the printed date. Supports template snippets.

        • locale string

          The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.

        • timezone string

          The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.

      • dissect object

        Additional properties are allowed.

        Hide dissect attributes Show dissect attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • The character(s) that separate the appended fields.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • pattern string Required

          The pattern to apply to the field.

      • Additional properties are allowed.

        Hide dot_expander attributes Show dot_expander attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • override boolean

          Controls the behavior when there is already an existing nested object that conflicts with the expanded field. When false, the processor will merge conflicts by combining the old and the new values into an array. When true, the value from the expanded field will overwrite the existing value.

        • path string

          The field that contains the field to expand. Only required if the field to expand is part another object field, because the field option can only understand leaf fields.

      • drop object

        Additional properties are allowed.

        Hide drop attributes Show drop attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

      • enrich object

        Additional properties are allowed.

        Hide enrich attributes Show enrich attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • The maximum number of matched documents to include under the configured target field. The target_field will be turned into a json array if max_matches is higher than 1, otherwise target_field will become a json object. In order to avoid documents getting too large, the maximum allowed value is 128.

        • override boolean

          If processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

        • policy_name string Required

          The name of the enrich policy to use.

        • Values are intersects, disjoint, within, or contains.

        • target_field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • fail object

        Additional properties are allowed.

        Hide fail attributes Show fail attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • message string Required

          The error message thrown by the processor. Supports template snippets.

      • Additional properties are allowed.

        Hide fingerprint attributes Show fingerprint attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • fields string | array[string] Required
        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • salt string

          Salt value for the hash function.

        • method string

          Values are MD5, SHA-1, SHA-256, SHA-512, or MurmurHash3.

        • If true, the processor ignores any missing fields. If all fields are missing, the processor silently exits without modifying the document.

      • foreach object

        Additional properties are allowed.

        Hide foreach attributes Show foreach attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true, the processor silently exits without changing the document if the field is null or missing.

        • processor object Required

          Additional properties are allowed.

      • Additional properties are allowed.

        Hide ip_location attributes Show ip_location attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • first_only boolean

          If true, only the first found IP location data will be returned, even if the field contains an array.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • properties array[string]

          Controls what properties are added to the target_field based on the IP location lookup.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

      • geo_grid object

        Additional properties are allowed.

        Hide geo_grid attributes Show geo_grid attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          The field to interpret as a geo-tile.= The field format is determined by the tile_type.

        • tile_type string Required

          Values are geotile, geohex, or geohash.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • Values are geojson or wkt.

      • geoip object

        Additional properties are allowed.

        Hide geoip attributes Show geoip attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • first_only boolean

          If true, only the first found geoip data will be returned, even if the field contains an array.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • properties array[string]

          Controls what properties are added to the target_field based on the geoip lookup.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

      • grok object

        Additional properties are allowed.

        Hide grok attributes Show grok attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • Must be disabled or v1. If v1, the processor uses patterns with Elastic Common Schema (ECS) field names.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • A map of pattern-name and pattern tuples defining custom patterns to be used by the current processor. Patterns matching existing names will override the pre-existing definition.

          Hide pattern_definitions attribute Show pattern_definitions attribute object
          • * string Additional properties
        • patterns array[string] Required

          An ordered list of grok expression to match and extract named captures with. Returns on the first expression in the list that matches.

        • When true, _ingest._grok_match_index will be inserted into your matched document’s metadata with the index into the pattern found in patterns that matched.

      • gsub object

        Additional properties are allowed.

        Hide gsub attributes Show gsub attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • pattern string Required

          The pattern to be replaced.

        • replacement string Required

          The string to replace the matching patterns with.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Additional properties are allowed.

        Hide html_strip attributes Show html_strip attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist or is null, the processor quietly exits without modifying the document,

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Additional properties are allowed.

        Hide inference attributes Show inference attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • model_id string Required
        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Maps the document field names to the known field names of the model. This mapping takes precedence over any default mappings provided in the model configuration.

          Hide field_map attribute Show field_map attribute object
          • * object Additional properties

            Additional properties are allowed.

        • Additional properties are allowed.

          Hide inference_config attributes Show inference_config attributes object
      • join object

        Additional properties are allowed.

        Hide join attributes Show join attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • separator string Required

          The separator character.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • json object

        Additional properties are allowed.

        Hide json attributes Show json attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • Flag that forces the parsed JSON to be added at the top level of the document. target_field must not be set when this option is chosen.

        • Values are replace or merge.

        • When set to true, the JSON parser will not fail if the JSON contains duplicate keys. Instead, the last encountered value for any duplicate key wins.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • kv object

        Additional properties are allowed.

        Hide kv attributes Show kv attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • exclude_keys array[string]

          List of keys to exclude from document.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • field_split string Required

          Regex pattern to use for splitting key-value pairs.

        • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • include_keys array[string]

          List of keys to filter and insert into document. Defaults to including all keys.

        • prefix string

          Prefix to be added to extracted keys.

        • If true. strip brackets (), <>, [] as well as quotes ' and " from extracted values.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • trim_key string

          String of characters to trim from extracted keys.

        • String of characters to trim from extracted values.

        • value_split string Required

          Regex pattern to use for splitting the key from the value within a key-value pair.

      • Additional properties are allowed.

        Hide lowercase attributes Show lowercase attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Additional properties are allowed.

        Hide network_direction attributes Show network_direction attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • internal_networks array[string]

          List of internal networks. Supports IPv4 and IPv6 addresses and ranges in CIDR notation. Also supports the named ranges listed below. These may be constructed with template snippets. Must specify only one of internal_networks or internal_networks_field.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and any required fields are missing, the processor quietly exits without modifying the document.

      • pipeline object

        Additional properties are allowed.

        Hide pipeline attributes Show pipeline attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • name string Required
        • Whether to ignore missing pipelines instead of failing.

      • redact object

        Additional properties are allowed.

        Hide redact attributes Show redact attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • patterns array[string] Required

          A list of grok expressions to match and redact named captures with

        • Hide pattern_definitions attribute Show pattern_definitions attribute object
          • * string Additional properties
        • prefix string

          Start a redacted section with this token

        • suffix string

          End a redacted section with this token

        • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • If true and the current license does not support running redact processors, then the processor quietly exits without modifying the document

        • If true then ingest metadata _ingest._redact._is_redacted is set to true if the document has been redacted

      • Additional properties are allowed.

        Hide registered_domain attributes Show registered_domain attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and any required fields are missing, the processor quietly exits without modifying the document.

      • remove object

        Additional properties are allowed.

        Hide remove attributes Show remove attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string | array[string] Required
        • keep string | array[string]
        • If true and field does not exist or is null, the processor quietly exits without modifying the document.

      • rename object

        Additional properties are allowed.

        Hide rename attributes Show rename attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • target_field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • reroute object

        Additional properties are allowed.

        Hide reroute attributes Show reroute attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • A static value for the target. Can’t be set when the dataset or namespace option is set.

        • dataset string | array[string]

          Field references or a static value for the dataset part of the data stream name. In addition to the criteria for index names, cannot contain - and must be no longer than 100 characters. Example values are nginx.access and nginx.error.

          Supports field references with a mustache-like syntax (denoted as {{double}} or {{{triple}}} curly braces). When resolving field references, the processor replaces invalid characters with _. Uses the part of the index name as a fallback if all field references resolve to a null, missing, or non-string value.

          default {{data_stream.dataset}}

        • namespace string | array[string]

          Field references or a static value for the namespace part of the data stream name. See the criteria for index names for allowed characters. Must be no longer than 100 characters.

          Supports field references with a mustache-like syntax (denoted as {{double}} or {{{triple}}} curly braces). When resolving field references, the processor replaces invalid characters with _. Uses the part of the index name as a fallback if all field references resolve to a null, missing, or non-string value.

          default {{data_stream.namespace}}

      • script object

        Additional properties are allowed.

        Hide script attributes Show script attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • id string
        • lang string

          Script language.

        • params object

          Object containing parameters for the script.

          Hide params attribute Show params attribute object
          • * object Additional properties

            Additional properties are allowed.

        • source string

          Inline script. If no id is specified, this parameter is required.

      • set object

        Additional properties are allowed.

        Hide set attributes Show set attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and value is a template snippet that evaluates to null or the empty string, the processor quietly exits without modifying the document.

        • The media type for encoding value. Applies only when value is a template snippet. Must be one of application/json, text/plain, or application/x-www-form-urlencoded.

        • override boolean

          If true processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

        • value object

          The value to be set for the field. Supports template snippets. May specify only one of value or copy_from.

          Additional properties are allowed.

      • Additional properties are allowed.

        Hide set_security_user attributes Show set_security_user attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • properties array[string]

          Controls what user related properties are added to the field.

      • sort object

        Additional properties are allowed.

        Hide sort attributes Show sort attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • order string

          Values are asc or desc.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • split object

        Additional properties are allowed.

        Hide split attributes Show split attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • Preserves empty trailing fields, if any.

        • separator string Required

          A regex which matches the separator, for example, , or \s+.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Additional properties are allowed.

        Hide terminate attributes Show terminate attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

      • trim object

        Additional properties are allowed.

        Hide trim attributes Show trim attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Additional properties are allowed.

        Hide uppercase attributes Show uppercase attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Additional properties are allowed.

        Hide urldecode attributes Show urldecode attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Additional properties are allowed.

        Hide uri_parts attributes Show uri_parts attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • If true, the processor copies the unparsed URI to <target_field>.original.

        • If true, the processor removes the field after parsing the URI string. If parsing fails, the processor does not remove the field.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Additional properties are allowed.

        Hide user_agent attributes Show user_agent attributes object
        • Description of the processor. Useful for describing the purpose of the processor or its configuration.

        • if string

          Conditionally execute the processor.

        • Ignore failures for the processor.

        • on_failure array[object]

          Handle failures for the processor.

          Additional properties are allowed.

        • tag string

          Identifier for the processor. Useful for debugging and metrics.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • If true and field does not exist, the processor quietly exits without modifying the document.

        • The name of the file in the config/ingest-user-agent directory containing the regular expressions for parsing the user agent string. Both the directory and the file have to be created before starting Elasticsearch. If not specified, ingest-user-agent will use the regexes.yaml from uap-core it ships with.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • properties array[string]

          Controls what properties are added to target_field.

          Values are name, os, device, original, or version.

        • Extracts device type from the user agent string on a best-effort basis.

    • version number
    • deprecated boolean

      Marks this ingest pipeline as deprecated. When a deprecated ingest pipeline is referenced as the default or final pipeline when creating or updating a non-deprecated index template, Elasticsearch will emit a deprecation warning.

    • _meta object
      Hide _meta attribute Show _meta attribute object
      • * object Additional properties

        Additional properties are allowed.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • docs array[object] Required
      Hide docs attributes Show docs attributes object
      • doc object

        Additional properties are allowed.

        Hide doc attributes Show doc attributes object
        • _id string Required
        • _index string Required
        • _ingest object Required

          Additional properties are allowed.

          Hide _ingest attributes Show _ingest attributes object
        • _routing string

          Value used to send the document to a specific primary shard.

        • _source object Required

          JSON body for the document.

          Hide _source attribute Show _source attribute object
          • * object Additional properties

            Additional properties are allowed.

        • _version number | string

          Some APIs will return values such as numbers also as a string (notably epoch timestamps). This behavior is used to capture this behavior while keeping the semantics of the field type.

          Depending on the target language, code generators can keep the union or remove it and leniently parse strings to the target type.

        • Values are internal, external, external_gte, or force.

      • error object

        Additional properties are allowed.

        Hide error attributes Show error attributes object
        • type string Required

          The type of error

        • reason string

          A human-readable explanation of the error, in English.

        • The server stack trace. Present only if the error_trace=true parameter was sent with the request.

        • Additional properties are allowed.

        • root_cause array[object]

          Additional properties are allowed.

        • suppressed array[object]

          Additional properties are allowed.

      • processor_results array[object]
        Hide processor_results attributes Show processor_results attributes object
        • doc object

          Additional properties are allowed.

          Hide doc attributes Show doc attributes object
        • tag string
        • status string

          Values are success, failure, simulated, or throttled.

        • Additional properties are allowed.

          Hide ignored_error attributes Show ignored_error attributes object
          • type string Required

            The type of error

          • reason string

            A human-readable explanation of the error, in English.

          • The server stack trace. Present only if the error_trace=true parameter was sent with the request.

          • Additional properties are allowed.

          • root_cause array[object]
          • suppressed array[object]
        • error object

          Additional properties are allowed.

          Hide error attributes Show error attributes object
          • type string Required

            The type of error

          • reason string

            A human-readable explanation of the error, in English.

          • The server stack trace. Present only if the error_trace=true parameter was sent with the request.

          • Additional properties are allowed.

          • root_cause array[object]
          • suppressed array[object]
POST /_ingest/pipeline/{id}/_simulate
curl \
 -X POST http://api.example.com/_ingest/pipeline/{id}/_simulate \
 -H "Content-Type: application/json" \
 -d '"{\n  \"pipeline\" :\n  {\n    \"description\": \"_description\",\n    \"processors\": [\n      {\n        \"set\" : {\n          \"field\" : \"field2\",\n          \"value\" : \"_value\"\n        }\n      }\n    ]\n  },\n  \"docs\": [\n    {\n      \"_index\": \"index\",\n      \"_id\": \"id\",\n      \"_source\": {\n        \"foo\": \"bar\"\n      }\n    },\n    {\n      \"_index\": \"index\",\n      \"_id\": \"id\",\n      \"_source\": {\n        \"foo\": \"rab\"\n      }\n    }\n  ]\n}"'
Request example
You can specify the used pipeline either in the request body or as a path parameter.
{
  "pipeline" :
  {
    "description": "_description",
    "processors": [
      {
        "set" : {
          "field" : "field2",
          "value" : "_value"
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": "rab"
      }
    }
  ]
}
Response examples (200)
A successful response for running an ingest pipeline against a set of provided documents.
{
   "docs": [
      {
         "doc": {
            "_id": "id",
            "_index": "index",
            "_version": "-3",
            "_source": {
               "field2": "_value",
               "foo": "bar"
            },
            "_ingest": {
               "timestamp": "2017-05-04T22:30:03.187Z"
            }
         }
      },
      {
         "doc": {
            "_id": "id",
            "_index": "index",
            "_version": "-3",
            "_source": {
               "field2": "_value",
               "foo": "rab"
            },
            "_ingest": {
               "timestamp": "2017-05-04T22:30:03.188Z"
            }
         }
      }
   ]
}
































































































































































































Get anomaly detection job results for buckets Added in 5.4.0

GET /_ml/anomaly_detectors/{job_id}/results/buckets/{timestamp}

The API presents a chronological view of the records, grouped by bucket.

Path parameters

  • job_id string Required

    Identifier for the anomaly detection job.

  • timestamp string | number Required

    The timestamp of a single bucket result. If you do not specify this parameter, the API returns information about all buckets.

Query parameters

  • Returns buckets with anomaly scores greater or equal than this value.

  • desc boolean

    If true, the buckets are sorted in descending order.

  • end string | number

    Returns buckets with timestamps earlier than this time. -1 means it is unset and results are not limited to specific timestamps.

  • If true, the output excludes interim results.

  • expand boolean

    If true, the output includes anomaly records.

  • from number

    Skips the specified number of buckets.

  • size number

    Specifies the maximum number of buckets to obtain.

  • sort string

    Specifies the sort field for the requested buckets.

  • start string | number

    Returns buckets with timestamps after this time. -1 means it is unset and results are not limited to specific timestamps.

application/json

Body

  • Refer to the description for the anomaly_score query parameter.

  • desc boolean

    Refer to the description for the desc query parameter.

  • end string | number

    A date and time, either as a string whose format can depend on the context (defaulting to ISO 8601), or a number of milliseconds since the Epoch. Elasticsearch accepts both as input, but will generally output a string representation.

  • Refer to the description for the exclude_interim query parameter.

  • expand boolean

    Refer to the description for the expand query parameter.

  • page object

    Additional properties are allowed.

    Hide page attributes Show page attributes object
    • from number

      Skips the specified number of items.

    • size number

      Specifies the maximum number of items to obtain.

  • sort string

    Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

  • start string | number

    A date and time, either as a string whose format can depend on the context (defaulting to ISO 8601), or a number of milliseconds since the Epoch. Elasticsearch accepts both as input, but will generally output a string representation.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • buckets array[object] Required
      Hide buckets attributes Show buckets attributes object
      • anomaly_score number Required

        The maximum anomaly score, between 0-100, for any of the bucket influencers. This is an overall, rate-limited score for the job. All the anomaly records in the bucket contribute to this score. This value might be updated as new data is analyzed.

      • bucket_influencers array[object] Required
        Hide bucket_influencers attributes Show bucket_influencers attributes object
        • anomaly_score number Required

          A normalized score between 0-100, which is calculated for each bucket influencer. This score might be updated as newer data is analyzed.

        • Time unit for seconds

        • influencer_field_name string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • initial_anomaly_score number Required

          The score between 0-100 for each bucket influencer. This score is the initial value that was calculated at the time the bucket was processed.

        • is_interim boolean Required

          If true, this is an interim result. In other words, the results are calculated based on partial input data.

        • job_id string Required
        • probability number Required

          The probability that the bucket has this behavior, in the range 0 to 1. This value can be held to a high precision of over 300 decimal places, so the anomaly_score is provided as a human-readable and friendly interpretation of this.

        • raw_anomaly_score number Required

          Internal.

        • result_type string Required

          Internal. This value is always set to bucket_influencer.

        • Time unit for milliseconds

      • Time unit for seconds

      • event_count number Required

        The number of input data records processed in this bucket.

      • initial_anomaly_score number Required

        The maximum anomaly score for any of the bucket influencers. This is the initial value that was calculated at the time the bucket was processed.

      • is_interim boolean Required

        If true, this is an interim result. In other words, the results are calculated based on partial input data.

      • job_id string Required
      • Time unit for milliseconds

      • result_type string Required

        Internal. This value is always set to bucket.

      • Time unit for milliseconds

      • timestamp_string string | number

        A date and time, either as a string whose format can depend on the context (defaulting to ISO 8601), or a number of milliseconds since the Epoch. Elasticsearch accepts both as input, but will generally output a string representation.

    • count number Required
GET /_ml/anomaly_detectors/{job_id}/results/buckets/{timestamp}
curl \
 -X GET http://api.example.com/_ml/anomaly_detectors/{job_id}/results/buckets/{timestamp} \
 -H "Content-Type: application/json" \
 -d '{"anomaly_score":42.0,"desc":true,"":"string","exclude_interim":true,"expand":true,"page":{"from":42.0,"size":42.0},"sort":"string"}'
Request examples
{
  "anomaly_score": 42.0,
  "desc": true,
  "": "string",
  "exclude_interim": true,
  "expand": true,
  "page": {
    "from": 42.0,
    "size": 42.0
  },
  "sort": "string"
}
Response examples (200)
{
  "buckets": [
    {
      "anomaly_score": 42.0,
      "bucket_influencers": [
        {
          "anomaly_score": 42.0,
          "": "string",
          "influencer_field_name": "string",
          "initial_anomaly_score": 42.0,
          "is_interim": true,
          "job_id": "string",
          "probability": 42.0,
          "raw_anomaly_score": 42.0,
          "result_type": "string"
        }
      ],
      "": "string",
      "event_count": 42.0,
      "initial_anomaly_score": 42.0,
      "is_interim": true,
      "job_id": "string",
      "result_type": "string"
    }
  ],
  "count": 42.0
}




















































































































Preview a datafeed Added in 5.4.0

GET /_ml/datafeeds/{datafeed_id}/_preview

This API returns the first "page" of search results from a datafeed. You can preview an existing datafeed or provide configuration details for a datafeed and anomaly detection job in the API. The preview shows the structure of the data that will be passed to the anomaly detection engine. IMPORTANT: When Elasticsearch security features are enabled, the preview uses the credentials of the user that called the API. However, when the datafeed starts it uses the roles of the last user that created or updated the datafeed. To get a preview that accurately reflects the behavior of the datafeed, use the appropriate credentials. You can also use secondary authorization headers to supply the credentials.

Path parameters

  • datafeed_id string Required

    A numerical character string that uniquely identifies the datafeed. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters. NOTE: If you use this path parameter, you cannot provide datafeed or anomaly detection job configuration details in the request body.

Query parameters

  • start string | number

    The start time from where the datafeed preview should begin

  • end string | number

    The end time when the datafeed preview should stop

application/json

Body

  • Additional properties are allowed.

    Hide datafeed_config attributes Show datafeed_config attributes object
    • If set, the datafeed performs aggregation searches. Support for aggregations is limited and should be used only with low cardinality data.

    • Additional properties are allowed.

      Hide chunking_config attributes Show chunking_config attributes object
      • mode string Required

        Values are auto, manual, or off.

      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • Additional properties are allowed.

      Hide delayed_data_check_config attributes Show delayed_data_check_config attributes object
      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • enabled boolean Required

        Specifies whether the datafeed periodically checks for delayed data.

    • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • indices string | array[string]
    • Additional properties are allowed.

      Hide indices_options attributes Show indices_options attributes object
      • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.

      • expand_wildcards string | array[string]
      • If true, missing or closed indices are not included in the response.

      • If true, concrete, expanded or aliased indices are ignored when frozen.

    • job_id string
    • If a real-time datafeed has never seen any data (including during any initial training period) then it will automatically stop itself and close its associated job after this many real-time searches that return no documents. In other words, it will stop after frequency times max_empty_searches of real-time operation. If not set then a datafeed with no end time that sees no data will remain started until it is explicitly stopped.

    • query object

      An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

      Additional properties are allowed.

    • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • Hide runtime_mappings attribute Show runtime_mappings attribute object
      • * object Additional properties

        Additional properties are allowed.

        Hide * attributes Show * attributes object
        • fields object

          For type composite

          Hide fields attribute Show fields attribute object
          • * object Additional properties

            Additional properties are allowed.

            Hide * attribute Show * attribute object
            • type string Required

              Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

        • fetch_fields array[object]

          For type lookup

          Hide fetch_fields attributes Show fetch_fields attributes object
          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • format string
        • format string

          A custom format for date type runtime fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • script object

          Additional properties are allowed.

          Hide script attributes Show script attributes object
          • source string

            The script source.

          • id string
          • params object

            Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

            Hide params attribute Show params attribute object
            • * object Additional properties

              Additional properties are allowed.

          • lang string

            Any of:

            Values are painless, expression, mustache, or java.

          • options object
            Hide options attribute Show options attribute object
            • * string Additional properties
        • type string Required

          Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

    • Specifies scripts that evaluate custom expressions and returns script fields to the datafeed. The detector configuration objects in a job can contain functions that use these script fields.

      Hide script_fields attribute Show script_fields attribute object
      • * object Additional properties

        Additional properties are allowed.

        Hide * attributes Show * attributes object
        • script object Required

          Additional properties are allowed.

          Hide script attributes Show script attributes object
          • source string

            The script source.

          • id string
          • params object

            Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

            Hide params attribute Show params attribute object
            • * object Additional properties

              Additional properties are allowed.

          • lang string

            Any of:

            Values are painless, expression, mustache, or java.

          • options object
            Hide options attribute Show options attribute object
            • * string Additional properties
    • The size parameter that is used in Elasticsearch searches when the datafeed does not use aggregations. The maximum value is the value of index.max_result_window, which is 10,000 by default.

  • Additional properties are allowed.

    Hide job_config attributes Show job_config attributes object
    • Advanced configuration option. Specifies whether this job can open when there is insufficient machine learning node capacity for it to be immediately assigned to a node.

    • analysis_config object Required

      Additional properties are allowed.

      Hide analysis_config attributes Show analysis_config attributes object
      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • categorization_analyzer string | object

        One of:
      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • If categorization_field_name is specified, you can also define optional filters. This property expects an array of regular expressions. The expressions are used to filter out matching sequences from the categorization field values. You can use this functionality to fine tune the categorization by excluding sequences from consideration when categories are defined. For example, you can exclude SQL statements that appear in your log files. This property cannot be used at the same time as categorization_analyzer. If you only want to define simple regular expression filters that are applied prior to tokenization, setting this property is the easiest method. If you also want to customize the tokenizer or post-tokenization filtering, use the categorization_analyzer property instead and include the filters as pattern_replace character filters. The effect is exactly the same.

      • detectors array[object] Required

        Detector configuration objects specify which data fields a job analyzes. They also specify which analytical functions are used. You can specify multiple detectors for a job. If the detectors array does not contain at least one detector, no analysis can occur and an error is returned.

        Hide detectors attributes Show detectors attributes object
        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • custom_rules array[object]

          Custom rules enable you to customize the way detectors operate. For example, a rule may dictate conditions under which results should be skipped. Kibana refers to custom rules as job rules.

          Hide custom_rules attributes Show custom_rules attributes object
          • actions array[string]

            The set of actions to be triggered when the rule applies. If more than one action is specified the effects of all actions are combined.

            Values are skip_result or skip_model_update.

          • conditions array[object]

            An array of numeric conditions when the rule applies. A rule must either have a non-empty scope or at least one condition. Multiple conditions are combined together with a logical AND.

          • scope object

            A scope of series where the rule applies. A rule must either have a non-empty scope or at least one condition. By default, the scope includes all series. Scoping is allowed for any of the fields that are also specified in by_field_name, over_field_name, or partition_field_name.

        • A description of the detector.

        • A unique identifier for the detector. This identifier is based on the order of the detectors in the analysis_config, starting at zero. If you specify a value for this property, it is ignored.

        • Values are all, none, by, or over.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • function string

          The analysis function that is used. For example, count, rare, mean, min, max, or sum.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • use_null boolean

          Defines whether a new series is used as the null series when there is no value for the by or partition fields.

      • influencers array[string]

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • latency string

        A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • This functionality is reserved for internal use. It is not supported for use in customer environments and is not subject to the support SLA of official GA features. If set to true, the analysis will automatically find correlations between metrics for a given by field value and report anomalies when those correlations cease to hold. For example, suppose CPU and memory usage on host A is usually highly correlated with the same metrics on host B. Perhaps this correlation occurs because they are running a load-balanced application. If you enable this property, anomalies will be reported when, for example, CPU usage on host A is high and the value of CPU usage on host B is low. That is to say, you’ll see an anomaly when the CPU of host A is unusual given the CPU of host B. To use the multivariate_by_fields property, you must also specify by_field_name in your detector.

      • Additional properties are allowed.

        Hide per_partition_categorization attributes Show per_partition_categorization attributes object
        • enabled boolean

          To enable this setting, you must also set the partition_field_name property to the same value in every detector that uses the keyword mlcategory. Otherwise, job creation fails.

        • This setting can be set to true only if per-partition categorization is enabled. If true, both categorization and subsequent anomaly detection stops for partitions where the categorization status changes to warn. This setting makes it viable to have a job where it is expected that categorization works well for some partitions but not others; you do not pay the cost of bad categorization forever in the partitions where it works badly.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

    • Additional properties are allowed.

      Hide analysis_limits attributes Show analysis_limits attributes object
    • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • Custom metadata about the job

      Additional properties are allowed.

    • Advanced configuration option, which affects the automatic removal of old model snapshots for this job. It specifies a period of time (in days) after which only the first snapshot per day is retained. This period is relative to the timestamp of the most recent snapshot for this job.

    • data_description object Required

      Additional properties are allowed.

      Hide data_description attributes Show data_description attributes object
      • format string

        Only JSON format is supported at this time.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • The time format, which can be epoch, epoch_ms, or a custom pattern. The value epoch refers to UNIX or Epoch time (the number of seconds since 1 Jan 1970). The value epoch_ms indicates that time is measured in milliseconds since the epoch. The epoch and epoch_ms time formats accept either integer or real values. Custom patterns must conform to the Java DateTimeFormatter class. When you use date-time formatting patterns, it is recommended that you provide the full date, time and time zone. For example: yyyy-MM-dd'T'HH:mm:ssX. If the pattern that you specify is not sufficient to produce a complete timestamp, job creation fails.

    • Additional properties are allowed.

      Hide datafeed_config attributes Show datafeed_config attributes object
      • If set, the datafeed performs aggregation searches. Support for aggregations is limited and should be used only with low cardinality data.

      • Additional properties are allowed.

        Hide chunking_config attributes Show chunking_config attributes object
        • mode string Required

          Values are auto, manual, or off.

        • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • Additional properties are allowed.

        Hide delayed_data_check_config attributes Show delayed_data_check_config attributes object
        • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

        • enabled boolean Required

          Specifies whether the datafeed periodically checks for delayed data.

      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • indices string | array[string]
      • Additional properties are allowed.

        Hide indices_options attributes Show indices_options attributes object
        • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.

        • expand_wildcards string | array[string]
        • If true, missing or closed indices are not included in the response.

        • If true, concrete, expanded or aliased indices are ignored when frozen.

      • job_id string
      • If a real-time datafeed has never seen any data (including during any initial training period) then it will automatically stop itself and close its associated job after this many real-time searches that return no documents. In other words, it will stop after frequency times max_empty_searches of real-time operation. If not set then a datafeed with no end time that sees no data will remain started until it is explicitly stopped.

      • query object

        An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

        Additional properties are allowed.

      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • Hide runtime_mappings attribute Show runtime_mappings attribute object
        • * object Additional properties

          Additional properties are allowed.

          Hide * attributes Show * attributes object
          • fields object

            For type composite

            Hide fields attribute Show fields attribute object
            • * object Additional properties

              Additional properties are allowed.

              Hide * attribute Show * attribute object
              • type string Required

                Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

          • fetch_fields array[object]

            For type lookup

            Hide fetch_fields attributes Show fetch_fields attributes object
            • field string Required

              Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

            • format string
          • format string

            A custom format for date type runtime fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • script object

            Additional properties are allowed.

            Hide script attributes Show script attributes object
            • source string

              The script source.

            • id string
            • params object

              Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

              Hide params attribute Show params attribute object
              • * object Additional properties

                Additional properties are allowed.

            • lang string

              Any of:

              Values are painless, expression, mustache, or java.

            • options object
              Hide options attribute Show options attribute object
              • * string Additional properties
          • type string Required

            Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

      • Specifies scripts that evaluate custom expressions and returns script fields to the datafeed. The detector configuration objects in a job can contain functions that use these script fields.

        Hide script_fields attribute Show script_fields attribute object
        • * object Additional properties

          Additional properties are allowed.

          Hide * attributes Show * attributes object
          • script object Required

            Additional properties are allowed.

            Hide script attributes Show script attributes object
            • source string

              The script source.

            • id string
            • params object

              Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

              Hide params attribute Show params attribute object
              • * object Additional properties

                Additional properties are allowed.

            • lang string

              Any of:

              Values are painless, expression, mustache, or java.

            • options object
              Hide options attribute Show options attribute object
              • * string Additional properties
      • The size parameter that is used in Elasticsearch searches when the datafeed does not use aggregations. The maximum value is the value of index.max_result_window, which is 10,000 by default.

    • A description of the job.

    • groups array[string]

      A list of job groups. A job can belong to no groups or many.

    • job_id string
    • job_type string

      Reserved for future use, currently set to anomaly_detector.

    • Additional properties are allowed.

      Hide model_plot_config attributes Show model_plot_config attributes object
      • If true, enables calculation and storage of the model change annotations for each entity that is being analyzed.

      • enabled boolean

        If true, enables calculation and storage of the model bounds for each entity that is being analyzed.

      • terms string

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

    • Advanced configuration option, which affects the automatic removal of old model snapshots for this job. It specifies the maximum period of time (in days) that snapshots are retained. This period is relative to the timestamp of the most recent snapshot for this job. The default value is 10, which means snapshots ten days older than the newest snapshot are deleted.

    • Advanced configuration option. The period over which adjustments to the score are applied, as new data is seen. The default value is the longer of 30 days or 100 bucket_spans.

    • Advanced configuration option. The period of time (in days) that results are retained. Age is calculated relative to the timestamp of the latest bucket result. If this property has a non-null value, once per day at 00:30 (server time), results that are the specified number of days older than the latest bucket result are deleted from Elasticsearch. The default value is null, which means all results are retained. Annotations generated by the system also count as results for retention purposes; they are deleted after the same number of days as results. Annotations added by users are retained forever.

Responses

  • 200 application/json

    Additional properties are allowed.

GET /_ml/datafeeds/{datafeed_id}/_preview
curl \
 -X GET http://api.example.com/_ml/datafeeds/{datafeed_id}/_preview \
 -H "Content-Type: application/json" \
 -d '{"datafeed_config":{"aggregations":{},"chunking_config":{"mode":"auto","time_span":"string"},"datafeed_id":"string","delayed_data_check_config":{"check_window":"string","enabled":true},"frequency":"string","indices":"string","indices_options":{"allow_no_indices":true,"expand_wildcards":"string","ignore_unavailable":true,"ignore_throttled":true},"job_id":"string","max_empty_searches":42.0,"query":{},"query_delay":"string","runtime_mappings":{"additionalProperty1":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"},"additionalProperty2":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"}},"script_fields":{"additionalProperty1":{"script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"ignore_failure":true},"additionalProperty2":{"script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"ignore_failure":true}},"scroll_size":42.0},"job_config":{"allow_lazy_open":true,"analysis_config":{"bucket_span":"string","":"string","categorization_field_name":"string","categorization_filters":["string"],"detectors":[{"by_field_name":"string","custom_rules":[{"actions":["skip_result"],"conditions":[{}],"scope":{}}],"detector_description":"string","detector_index":42.0,"exclude_frequent":"all","field_name":"string","function":"string","over_field_name":"string","partition_field_name":"string","use_null":true}],"influencers":["string"],"latency":"string","model_prune_window":"string","multivariate_by_fields":true,"per_partition_categorization":{"enabled":true,"stop_on_warn":true},"summary_count_field_name":"string"},"analysis_limits":{"categorization_examples_limit":42.0,"":42.0},"background_persist_interval":"string","custom_settings":{},"daily_model_snapshot_retention_after_days":42.0,"data_description":{"format":"string","time_field":"string","time_format":"string","field_delimiter":"string"},"datafeed_config":{"aggregations":{},"chunking_config":{"mode":"auto","time_span":"string"},"datafeed_id":"string","delayed_data_check_config":{"check_window":"string","enabled":true},"frequency":"string","indices":"string","indices_options":{"allow_no_indices":true,"expand_wildcards":"string","ignore_unavailable":true,"ignore_throttled":true},"job_id":"string","max_empty_searches":42.0,"query":{},"query_delay":"string","runtime_mappings":{"additionalProperty1":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"},"additionalProperty2":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"}},"script_fields":{"additionalProperty1":{"script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"ignore_failure":true},"additionalProperty2":{"script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"ignore_failure":true}},"scroll_size":42.0},"description":"string","groups":["string"],"job_id":"string","job_type":"string","model_plot_config":{"annotations_enabled":true,"enabled":true,"terms":"string"},"model_snapshot_retention_days":42.0,"renormalization_window_days":42.0,"results_index_name":"string","results_retention_days":42.0}}'
Request examples
{
  "datafeed_config": {
    "aggregations": {},
    "chunking_config": {
      "mode": "auto",
      "time_span": "string"
    },
    "datafeed_id": "string",
    "delayed_data_check_config": {
      "check_window": "string",
      "enabled": true
    },
    "frequency": "string",
    "indices": "string",
    "indices_options": {
      "allow_no_indices": true,
      "expand_wildcards": "string",
      "ignore_unavailable": true,
      "ignore_throttled": true
    },
    "job_id": "string",
    "max_empty_searches": 42.0,
    "query": {},
    "query_delay": "string",
    "runtime_mappings": {
      "additionalProperty1": {
        "fields": {
          "additionalProperty1": {
            "type": "boolean"
          },
          "additionalProperty2": {
            "type": "boolean"
          }
        },
        "fetch_fields": [
          {
            "field": "string",
            "format": "string"
          }
        ],
        "format": "string",
        "input_field": "string",
        "target_field": "string",
        "target_index": "string",
        "script": {
          "source": "string",
          "id": "string",
          "params": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "": "painless",
          "options": {
            "additionalProperty1": "string",
            "additionalProperty2": "string"
          }
        },
        "type": "boolean"
      },
      "additionalProperty2": {
        "fields": {
          "additionalProperty1": {
            "type": "boolean"
          },
          "additionalProperty2": {
            "type": "boolean"
          }
        },
        "fetch_fields": [
          {
            "field": "string",
            "format": "string"
          }
        ],
        "format": "string",
        "input_field": "string",
        "target_field": "string",
        "target_index": "string",
        "script": {
          "source": "string",
          "id": "string",
          "params": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "": "painless",
          "options": {
            "additionalProperty1": "string",
            "additionalProperty2": "string"
          }
        },
        "type": "boolean"
      }
    },
    "script_fields": {
      "additionalProperty1": {
        "script": {
          "source": "string",
          "id": "string",
          "params": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "": "painless",
          "options": {
            "additionalProperty1": "string",
            "additionalProperty2": "string"
          }
        },
        "ignore_failure": true
      },
      "additionalProperty2": {
        "script": {
          "source": "string",
          "id": "string",
          "params": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "": "painless",
          "options": {
            "additionalProperty1": "string",
            "additionalProperty2": "string"
          }
        },
        "ignore_failure": true
      }
    },
    "scroll_size": 42.0
  },
  "job_config": {
    "allow_lazy_open": true,
    "analysis_config": {
      "bucket_span": "string",
      "": "string",
      "categorization_field_name": "string",
      "categorization_filters": [
        "string"
      ],
      "detectors": [
        {
          "by_field_name": "string",
          "custom_rules": [
            {
              "actions": [
                "skip_result"
              ],
              "conditions": [
                {}
              ],
              "scope": {}
            }
          ],
          "detector_description": "string",
          "detector_index": 42.0,
          "exclude_frequent": "all",
          "field_name": "string",
          "function": "string",
          "over_field_name": "string",
          "partition_field_name": "string",
          "use_null": true
        }
      ],
      "influencers": [
        "string"
      ],
      "latency": "string",
      "model_prune_window": "string",
      "multivariate_by_fields": true,
      "per_partition_categorization": {
        "enabled": true,
        "stop_on_warn": true
      },
      "summary_count_field_name": "string"
    },
    "analysis_limits": {
      "categorization_examples_limit": 42.0,
      "": 42.0
    },
    "background_persist_interval": "string",
    "custom_settings": {},
    "daily_model_snapshot_retention_after_days": 42.0,
    "data_description": {
      "format": "string",
      "time_field": "string",
      "time_format": "string",
      "field_delimiter": "string"
    },
    "datafeed_config": {
      "aggregations": {},
      "chunking_config": {
        "mode": "auto",
        "time_span": "string"
      },
      "datafeed_id": "string",
      "delayed_data_check_config": {
        "check_window": "string",
        "enabled": true
      },
      "frequency": "string",
      "indices": "string",
      "indices_options": {
        "allow_no_indices": true,
        "expand_wildcards": "string",
        "ignore_unavailable": true,
        "ignore_throttled": true
      },
      "job_id": "string",
      "max_empty_searches": 42.0,
      "query": {},
      "query_delay": "string",
      "runtime_mappings": {
        "additionalProperty1": {
          "fields": {
            "additionalProperty1": {
              "type": "boolean"
            },
            "additionalProperty2": {
              "type": "boolean"
            }
          },
          "fetch_fields": [
            {
              "field": "string",
              "format": "string"
            }
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {
              "additionalProperty1": {},
              "additionalProperty2": {}
            },
            "": "painless",
            "options": {
              "additionalProperty1": "string",
              "additionalProperty2": "string"
            }
          },
          "type": "boolean"
        },
        "additionalProperty2": {
          "fields": {
            "additionalProperty1": {
              "type": "boolean"
            },
            "additionalProperty2": {
              "type": "boolean"
            }
          },
          "fetch_fields": [
            {
              "field": "string",
              "format": "string"
            }
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {
              "additionalProperty1": {},
              "additionalProperty2": {}
            },
            "": "painless",
            "options": {
              "additionalProperty1": "string",
              "additionalProperty2": "string"
            }
          },
          "type": "boolean"
        }
      },
      "script_fields": {
        "additionalProperty1": {
          "script": {
            "source": "string",
            "id": "string",
            "params": {
              "additionalProperty1": {},
              "additionalProperty2": {}
            },
            "": "painless",
            "options": {
              "additionalProperty1": "string",
              "additionalProperty2": "string"
            }
          },
          "ignore_failure": true
        },
        "additionalProperty2": {
          "script": {
            "source": "string",
            "id": "string",
            "params": {
              "additionalProperty1": {},
              "additionalProperty2": {}
            },
            "": "painless",
            "options": {
              "additionalProperty1": "string",
              "additionalProperty2": "string"
            }
          },
          "ignore_failure": true
        }
      },
      "scroll_size": 42.0
    },
    "description": "string",
    "groups": [
      "string"
    ],
    "job_id": "string",
    "job_type": "string",
    "model_plot_config": {
      "annotations_enabled": true,
      "enabled": true,
      "terms": "string"
    },
    "model_snapshot_retention_days": 42.0,
    "renormalization_window_days": 42.0,
    "results_index_name": "string",
    "results_retention_days": 42.0
  }
}
Response examples (200)
[
  {}
]





























































Create a data frame analytics job Added in 7.3.0

PUT /_ml/data_frame/analytics/{id}

This API creates a data frame analytics job that performs an analysis on the source indices and stores the outcome in a destination index. By default, the query used in the source configuration is {"match_all": {}}.

If the destination index does not exist, it is created automatically when you start the job.

If you supply only a subset of the regression or classification parameters, hyperparameter optimization occurs. It determines a value for each of the undefined parameters.

Path parameters

  • id string Required

    Identifier for the data frame analytics job. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters.

application/json

Body Required

  • Specifies whether this job can start when there is insufficient machine learning node capacity for it to be immediately assigned to a node. If set to false and a machine learning node with capacity to run the job cannot be immediately found, the API returns an error. If set to true, the API does not return an error; the job waits in the starting state until sufficient machine learning node capacity is available. This behavior is also affected by the cluster-wide xpack.ml.max_lazy_ml_nodes setting.

  • analysis object Required

    Additional properties are allowed.

    Hide analysis attributes Show analysis attributes object
    • Additional properties are allowed.

      Hide classification attributes Show classification attributes object
      • alpha number

        Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This parameter affects loss calculations by acting as a multiplier of the tree depth. Higher alpha values result in shallower trees and faster training times. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to zero.

      • dependent_variable string Required

        Defines which field of the document is to be predicted. It must match one of the fields in the index being used to train. If this field is missing from a document, then that document will not be used for training, but a prediction with the trained model will be generated for it. It is also known as continuous target variable. For classification analysis, the data type of the field must be numeric (integer, short, long, byte), categorical (ip or keyword), or boolean. There must be no more than 30 different values in this field. For regression analysis, the data type of the field must be numeric.

      • Advanced configuration option. Controls the fraction of data that is used to compute the derivatives of the loss function for tree training. A small value results in the use of a small fraction of the data. If this value is set to be less than 1, accuracy typically improves. However, too small a value may result in poor convergence for the ensemble and so require more trees. By default, this value is calculated during hyperparameter optimization. It must be greater than zero and less than or equal to 1.

      • Advanced configuration option. Specifies whether the training process should finish if it is not finding any better performing models. If disabled, the training process can take significantly longer and the chance of finding a better performing model is unremarkable.

      • eta number

        Advanced configuration option. The shrinkage applied to the weights. Smaller values result in larger forests which have a better generalization error. However, larger forests cause slower training. By default, this value is calculated during hyperparameter optimization. It must be a value between 0.001 and 1.

      • Advanced configuration option. Specifies the rate at which eta increases for each new tree that is added to the forest. For example, a rate of 1.05 increases eta by 5% for each extra tree. By default, this value is calculated during hyperparameter optimization. It must be between 0.5 and 2.

      • Advanced configuration option. Defines the fraction of features that will be used when selecting a random bag for each candidate split. By default, this value is calculated during hyperparameter optimization.

      • feature_processors array[object]

        Advanced configuration option. A collection of feature preprocessors that modify one or more included fields. The analysis uses the resulting one or more features instead of the original document field. However, these features are ephemeral; they are not stored in the destination index. Multiple feature_processors entries can refer to the same document fields. Automatic categorical feature encoding still occurs for the fields that are unprocessed by a custom processor or that have categorical values. Use this property only if you want to override the automatic feature encoding of the specified fields.

        Hide feature_processors attributes Show feature_processors attributes object
        • Additional properties are allowed.

          Hide frequency_encoding attributes Show frequency_encoding attributes object
          • feature_name string Required
          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • frequency_map object Required

            The resulting frequency map for the field value. If the field value is missing from the frequency_map, the resulting value is 0.

        • Additional properties are allowed.

          Hide multi_encoding attribute Show multi_encoding attribute object
          • processors array[number] Required

            The ordered array of custom processors to execute. Must be more than 1.

        • Additional properties are allowed.

          Hide n_gram_encoding attributes Show n_gram_encoding attributes object
          • The feature name prefix. Defaults to ngram__.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • length number

            Specifies the length of the n-gram substring. Defaults to 50. Must be greater than 0.

          • n_grams array[number] Required

            Specifies which n-grams to gather. It’s an array of integer values where the minimum value is 1, and a maximum value is 5.

          • start number

            Specifies the zero-indexed start of the n-gram substring. Negative values are allowed for encoding n-grams of string suffixes. Defaults to 0.

          • custom boolean
        • Additional properties are allowed.

          Hide one_hot_encoding attributes Show one_hot_encoding attributes object
          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • hot_map string Required

            The one hot map mapping the field value with the column name.

        • Additional properties are allowed.

          Hide target_mean_encoding attributes Show target_mean_encoding attributes object
          • default_value number Required

            The default value if field value is not found in the target_map.

          • feature_name string Required
          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • target_map object Required

            The field value to target mean transition map.

      • gamma number

        Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies a linear penalty associated with the size of individual trees in the forest. A high gamma value causes training to prefer small trees. A small gamma value results in larger individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

      • lambda number

        Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies an L2 regularization term which applies to leaf weights of the individual trees in the forest. A high lambda value causes training to favor small leaf weights. This behavior makes the prediction function smoother at the expense of potentially not being able to capture relevant relationships between the features and the dependent variable. A small lambda value results in large individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

      • Advanced configuration option. A multiplier responsible for determining the maximum number of hyperparameter optimization steps in the Bayesian optimization procedure. The maximum number of steps is determined based on the number of undefined hyperparameters times the maximum optimization rounds per hyperparameter. By default, this value is calculated during hyperparameter optimization.

      • Advanced configuration option. Defines the maximum number of decision trees in the forest. The maximum value is 2000. By default, this value is calculated during hyperparameter optimization.

      • Advanced configuration option. Specifies the maximum number of feature importance values per document to return. By default, no feature importance calculation occurs.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Defines the seed for the random generator that is used to pick training data. By default, it is randomly generated. Set it to a specific value to use the same training data each time you start a job (assuming other related parameters such as source and analyzed_fields are the same).

      • Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This soft limit combines with the soft_tree_depth_tolerance to penalize trees that exceed the specified depth; the regularized loss increases quickly beyond this depth. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.

      • Advanced configuration option. This option controls how quickly the regularized loss increases when the tree depth exceeds soft_tree_depth_limit. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.01.

      • Defines the number of categories for which the predicted probabilities are reported. It must be non-negative or -1. If it is -1 or greater than the total number of categories, probabilities are reported for all categories; if you have a large number of categories, there could be a significant effect on the size of your destination index. NOTE: To use the AUC ROC evaluation method, num_top_classes must be set to -1 or a value greater than or equal to the total number of categories.

    • Additional properties are allowed.

      Hide outlier_detection attributes Show outlier_detection attributes object
      • Specifies whether the feature influence calculation is enabled.

      • The minimum outlier score that a document needs to have in order to calculate its feature influence score. Value range: 0-1.

      • method string

        The method that outlier detection uses. Available methods are lof, ldof, distance_kth_nn, distance_knn, and ensemble. The default value is ensemble, which means that outlier detection uses an ensemble of different methods and normalises and combines their individual outlier scores to obtain the overall outlier score.

      • Defines the value for how many nearest neighbors each method of outlier detection uses to calculate its outlier score. When the value is not set, different values are used for different ensemble members. This default behavior helps improve the diversity in the ensemble; only override it if you are confident that the value you choose is appropriate for the data set.

      • The proportion of the data set that is assumed to be outlying prior to outlier detection. For example, 0.05 means it is assumed that 5% of values are real outliers and 95% are inliers.

      • If true, the following operation is performed on the columns before computing outlier scores: (x_i - mean(x_i)) / sd(x_i).

    • Additional properties are allowed.

      Hide regression attributes Show regression attributes object
      • alpha number

        Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This parameter affects loss calculations by acting as a multiplier of the tree depth. Higher alpha values result in shallower trees and faster training times. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to zero.

      • dependent_variable string Required

        Defines which field of the document is to be predicted. It must match one of the fields in the index being used to train. If this field is missing from a document, then that document will not be used for training, but a prediction with the trained model will be generated for it. It is also known as continuous target variable. For classification analysis, the data type of the field must be numeric (integer, short, long, byte), categorical (ip or keyword), or boolean. There must be no more than 30 different values in this field. For regression analysis, the data type of the field must be numeric.

      • Advanced configuration option. Controls the fraction of data that is used to compute the derivatives of the loss function for tree training. A small value results in the use of a small fraction of the data. If this value is set to be less than 1, accuracy typically improves. However, too small a value may result in poor convergence for the ensemble and so require more trees. By default, this value is calculated during hyperparameter optimization. It must be greater than zero and less than or equal to 1.

      • Advanced configuration option. Specifies whether the training process should finish if it is not finding any better performing models. If disabled, the training process can take significantly longer and the chance of finding a better performing model is unremarkable.

      • eta number

        Advanced configuration option. The shrinkage applied to the weights. Smaller values result in larger forests which have a better generalization error. However, larger forests cause slower training. By default, this value is calculated during hyperparameter optimization. It must be a value between 0.001 and 1.

      • Advanced configuration option. Specifies the rate at which eta increases for each new tree that is added to the forest. For example, a rate of 1.05 increases eta by 5% for each extra tree. By default, this value is calculated during hyperparameter optimization. It must be between 0.5 and 2.

      • Advanced configuration option. Defines the fraction of features that will be used when selecting a random bag for each candidate split. By default, this value is calculated during hyperparameter optimization.

      • feature_processors array[object]

        Advanced configuration option. A collection of feature preprocessors that modify one or more included fields. The analysis uses the resulting one or more features instead of the original document field. However, these features are ephemeral; they are not stored in the destination index. Multiple feature_processors entries can refer to the same document fields. Automatic categorical feature encoding still occurs for the fields that are unprocessed by a custom processor or that have categorical values. Use this property only if you want to override the automatic feature encoding of the specified fields.

        Hide feature_processors attributes Show feature_processors attributes object
        • Additional properties are allowed.

          Hide frequency_encoding attributes Show frequency_encoding attributes object
          • feature_name string Required
          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • frequency_map object Required

            The resulting frequency map for the field value. If the field value is missing from the frequency_map, the resulting value is 0.

        • Additional properties are allowed.

          Hide multi_encoding attribute Show multi_encoding attribute object
          • processors array[number] Required

            The ordered array of custom processors to execute. Must be more than 1.

        • Additional properties are allowed.

          Hide n_gram_encoding attributes Show n_gram_encoding attributes object
          • The feature name prefix. Defaults to ngram__.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • length number

            Specifies the length of the n-gram substring. Defaults to 50. Must be greater than 0.

          • n_grams array[number] Required

            Specifies which n-grams to gather. It’s an array of integer values where the minimum value is 1, and a maximum value is 5.

          • start number

            Specifies the zero-indexed start of the n-gram substring. Negative values are allowed for encoding n-grams of string suffixes. Defaults to 0.

          • custom boolean
        • Additional properties are allowed.

          Hide one_hot_encoding attributes Show one_hot_encoding attributes object
          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • hot_map string Required

            The one hot map mapping the field value with the column name.

        • Additional properties are allowed.

          Hide target_mean_encoding attributes Show target_mean_encoding attributes object
          • default_value number Required

            The default value if field value is not found in the target_map.

          • feature_name string Required
          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • target_map object Required

            The field value to target mean transition map.

      • gamma number

        Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies a linear penalty associated with the size of individual trees in the forest. A high gamma value causes training to prefer small trees. A small gamma value results in larger individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

      • lambda number

        Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies an L2 regularization term which applies to leaf weights of the individual trees in the forest. A high lambda value causes training to favor small leaf weights. This behavior makes the prediction function smoother at the expense of potentially not being able to capture relevant relationships between the features and the dependent variable. A small lambda value results in large individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

      • Advanced configuration option. A multiplier responsible for determining the maximum number of hyperparameter optimization steps in the Bayesian optimization procedure. The maximum number of steps is determined based on the number of undefined hyperparameters times the maximum optimization rounds per hyperparameter. By default, this value is calculated during hyperparameter optimization.

      • Advanced configuration option. Defines the maximum number of decision trees in the forest. The maximum value is 2000. By default, this value is calculated during hyperparameter optimization.

      • Advanced configuration option. Specifies the maximum number of feature importance values per document to return. By default, no feature importance calculation occurs.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Defines the seed for the random generator that is used to pick training data. By default, it is randomly generated. Set it to a specific value to use the same training data each time you start a job (assuming other related parameters such as source and analyzed_fields are the same).

      • Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This soft limit combines with the soft_tree_depth_tolerance to penalize trees that exceed the specified depth; the regularized loss increases quickly beyond this depth. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.

      • Advanced configuration option. This option controls how quickly the regularized loss increases when the tree depth exceeds soft_tree_depth_limit. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.01.

      • The loss function used during regression. Available options are mse (mean squared error), msle (mean squared logarithmic error), huber (Pseudo-Huber loss).

      • A positive number that is used as a parameter to the loss_function.

  • Additional properties are allowed.

    Hide analyzed_fields attributes Show analyzed_fields attributes object
    • includes array[string] Required

      An array of strings that defines the fields that will be excluded from the analysis. You do not need to add fields with unsupported data types to excludes, these fields are excluded from the analysis automatically.

    • excludes array[string] Required

      An array of strings that defines the fields that will be included in the analysis.

  • A description of the job.

  • dest object Required

    Additional properties are allowed.

    Hide dest attributes Show dest attributes object
    • index string Required
    • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

  • The maximum number of threads to be used by the analysis. Using more threads may decrease the time necessary to complete the analysis at the cost of using more CPU. Note that the process may use additional threads for operational functionality other than the analysis itself.

  • _meta object
    Hide _meta attribute Show _meta attribute object
    • * object Additional properties

      Additional properties are allowed.

  • The approximate maximum amount of memory resources that are permitted for analytical processing. If your elasticsearch.yml file contains an xpack.ml.max_model_memory_limit setting, an error occurs when you try to create data frame analytics jobs that have model_memory_limit values greater than that setting.

  • source object Required

    Additional properties are allowed.

    Hide source attributes Show source attributes object
    • index string | array[string] Required
    • Hide runtime_mappings attribute Show runtime_mappings attribute object
      • * object Additional properties

        Additional properties are allowed.

        Hide * attributes Show * attributes object
        • fields object

          For type composite

          Hide fields attribute Show fields attribute object
          • * object Additional properties

            Additional properties are allowed.

            Hide * attribute Show * attribute object
            • type string Required

              Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

        • fetch_fields array[object]

          For type lookup

          Hide fetch_fields attributes Show fetch_fields attributes object
          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • format string
        • format string

          A custom format for date type runtime fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • script object

          Additional properties are allowed.

          Hide script attributes Show script attributes object
          • source string

            The script source.

          • id string
          • params object

            Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

            Hide params attribute Show params attribute object
            • * object Additional properties

              Additional properties are allowed.

          • lang string

            Any of:

            Values are painless, expression, mustache, or java.

          • options object
            Hide options attribute Show options attribute object
            • * string Additional properties
        • type string Required

          Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

    • _source object

      Additional properties are allowed.

      Hide _source attributes Show _source attributes object
      • includes array[string] Required

        An array of strings that defines the fields that will be excluded from the analysis. You do not need to add fields with unsupported data types to excludes, these fields are excluded from the analysis automatically.

      • excludes array[string] Required

        An array of strings that defines the fields that will be included in the analysis.

    • query object

      The Elasticsearch query domain-specific language (DSL). This value corresponds to the query object in an Elasticsearch search POST body. All the options that are supported by Elasticsearch can be used, as this object is passed verbatim to Elasticsearch. By default, this property has the following value: {"match_all": {}}.

      Additional properties are allowed.

  • headers object
  • version string

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • Additional properties are allowed.

      Hide authorization attributes Show authorization attributes object
      • api_key object

        Additional properties are allowed.

        Hide api_key attributes Show api_key attributes object
        • id string Required

          The identifier for the API key.

        • name string Required

          The name of the API key.

      • roles array[string]

        If a user ID was used for the most recent update to the job, its roles at the time of the update are listed in the response.

      • If a service account was used for the most recent update to the job, the account name is listed in the response.

    • allow_lazy_start boolean Required
    • analysis object Required

      Additional properties are allowed.

      Hide analysis attributes Show analysis attributes object
      • Additional properties are allowed.

        Hide classification attributes Show classification attributes object
        • alpha number

          Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This parameter affects loss calculations by acting as a multiplier of the tree depth. Higher alpha values result in shallower trees and faster training times. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to zero.

        • dependent_variable string Required

          Defines which field of the document is to be predicted. It must match one of the fields in the index being used to train. If this field is missing from a document, then that document will not be used for training, but a prediction with the trained model will be generated for it. It is also known as continuous target variable. For classification analysis, the data type of the field must be numeric (integer, short, long, byte), categorical (ip or keyword), or boolean. There must be no more than 30 different values in this field. For regression analysis, the data type of the field must be numeric.

        • Advanced configuration option. Controls the fraction of data that is used to compute the derivatives of the loss function for tree training. A small value results in the use of a small fraction of the data. If this value is set to be less than 1, accuracy typically improves. However, too small a value may result in poor convergence for the ensemble and so require more trees. By default, this value is calculated during hyperparameter optimization. It must be greater than zero and less than or equal to 1.

        • Advanced configuration option. Specifies whether the training process should finish if it is not finding any better performing models. If disabled, the training process can take significantly longer and the chance of finding a better performing model is unremarkable.

        • eta number

          Advanced configuration option. The shrinkage applied to the weights. Smaller values result in larger forests which have a better generalization error. However, larger forests cause slower training. By default, this value is calculated during hyperparameter optimization. It must be a value between 0.001 and 1.

        • Advanced configuration option. Specifies the rate at which eta increases for each new tree that is added to the forest. For example, a rate of 1.05 increases eta by 5% for each extra tree. By default, this value is calculated during hyperparameter optimization. It must be between 0.5 and 2.

        • Advanced configuration option. Defines the fraction of features that will be used when selecting a random bag for each candidate split. By default, this value is calculated during hyperparameter optimization.

        • feature_processors array[object]

          Advanced configuration option. A collection of feature preprocessors that modify one or more included fields. The analysis uses the resulting one or more features instead of the original document field. However, these features are ephemeral; they are not stored in the destination index. Multiple feature_processors entries can refer to the same document fields. Automatic categorical feature encoding still occurs for the fields that are unprocessed by a custom processor or that have categorical values. Use this property only if you want to override the automatic feature encoding of the specified fields.

          Hide feature_processors attributes Show feature_processors attributes object
        • gamma number

          Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies a linear penalty associated with the size of individual trees in the forest. A high gamma value causes training to prefer small trees. A small gamma value results in larger individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

        • lambda number

          Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies an L2 regularization term which applies to leaf weights of the individual trees in the forest. A high lambda value causes training to favor small leaf weights. This behavior makes the prediction function smoother at the expense of potentially not being able to capture relevant relationships between the features and the dependent variable. A small lambda value results in large individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

        • Advanced configuration option. A multiplier responsible for determining the maximum number of hyperparameter optimization steps in the Bayesian optimization procedure. The maximum number of steps is determined based on the number of undefined hyperparameters times the maximum optimization rounds per hyperparameter. By default, this value is calculated during hyperparameter optimization.

        • Advanced configuration option. Defines the maximum number of decision trees in the forest. The maximum value is 2000. By default, this value is calculated during hyperparameter optimization.

        • Advanced configuration option. Specifies the maximum number of feature importance values per document to return. By default, no feature importance calculation occurs.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Defines the seed for the random generator that is used to pick training data. By default, it is randomly generated. Set it to a specific value to use the same training data each time you start a job (assuming other related parameters such as source and analyzed_fields are the same).

        • Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This soft limit combines with the soft_tree_depth_tolerance to penalize trees that exceed the specified depth; the regularized loss increases quickly beyond this depth. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.

        • Advanced configuration option. This option controls how quickly the regularized loss increases when the tree depth exceeds soft_tree_depth_limit. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.01.

        • Defines the number of categories for which the predicted probabilities are reported. It must be non-negative or -1. If it is -1 or greater than the total number of categories, probabilities are reported for all categories; if you have a large number of categories, there could be a significant effect on the size of your destination index. NOTE: To use the AUC ROC evaluation method, num_top_classes must be set to -1 or a value greater than or equal to the total number of categories.

      • Additional properties are allowed.

        Hide outlier_detection attributes Show outlier_detection attributes object
        • Specifies whether the feature influence calculation is enabled.

        • The minimum outlier score that a document needs to have in order to calculate its feature influence score. Value range: 0-1.

        • method string

          The method that outlier detection uses. Available methods are lof, ldof, distance_kth_nn, distance_knn, and ensemble. The default value is ensemble, which means that outlier detection uses an ensemble of different methods and normalises and combines their individual outlier scores to obtain the overall outlier score.

        • Defines the value for how many nearest neighbors each method of outlier detection uses to calculate its outlier score. When the value is not set, different values are used for different ensemble members. This default behavior helps improve the diversity in the ensemble; only override it if you are confident that the value you choose is appropriate for the data set.

        • The proportion of the data set that is assumed to be outlying prior to outlier detection. For example, 0.05 means it is assumed that 5% of values are real outliers and 95% are inliers.

        • If true, the following operation is performed on the columns before computing outlier scores: (x_i - mean(x_i)) / sd(x_i).

      • Additional properties are allowed.

        Hide regression attributes Show regression attributes object
        • alpha number

          Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This parameter affects loss calculations by acting as a multiplier of the tree depth. Higher alpha values result in shallower trees and faster training times. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to zero.

        • dependent_variable string Required

          Defines which field of the document is to be predicted. It must match one of the fields in the index being used to train. If this field is missing from a document, then that document will not be used for training, but a prediction with the trained model will be generated for it. It is also known as continuous target variable. For classification analysis, the data type of the field must be numeric (integer, short, long, byte), categorical (ip or keyword), or boolean. There must be no more than 30 different values in this field. For regression analysis, the data type of the field must be numeric.

        • Advanced configuration option. Controls the fraction of data that is used to compute the derivatives of the loss function for tree training. A small value results in the use of a small fraction of the data. If this value is set to be less than 1, accuracy typically improves. However, too small a value may result in poor convergence for the ensemble and so require more trees. By default, this value is calculated during hyperparameter optimization. It must be greater than zero and less than or equal to 1.

        • Advanced configuration option. Specifies whether the training process should finish if it is not finding any better performing models. If disabled, the training process can take significantly longer and the chance of finding a better performing model is unremarkable.

        • eta number

          Advanced configuration option. The shrinkage applied to the weights. Smaller values result in larger forests which have a better generalization error. However, larger forests cause slower training. By default, this value is calculated during hyperparameter optimization. It must be a value between 0.001 and 1.

        • Advanced configuration option. Specifies the rate at which eta increases for each new tree that is added to the forest. For example, a rate of 1.05 increases eta by 5% for each extra tree. By default, this value is calculated during hyperparameter optimization. It must be between 0.5 and 2.

        • Advanced configuration option. Defines the fraction of features that will be used when selecting a random bag for each candidate split. By default, this value is calculated during hyperparameter optimization.

        • feature_processors array[object]

          Advanced configuration option. A collection of feature preprocessors that modify one or more included fields. The analysis uses the resulting one or more features instead of the original document field. However, these features are ephemeral; they are not stored in the destination index. Multiple feature_processors entries can refer to the same document fields. Automatic categorical feature encoding still occurs for the fields that are unprocessed by a custom processor or that have categorical values. Use this property only if you want to override the automatic feature encoding of the specified fields.

          Hide feature_processors attributes Show feature_processors attributes object
        • gamma number

          Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies a linear penalty associated with the size of individual trees in the forest. A high gamma value causes training to prefer small trees. A small gamma value results in larger individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

        • lambda number

          Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies an L2 regularization term which applies to leaf weights of the individual trees in the forest. A high lambda value causes training to favor small leaf weights. This behavior makes the prediction function smoother at the expense of potentially not being able to capture relevant relationships between the features and the dependent variable. A small lambda value results in large individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

        • Advanced configuration option. A multiplier responsible for determining the maximum number of hyperparameter optimization steps in the Bayesian optimization procedure. The maximum number of steps is determined based on the number of undefined hyperparameters times the maximum optimization rounds per hyperparameter. By default, this value is calculated during hyperparameter optimization.

        • Advanced configuration option. Defines the maximum number of decision trees in the forest. The maximum value is 2000. By default, this value is calculated during hyperparameter optimization.

        • Advanced configuration option. Specifies the maximum number of feature importance values per document to return. By default, no feature importance calculation occurs.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Defines the seed for the random generator that is used to pick training data. By default, it is randomly generated. Set it to a specific value to use the same training data each time you start a job (assuming other related parameters such as source and analyzed_fields are the same).

        • Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This soft limit combines with the soft_tree_depth_tolerance to penalize trees that exceed the specified depth; the regularized loss increases quickly beyond this depth. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.

        • Advanced configuration option. This option controls how quickly the regularized loss increases when the tree depth exceeds soft_tree_depth_limit. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.01.

        • The loss function used during regression. Available options are mse (mean squared error), msle (mean squared logarithmic error), huber (Pseudo-Huber loss).

        • A positive number that is used as a parameter to the loss_function.

    • Additional properties are allowed.

      Hide analyzed_fields attributes Show analyzed_fields attributes object
      • includes array[string] Required

        An array of strings that defines the fields that will be excluded from the analysis. You do not need to add fields with unsupported data types to excludes, these fields are excluded from the analysis automatically.

      • excludes array[string] Required

        An array of strings that defines the fields that will be included in the analysis.

    • Time unit for milliseconds

    • dest object Required

      Additional properties are allowed.

      Hide dest attributes Show dest attributes object
      • index string Required
      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

    • id string Required
    • max_num_threads number Required
    • _meta object
      Hide _meta attribute Show _meta attribute object
      • * object Additional properties

        Additional properties are allowed.

    • model_memory_limit string Required
    • source object Required

      Additional properties are allowed.

      Hide source attributes Show source attributes object
      • index string | array[string] Required
      • Hide runtime_mappings attribute Show runtime_mappings attribute object
        • * object Additional properties

          Additional properties are allowed.

          Hide * attributes Show * attributes object
          • fields object

            For type composite

            Hide fields attribute Show fields attribute object
            • * object Additional properties

              Additional properties are allowed.

              Hide * attribute Show * attribute object
              • type string Required

                Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

          • fetch_fields array[object]

            For type lookup

            Hide fetch_fields attributes Show fetch_fields attributes object
            • field string Required

              Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

            • format string
          • format string

            A custom format for date type runtime fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • script object

            Additional properties are allowed.

            Hide script attributes Show script attributes object
            • source string

              The script source.

            • id string
            • params object

              Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

              Hide params attribute Show params attribute object
              • * object Additional properties

                Additional properties are allowed.

            • lang string

              Any of:

              Values are painless, expression, mustache, or java.

            • options object
              Hide options attribute Show options attribute object
              • * string Additional properties
          • type string Required

            Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

      • _source object

        Additional properties are allowed.

        Hide _source attributes Show _source attributes object
        • includes array[string] Required

          An array of strings that defines the fields that will be excluded from the analysis. You do not need to add fields with unsupported data types to excludes, these fields are excluded from the analysis automatically.

        • excludes array[string] Required

          An array of strings that defines the fields that will be included in the analysis.

      • query object

        The Elasticsearch query domain-specific language (DSL). This value corresponds to the query object in an Elasticsearch search POST body. All the options that are supported by Elasticsearch can be used, as this object is passed verbatim to Elasticsearch. By default, this property has the following value: {"match_all": {}}.

        Additional properties are allowed.

    • version string Required
PUT /_ml/data_frame/analytics/{id}
curl \
 -X PUT http://api.example.com/_ml/data_frame/analytics/{id} \
 -H "Content-Type: application/json" \
 -d '{"allow_lazy_start":true,"analysis":{"":{"alpha":42.0,"dependent_variable":"string","downsample_factor":42.0,"early_stopping_enabled":true,"eta":42.0,"eta_growth_rate_per_tree":42.0,"feature_bag_fraction":42.0,"feature_processors":[{"frequency_encoding":{"feature_name":"string","field":"string","frequency_map":{}},"multi_encoding":{"processors":[42.0]},"n_gram_encoding":{"feature_prefix":"string","field":"string","length":42.0,"n_grams":[42.0],"start":42.0,"custom":true},"one_hot_encoding":{"field":"string","hot_map":"string"},"target_mean_encoding":{"default_value":42.0,"feature_name":"string","field":"string","target_map":{}}}],"gamma":42.0,"lambda":42.0,"max_optimization_rounds_per_hyperparameter":42.0,"max_trees":42.0,"num_top_feature_importance_values":42.0,"prediction_field_name":"string","randomize_seed":42.0,"soft_tree_depth_limit":42.0,"soft_tree_depth_tolerance":42.0,"":"string","loss_function":"string","loss_function_parameter":42.0},"outlier_detection":{"compute_feature_influence":true,"feature_influence_threshold":42.0,"method":"string","n_neighbors":42.0,"outlier_fraction":42.0,"standardization_enabled":true}},"analyzed_fields":{"includes":["string"],"excludes":["string"]},"description":"string","dest":{"index":"string","results_field":"string"},"max_num_threads":42.0,"_meta":{"additionalProperty1":{},"additionalProperty2":{}},"model_memory_limit":"string","source":{"index":"string","runtime_mappings":{"additionalProperty1":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"},"additionalProperty2":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"}},"_source":{"includes":["string"],"excludes":["string"]},"query":{}},"headers":{},"version":"string"}'
Request examples
{
  "allow_lazy_start": true,
  "analysis": {
    "": {
      "alpha": 42.0,
      "dependent_variable": "string",
      "downsample_factor": 42.0,
      "early_stopping_enabled": true,
      "eta": 42.0,
      "eta_growth_rate_per_tree": 42.0,
      "feature_bag_fraction": 42.0,
      "feature_processors": [
        {
          "frequency_encoding": {
            "feature_name": "string",
            "field": "string",
            "frequency_map": {}
          },
          "multi_encoding": {
            "processors": [
              42.0
            ]
          },
          "n_gram_encoding": {
            "feature_prefix": "string",
            "field": "string",
            "length": 42.0,
            "n_grams": [
              42.0
            ],
            "start": 42.0,
            "custom": true
          },
          "one_hot_encoding": {
            "field": "string",
            "hot_map": "string"
          },
          "target_mean_encoding": {
            "default_value": 42.0,
            "feature_name": "string",
            "field": "string",
            "target_map": {}
          }
        }
      ],
      "gamma": 42.0,
      "lambda": 42.0,
      "max_optimization_rounds_per_hyperparameter": 42.0,
      "max_trees": 42.0,
      "num_top_feature_importance_values": 42.0,
      "prediction_field_name": "string",
      "randomize_seed": 42.0,
      "soft_tree_depth_limit": 42.0,
      "soft_tree_depth_tolerance": 42.0,
      "": "string",
      "loss_function": "string",
      "loss_function_parameter": 42.0
    },
    "outlier_detection": {
      "compute_feature_influence": true,
      "feature_influence_threshold": 42.0,
      "method": "string",
      "n_neighbors": 42.0,
      "outlier_fraction": 42.0,
      "standardization_enabled": true
    }
  },
  "analyzed_fields": {
    "includes": [
      "string"
    ],
    "excludes": [
      "string"
    ]
  },
  "description": "string",
  "dest": {
    "index": "string",
    "results_field": "string"
  },
  "max_num_threads": 42.0,
  "_meta": {
    "additionalProperty1": {},
    "additionalProperty2": {}
  },
  "model_memory_limit": "string",
  "source": {
    "index": "string",
    "runtime_mappings": {
      "additionalProperty1": {
        "fields": {
          "additionalProperty1": {
            "type": "boolean"
          },
          "additionalProperty2": {
            "type": "boolean"
          }
        },
        "fetch_fields": [
          {
            "field": "string",
            "format": "string"
          }
        ],
        "format": "string",
        "input_field": "string",
        "target_field": "string",
        "target_index": "string",
        "script": {
          "source": "string",
          "id": "string",
          "params": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "": "painless",
          "options": {
            "additionalProperty1": "string",
            "additionalProperty2": "string"
          }
        },
        "type": "boolean"
      },
      "additionalProperty2": {
        "fields": {
          "additionalProperty1": {
            "type": "boolean"
          },
          "additionalProperty2": {
            "type": "boolean"
          }
        },
        "fetch_fields": [
          {
            "field": "string",
            "format": "string"
          }
        ],
        "format": "string",
        "input_field": "string",
        "target_field": "string",
        "target_index": "string",
        "script": {
          "source": "string",
          "id": "string",
          "params": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "": "painless",
          "options": {
            "additionalProperty1": "string",
            "additionalProperty2": "string"
          }
        },
        "type": "boolean"
      }
    },
    "_source": {
      "includes": [
        "string"
      ],
      "excludes": [
        "string"
      ]
    },
    "query": {}
  },
  "headers": {},
  "version": "string"
}
Response examples (200)
{
  "authorization": {
    "api_key": {
      "id": "string",
      "name": "string"
    },
    "roles": [
      "string"
    ],
    "service_account": "string"
  },
  "allow_lazy_start": true,
  "analysis": {
    "": {
      "alpha": 42.0,
      "dependent_variable": "string",
      "downsample_factor": 42.0,
      "early_stopping_enabled": true,
      "eta": 42.0,
      "eta_growth_rate_per_tree": 42.0,
      "feature_bag_fraction": 42.0,
      "feature_processors": [
        {
          "frequency_encoding": {},
          "multi_encoding": {},
          "n_gram_encoding": {},
          "one_hot_encoding": {},
          "target_mean_encoding": {}
        }
      ],
      "gamma": 42.0,
      "lambda": 42.0,
      "max_optimization_rounds_per_hyperparameter": 42.0,
      "max_trees": 42.0,
      "num_top_feature_importance_values": 42.0,
      "prediction_field_name": "string",
      "randomize_seed": 42.0,
      "soft_tree_depth_limit": 42.0,
      "soft_tree_depth_tolerance": 42.0,
      "": "string",
      "loss_function": "string",
      "loss_function_parameter": 42.0
    },
    "outlier_detection": {
      "compute_feature_influence": true,
      "feature_influence_threshold": 42.0,
      "method": "string",
      "n_neighbors": 42.0,
      "outlier_fraction": 42.0,
      "standardization_enabled": true
    }
  },
  "analyzed_fields": {
    "includes": [
      "string"
    ],
    "excludes": [
      "string"
    ]
  },
  "": 42.0,
  "description": "string",
  "dest": {
    "index": "string",
    "results_field": "string"
  },
  "id": "string",
  "max_num_threads": 42.0,
  "_meta": {
    "additionalProperty1": {},
    "additionalProperty2": {}
  },
  "model_memory_limit": "string",
  "source": {
    "index": "string",
    "runtime_mappings": {
      "additionalProperty1": {
        "fields": {
          "additionalProperty1": {
            "type": "boolean"
          },
          "additionalProperty2": {
            "type": "boolean"
          }
        },
        "fetch_fields": [
          {
            "field": "string",
            "format": "string"
          }
        ],
        "format": "string",
        "input_field": "string",
        "target_field": "string",
        "target_index": "string",
        "script": {
          "source": "string",
          "id": "string",
          "params": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "": "painless",
          "options": {
            "additionalProperty1": "string",
            "additionalProperty2": "string"
          }
        },
        "type": "boolean"
      },
      "additionalProperty2": {
        "fields": {
          "additionalProperty1": {
            "type": "boolean"
          },
          "additionalProperty2": {
            "type": "boolean"
          }
        },
        "fetch_fields": [
          {
            "field": "string",
            "format": "string"
          }
        ],
        "format": "string",
        "input_field": "string",
        "target_field": "string",
        "target_index": "string",
        "script": {
          "source": "string",
          "id": "string",
          "params": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "": "painless",
          "options": {
            "additionalProperty1": "string",
            "additionalProperty2": "string"
          }
        },
        "type": "boolean"
      }
    },
    "_source": {
      "includes": [
        "string"
      ],
      "excludes": [
        "string"
      ]
    },
    "query": {}
  },
  "version": "string"
}




































Preview features used by data frame analytics Added in 7.13.0

GET /_ml/data_frame/analytics/_preview

Previews the extracted features used by a data frame analytics config.

application/json

Body

  • config object

    Additional properties are allowed.

    Hide config attributes Show config attributes object
    • source object Required

      Additional properties are allowed.

      Hide source attributes Show source attributes object
      • index string | array[string] Required
      • Hide runtime_mappings attribute Show runtime_mappings attribute object
        • * object Additional properties

          Additional properties are allowed.

          Hide * attributes Show * attributes object
          • fields object

            For type composite

            Hide fields attribute Show fields attribute object
            • * object Additional properties

              Additional properties are allowed.

              Hide * attribute Show * attribute object
              • type string Required

                Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

          • fetch_fields array[object]

            For type lookup

            Hide fetch_fields attributes Show fetch_fields attributes object
            • field string Required

              Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

            • format string
          • format string

            A custom format for date type runtime fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • script object

            Additional properties are allowed.

            Hide script attributes Show script attributes object
            • source string

              The script source.

            • id string
            • params object

              Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

              Hide params attribute Show params attribute object
              • * object Additional properties

                Additional properties are allowed.

            • lang string

              Any of:

              Values are painless, expression, mustache, or java.

            • options object
              Hide options attribute Show options attribute object
              • * string Additional properties
          • type string Required

            Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

      • _source object

        Additional properties are allowed.

        Hide _source attributes Show _source attributes object
        • includes array[string] Required

          An array of strings that defines the fields that will be excluded from the analysis. You do not need to add fields with unsupported data types to excludes, these fields are excluded from the analysis automatically.

        • excludes array[string] Required

          An array of strings that defines the fields that will be included in the analysis.

      • query object

        The Elasticsearch query domain-specific language (DSL). This value corresponds to the query object in an Elasticsearch search POST body. All the options that are supported by Elasticsearch can be used, as this object is passed verbatim to Elasticsearch. By default, this property has the following value: {"match_all": {}}.

        Additional properties are allowed.

    • analysis object Required

      Additional properties are allowed.

      Hide analysis attributes Show analysis attributes object
      • Additional properties are allowed.

        Hide classification attributes Show classification attributes object
        • alpha number

          Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This parameter affects loss calculations by acting as a multiplier of the tree depth. Higher alpha values result in shallower trees and faster training times. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to zero.

        • dependent_variable string Required

          Defines which field of the document is to be predicted. It must match one of the fields in the index being used to train. If this field is missing from a document, then that document will not be used for training, but a prediction with the trained model will be generated for it. It is also known as continuous target variable. For classification analysis, the data type of the field must be numeric (integer, short, long, byte), categorical (ip or keyword), or boolean. There must be no more than 30 different values in this field. For regression analysis, the data type of the field must be numeric.

        • Advanced configuration option. Controls the fraction of data that is used to compute the derivatives of the loss function for tree training. A small value results in the use of a small fraction of the data. If this value is set to be less than 1, accuracy typically improves. However, too small a value may result in poor convergence for the ensemble and so require more trees. By default, this value is calculated during hyperparameter optimization. It must be greater than zero and less than or equal to 1.

        • Advanced configuration option. Specifies whether the training process should finish if it is not finding any better performing models. If disabled, the training process can take significantly longer and the chance of finding a better performing model is unremarkable.

        • eta number

          Advanced configuration option. The shrinkage applied to the weights. Smaller values result in larger forests which have a better generalization error. However, larger forests cause slower training. By default, this value is calculated during hyperparameter optimization. It must be a value between 0.001 and 1.

        • Advanced configuration option. Specifies the rate at which eta increases for each new tree that is added to the forest. For example, a rate of 1.05 increases eta by 5% for each extra tree. By default, this value is calculated during hyperparameter optimization. It must be between 0.5 and 2.

        • Advanced configuration option. Defines the fraction of features that will be used when selecting a random bag for each candidate split. By default, this value is calculated during hyperparameter optimization.

        • feature_processors array[object]

          Advanced configuration option. A collection of feature preprocessors that modify one or more included fields. The analysis uses the resulting one or more features instead of the original document field. However, these features are ephemeral; they are not stored in the destination index. Multiple feature_processors entries can refer to the same document fields. Automatic categorical feature encoding still occurs for the fields that are unprocessed by a custom processor or that have categorical values. Use this property only if you want to override the automatic feature encoding of the specified fields.

          Hide feature_processors attributes Show feature_processors attributes object
        • gamma number

          Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies a linear penalty associated with the size of individual trees in the forest. A high gamma value causes training to prefer small trees. A small gamma value results in larger individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

        • lambda number

          Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies an L2 regularization term which applies to leaf weights of the individual trees in the forest. A high lambda value causes training to favor small leaf weights. This behavior makes the prediction function smoother at the expense of potentially not being able to capture relevant relationships between the features and the dependent variable. A small lambda value results in large individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

        • Advanced configuration option. A multiplier responsible for determining the maximum number of hyperparameter optimization steps in the Bayesian optimization procedure. The maximum number of steps is determined based on the number of undefined hyperparameters times the maximum optimization rounds per hyperparameter. By default, this value is calculated during hyperparameter optimization.

        • Advanced configuration option. Defines the maximum number of decision trees in the forest. The maximum value is 2000. By default, this value is calculated during hyperparameter optimization.

        • Advanced configuration option. Specifies the maximum number of feature importance values per document to return. By default, no feature importance calculation occurs.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Defines the seed for the random generator that is used to pick training data. By default, it is randomly generated. Set it to a specific value to use the same training data each time you start a job (assuming other related parameters such as source and analyzed_fields are the same).

        • Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This soft limit combines with the soft_tree_depth_tolerance to penalize trees that exceed the specified depth; the regularized loss increases quickly beyond this depth. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.

        • Advanced configuration option. This option controls how quickly the regularized loss increases when the tree depth exceeds soft_tree_depth_limit. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.01.

        • Defines the number of categories for which the predicted probabilities are reported. It must be non-negative or -1. If it is -1 or greater than the total number of categories, probabilities are reported for all categories; if you have a large number of categories, there could be a significant effect on the size of your destination index. NOTE: To use the AUC ROC evaluation method, num_top_classes must be set to -1 or a value greater than or equal to the total number of categories.

      • Additional properties are allowed.

        Hide outlier_detection attributes Show outlier_detection attributes object
        • Specifies whether the feature influence calculation is enabled.

        • The minimum outlier score that a document needs to have in order to calculate its feature influence score. Value range: 0-1.

        • method string

          The method that outlier detection uses. Available methods are lof, ldof, distance_kth_nn, distance_knn, and ensemble. The default value is ensemble, which means that outlier detection uses an ensemble of different methods and normalises and combines their individual outlier scores to obtain the overall outlier score.

        • Defines the value for how many nearest neighbors each method of outlier detection uses to calculate its outlier score. When the value is not set, different values are used for different ensemble members. This default behavior helps improve the diversity in the ensemble; only override it if you are confident that the value you choose is appropriate for the data set.

        • The proportion of the data set that is assumed to be outlying prior to outlier detection. For example, 0.05 means it is assumed that 5% of values are real outliers and 95% are inliers.

        • If true, the following operation is performed on the columns before computing outlier scores: (x_i - mean(x_i)) / sd(x_i).

      • Additional properties are allowed.

        Hide regression attributes Show regression attributes object
        • alpha number

          Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This parameter affects loss calculations by acting as a multiplier of the tree depth. Higher alpha values result in shallower trees and faster training times. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to zero.

        • dependent_variable string Required

          Defines which field of the document is to be predicted. It must match one of the fields in the index being used to train. If this field is missing from a document, then that document will not be used for training, but a prediction with the trained model will be generated for it. It is also known as continuous target variable. For classification analysis, the data type of the field must be numeric (integer, short, long, byte), categorical (ip or keyword), or boolean. There must be no more than 30 different values in this field. For regression analysis, the data type of the field must be numeric.

        • Advanced configuration option. Controls the fraction of data that is used to compute the derivatives of the loss function for tree training. A small value results in the use of a small fraction of the data. If this value is set to be less than 1, accuracy typically improves. However, too small a value may result in poor convergence for the ensemble and so require more trees. By default, this value is calculated during hyperparameter optimization. It must be greater than zero and less than or equal to 1.

        • Advanced configuration option. Specifies whether the training process should finish if it is not finding any better performing models. If disabled, the training process can take significantly longer and the chance of finding a better performing model is unremarkable.

        • eta number

          Advanced configuration option. The shrinkage applied to the weights. Smaller values result in larger forests which have a better generalization error. However, larger forests cause slower training. By default, this value is calculated during hyperparameter optimization. It must be a value between 0.001 and 1.

        • Advanced configuration option. Specifies the rate at which eta increases for each new tree that is added to the forest. For example, a rate of 1.05 increases eta by 5% for each extra tree. By default, this value is calculated during hyperparameter optimization. It must be between 0.5 and 2.

        • Advanced configuration option. Defines the fraction of features that will be used when selecting a random bag for each candidate split. By default, this value is calculated during hyperparameter optimization.

        • feature_processors array[object]

          Advanced configuration option. A collection of feature preprocessors that modify one or more included fields. The analysis uses the resulting one or more features instead of the original document field. However, these features are ephemeral; they are not stored in the destination index. Multiple feature_processors entries can refer to the same document fields. Automatic categorical feature encoding still occurs for the fields that are unprocessed by a custom processor or that have categorical values. Use this property only if you want to override the automatic feature encoding of the specified fields.

          Hide feature_processors attributes Show feature_processors attributes object
        • gamma number

          Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies a linear penalty associated with the size of individual trees in the forest. A high gamma value causes training to prefer small trees. A small gamma value results in larger individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

        • lambda number

          Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies an L2 regularization term which applies to leaf weights of the individual trees in the forest. A high lambda value causes training to favor small leaf weights. This behavior makes the prediction function smoother at the expense of potentially not being able to capture relevant relationships between the features and the dependent variable. A small lambda value results in large individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

        • Advanced configuration option. A multiplier responsible for determining the maximum number of hyperparameter optimization steps in the Bayesian optimization procedure. The maximum number of steps is determined based on the number of undefined hyperparameters times the maximum optimization rounds per hyperparameter. By default, this value is calculated during hyperparameter optimization.

        • Advanced configuration option. Defines the maximum number of decision trees in the forest. The maximum value is 2000. By default, this value is calculated during hyperparameter optimization.

        • Advanced configuration option. Specifies the maximum number of feature importance values per document to return. By default, no feature importance calculation occurs.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Defines the seed for the random generator that is used to pick training data. By default, it is randomly generated. Set it to a specific value to use the same training data each time you start a job (assuming other related parameters such as source and analyzed_fields are the same).

        • Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This soft limit combines with the soft_tree_depth_tolerance to penalize trees that exceed the specified depth; the regularized loss increases quickly beyond this depth. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.

        • Advanced configuration option. This option controls how quickly the regularized loss increases when the tree depth exceeds soft_tree_depth_limit. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.01.

        • The loss function used during regression. Available options are mse (mean squared error), msle (mean squared logarithmic error), huber (Pseudo-Huber loss).

        • A positive number that is used as a parameter to the loss_function.

    • Additional properties are allowed.

      Hide analyzed_fields attributes Show analyzed_fields attributes object
      • includes array[string] Required

        An array of strings that defines the fields that will be excluded from the analysis. You do not need to add fields with unsupported data types to excludes, these fields are excluded from the analysis automatically.

      • excludes array[string] Required

        An array of strings that defines the fields that will be included in the analysis.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • feature_values array[object] Required

      An array of objects that contain feature name and value pairs. The features have been processed and indicate what will be sent to the model for training.

      Hide feature_values attribute Show feature_values attribute object
      • * string Additional properties
GET /_ml/data_frame/analytics/_preview
curl \
 -X GET http://api.example.com/_ml/data_frame/analytics/_preview \
 -H "Content-Type: application/json" \
 -d '{"config":{"source":{"index":"string","runtime_mappings":{"additionalProperty1":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"},"additionalProperty2":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"}},"_source":{"includes":["string"],"excludes":["string"]},"query":{}},"analysis":{"":{"alpha":42.0,"dependent_variable":"string","downsample_factor":42.0,"early_stopping_enabled":true,"eta":42.0,"eta_growth_rate_per_tree":42.0,"feature_bag_fraction":42.0,"feature_processors":[{"frequency_encoding":{},"multi_encoding":{},"n_gram_encoding":{},"one_hot_encoding":{},"target_mean_encoding":{}}],"gamma":42.0,"lambda":42.0,"max_optimization_rounds_per_hyperparameter":42.0,"max_trees":42.0,"num_top_feature_importance_values":42.0,"prediction_field_name":"string","randomize_seed":42.0,"soft_tree_depth_limit":42.0,"soft_tree_depth_tolerance":42.0,"":"string","loss_function":"string","loss_function_parameter":42.0},"outlier_detection":{"compute_feature_influence":true,"feature_influence_threshold":42.0,"method":"string","n_neighbors":42.0,"outlier_fraction":42.0,"standardization_enabled":true}},"model_memory_limit":"string","max_num_threads":42.0,"analyzed_fields":{"includes":["string"],"excludes":["string"]}}}'
Request examples
{
  "config": {
    "source": {
      "index": "string",
      "runtime_mappings": {
        "additionalProperty1": {
          "fields": {
            "additionalProperty1": {
              "type": "boolean"
            },
            "additionalProperty2": {
              "type": "boolean"
            }
          },
          "fetch_fields": [
            {
              "field": "string",
              "format": "string"
            }
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {
              "additionalProperty1": {},
              "additionalProperty2": {}
            },
            "": "painless",
            "options": {
              "additionalProperty1": "string",
              "additionalProperty2": "string"
            }
          },
          "type": "boolean"
        },
        "additionalProperty2": {
          "fields": {
            "additionalProperty1": {
              "type": "boolean"
            },
            "additionalProperty2": {
              "type": "boolean"
            }
          },
          "fetch_fields": [
            {
              "field": "string",
              "format": "string"
            }
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {
              "additionalProperty1": {},
              "additionalProperty2": {}
            },
            "": "painless",
            "options": {
              "additionalProperty1": "string",
              "additionalProperty2": "string"
            }
          },
          "type": "boolean"
        }
      },
      "_source": {
        "includes": [
          "string"
        ],
        "excludes": [
          "string"
        ]
      },
      "query": {}
    },
    "analysis": {
      "": {
        "alpha": 42.0,
        "dependent_variable": "string",
        "downsample_factor": 42.0,
        "early_stopping_enabled": true,
        "eta": 42.0,
        "eta_growth_rate_per_tree": 42.0,
        "feature_bag_fraction": 42.0,
        "feature_processors": [
          {
            "frequency_encoding": {},
            "multi_encoding": {},
            "n_gram_encoding": {},
            "one_hot_encoding": {},
            "target_mean_encoding": {}
          }
        ],
        "gamma": 42.0,
        "lambda": 42.0,
        "max_optimization_rounds_per_hyperparameter": 42.0,
        "max_trees": 42.0,
        "num_top_feature_importance_values": 42.0,
        "prediction_field_name": "string",
        "randomize_seed": 42.0,
        "soft_tree_depth_limit": 42.0,
        "soft_tree_depth_tolerance": 42.0,
        "": "string",
        "loss_function": "string",
        "loss_function_parameter": 42.0
      },
      "outlier_detection": {
        "compute_feature_influence": true,
        "feature_influence_threshold": 42.0,
        "method": "string",
        "n_neighbors": 42.0,
        "outlier_fraction": 42.0,
        "standardization_enabled": true
      }
    },
    "model_memory_limit": "string",
    "max_num_threads": 42.0,
    "analyzed_fields": {
      "includes": [
        "string"
      ],
      "excludes": [
        "string"
      ]
    }
  }
}
Response examples (200)
{
  "feature_values": [
    {
      "additionalProperty1": "string",
      "additionalProperty2": "string"
    }
  ]
}

Preview features used by data frame analytics Added in 7.13.0

POST /_ml/data_frame/analytics/_preview

Previews the extracted features used by a data frame analytics config.

application/json

Body

  • config object

    Additional properties are allowed.

    Hide config attributes Show config attributes object
    • source object Required

      Additional properties are allowed.

      Hide source attributes Show source attributes object
      • index string | array[string] Required
      • Hide runtime_mappings attribute Show runtime_mappings attribute object
        • * object Additional properties

          Additional properties are allowed.

          Hide * attributes Show * attributes object
          • fields object

            For type composite

            Hide fields attribute Show fields attribute object
            • * object Additional properties

              Additional properties are allowed.

              Hide * attribute Show * attribute object
              • type string Required

                Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

          • fetch_fields array[object]

            For type lookup

            Hide fetch_fields attributes Show fetch_fields attributes object
            • field string Required

              Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

            • format string
          • format string

            A custom format for date type runtime fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • script object

            Additional properties are allowed.

            Hide script attributes Show script attributes object
            • source string

              The script source.

            • id string
            • params object

              Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

              Hide params attribute Show params attribute object
              • * object Additional properties

                Additional properties are allowed.

            • lang string

              Any of:

              Values are painless, expression, mustache, or java.

            • options object
              Hide options attribute Show options attribute object
              • * string Additional properties
          • type string Required

            Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

      • _source object

        Additional properties are allowed.

        Hide _source attributes Show _source attributes object
        • includes array[string] Required

          An array of strings that defines the fields that will be excluded from the analysis. You do not need to add fields with unsupported data types to excludes, these fields are excluded from the analysis automatically.

        • excludes array[string] Required

          An array of strings that defines the fields that will be included in the analysis.

      • query object

        The Elasticsearch query domain-specific language (DSL). This value corresponds to the query object in an Elasticsearch search POST body. All the options that are supported by Elasticsearch can be used, as this object is passed verbatim to Elasticsearch. By default, this property has the following value: {"match_all": {}}.

        Additional properties are allowed.

    • analysis object Required

      Additional properties are allowed.

      Hide analysis attributes Show analysis attributes object
      • Additional properties are allowed.

        Hide classification attributes Show classification attributes object
        • alpha number

          Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This parameter affects loss calculations by acting as a multiplier of the tree depth. Higher alpha values result in shallower trees and faster training times. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to zero.

        • dependent_variable string Required

          Defines which field of the document is to be predicted. It must match one of the fields in the index being used to train. If this field is missing from a document, then that document will not be used for training, but a prediction with the trained model will be generated for it. It is also known as continuous target variable. For classification analysis, the data type of the field must be numeric (integer, short, long, byte), categorical (ip or keyword), or boolean. There must be no more than 30 different values in this field. For regression analysis, the data type of the field must be numeric.

        • Advanced configuration option. Controls the fraction of data that is used to compute the derivatives of the loss function for tree training. A small value results in the use of a small fraction of the data. If this value is set to be less than 1, accuracy typically improves. However, too small a value may result in poor convergence for the ensemble and so require more trees. By default, this value is calculated during hyperparameter optimization. It must be greater than zero and less than or equal to 1.

        • Advanced configuration option. Specifies whether the training process should finish if it is not finding any better performing models. If disabled, the training process can take significantly longer and the chance of finding a better performing model is unremarkable.

        • eta number

          Advanced configuration option. The shrinkage applied to the weights. Smaller values result in larger forests which have a better generalization error. However, larger forests cause slower training. By default, this value is calculated during hyperparameter optimization. It must be a value between 0.001 and 1.

        • Advanced configuration option. Specifies the rate at which eta increases for each new tree that is added to the forest. For example, a rate of 1.05 increases eta by 5% for each extra tree. By default, this value is calculated during hyperparameter optimization. It must be between 0.5 and 2.

        • Advanced configuration option. Defines the fraction of features that will be used when selecting a random bag for each candidate split. By default, this value is calculated during hyperparameter optimization.

        • feature_processors array[object]

          Advanced configuration option. A collection of feature preprocessors that modify one or more included fields. The analysis uses the resulting one or more features instead of the original document field. However, these features are ephemeral; they are not stored in the destination index. Multiple feature_processors entries can refer to the same document fields. Automatic categorical feature encoding still occurs for the fields that are unprocessed by a custom processor or that have categorical values. Use this property only if you want to override the automatic feature encoding of the specified fields.

          Hide feature_processors attributes Show feature_processors attributes object
        • gamma number

          Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies a linear penalty associated with the size of individual trees in the forest. A high gamma value causes training to prefer small trees. A small gamma value results in larger individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

        • lambda number

          Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies an L2 regularization term which applies to leaf weights of the individual trees in the forest. A high lambda value causes training to favor small leaf weights. This behavior makes the prediction function smoother at the expense of potentially not being able to capture relevant relationships between the features and the dependent variable. A small lambda value results in large individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

        • Advanced configuration option. A multiplier responsible for determining the maximum number of hyperparameter optimization steps in the Bayesian optimization procedure. The maximum number of steps is determined based on the number of undefined hyperparameters times the maximum optimization rounds per hyperparameter. By default, this value is calculated during hyperparameter optimization.

        • Advanced configuration option. Defines the maximum number of decision trees in the forest. The maximum value is 2000. By default, this value is calculated during hyperparameter optimization.

        • Advanced configuration option. Specifies the maximum number of feature importance values per document to return. By default, no feature importance calculation occurs.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Defines the seed for the random generator that is used to pick training data. By default, it is randomly generated. Set it to a specific value to use the same training data each time you start a job (assuming other related parameters such as source and analyzed_fields are the same).

        • Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This soft limit combines with the soft_tree_depth_tolerance to penalize trees that exceed the specified depth; the regularized loss increases quickly beyond this depth. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.

        • Advanced configuration option. This option controls how quickly the regularized loss increases when the tree depth exceeds soft_tree_depth_limit. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.01.

        • Defines the number of categories for which the predicted probabilities are reported. It must be non-negative or -1. If it is -1 or greater than the total number of categories, probabilities are reported for all categories; if you have a large number of categories, there could be a significant effect on the size of your destination index. NOTE: To use the AUC ROC evaluation method, num_top_classes must be set to -1 or a value greater than or equal to the total number of categories.

      • Additional properties are allowed.

        Hide outlier_detection attributes Show outlier_detection attributes object
        • Specifies whether the feature influence calculation is enabled.

        • The minimum outlier score that a document needs to have in order to calculate its feature influence score. Value range: 0-1.

        • method string

          The method that outlier detection uses. Available methods are lof, ldof, distance_kth_nn, distance_knn, and ensemble. The default value is ensemble, which means that outlier detection uses an ensemble of different methods and normalises and combines their individual outlier scores to obtain the overall outlier score.

        • Defines the value for how many nearest neighbors each method of outlier detection uses to calculate its outlier score. When the value is not set, different values are used for different ensemble members. This default behavior helps improve the diversity in the ensemble; only override it if you are confident that the value you choose is appropriate for the data set.

        • The proportion of the data set that is assumed to be outlying prior to outlier detection. For example, 0.05 means it is assumed that 5% of values are real outliers and 95% are inliers.

        • If true, the following operation is performed on the columns before computing outlier scores: (x_i - mean(x_i)) / sd(x_i).

      • Additional properties are allowed.

        Hide regression attributes Show regression attributes object
        • alpha number

          Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This parameter affects loss calculations by acting as a multiplier of the tree depth. Higher alpha values result in shallower trees and faster training times. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to zero.

        • dependent_variable string Required

          Defines which field of the document is to be predicted. It must match one of the fields in the index being used to train. If this field is missing from a document, then that document will not be used for training, but a prediction with the trained model will be generated for it. It is also known as continuous target variable. For classification analysis, the data type of the field must be numeric (integer, short, long, byte), categorical (ip or keyword), or boolean. There must be no more than 30 different values in this field. For regression analysis, the data type of the field must be numeric.

        • Advanced configuration option. Controls the fraction of data that is used to compute the derivatives of the loss function for tree training. A small value results in the use of a small fraction of the data. If this value is set to be less than 1, accuracy typically improves. However, too small a value may result in poor convergence for the ensemble and so require more trees. By default, this value is calculated during hyperparameter optimization. It must be greater than zero and less than or equal to 1.

        • Advanced configuration option. Specifies whether the training process should finish if it is not finding any better performing models. If disabled, the training process can take significantly longer and the chance of finding a better performing model is unremarkable.

        • eta number

          Advanced configuration option. The shrinkage applied to the weights. Smaller values result in larger forests which have a better generalization error. However, larger forests cause slower training. By default, this value is calculated during hyperparameter optimization. It must be a value between 0.001 and 1.

        • Advanced configuration option. Specifies the rate at which eta increases for each new tree that is added to the forest. For example, a rate of 1.05 increases eta by 5% for each extra tree. By default, this value is calculated during hyperparameter optimization. It must be between 0.5 and 2.

        • Advanced configuration option. Defines the fraction of features that will be used when selecting a random bag for each candidate split. By default, this value is calculated during hyperparameter optimization.

        • feature_processors array[object]

          Advanced configuration option. A collection of feature preprocessors that modify one or more included fields. The analysis uses the resulting one or more features instead of the original document field. However, these features are ephemeral; they are not stored in the destination index. Multiple feature_processors entries can refer to the same document fields. Automatic categorical feature encoding still occurs for the fields that are unprocessed by a custom processor or that have categorical values. Use this property only if you want to override the automatic feature encoding of the specified fields.

          Hide feature_processors attributes Show feature_processors attributes object
        • gamma number

          Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies a linear penalty associated with the size of individual trees in the forest. A high gamma value causes training to prefer small trees. A small gamma value results in larger individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

        • lambda number

          Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies an L2 regularization term which applies to leaf weights of the individual trees in the forest. A high lambda value causes training to favor small leaf weights. This behavior makes the prediction function smoother at the expense of potentially not being able to capture relevant relationships between the features and the dependent variable. A small lambda value results in large individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

        • Advanced configuration option. A multiplier responsible for determining the maximum number of hyperparameter optimization steps in the Bayesian optimization procedure. The maximum number of steps is determined based on the number of undefined hyperparameters times the maximum optimization rounds per hyperparameter. By default, this value is calculated during hyperparameter optimization.

        • Advanced configuration option. Defines the maximum number of decision trees in the forest. The maximum value is 2000. By default, this value is calculated during hyperparameter optimization.

        • Advanced configuration option. Specifies the maximum number of feature importance values per document to return. By default, no feature importance calculation occurs.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Defines the seed for the random generator that is used to pick training data. By default, it is randomly generated. Set it to a specific value to use the same training data each time you start a job (assuming other related parameters such as source and analyzed_fields are the same).

        • Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This soft limit combines with the soft_tree_depth_tolerance to penalize trees that exceed the specified depth; the regularized loss increases quickly beyond this depth. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.

        • Advanced configuration option. This option controls how quickly the regularized loss increases when the tree depth exceeds soft_tree_depth_limit. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.01.

        • The loss function used during regression. Available options are mse (mean squared error), msle (mean squared logarithmic error), huber (Pseudo-Huber loss).

        • A positive number that is used as a parameter to the loss_function.

    • Additional properties are allowed.

      Hide analyzed_fields attributes Show analyzed_fields attributes object
      • includes array[string] Required

        An array of strings that defines the fields that will be excluded from the analysis. You do not need to add fields with unsupported data types to excludes, these fields are excluded from the analysis automatically.

      • excludes array[string] Required

        An array of strings that defines the fields that will be included in the analysis.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • feature_values array[object] Required

      An array of objects that contain feature name and value pairs. The features have been processed and indicate what will be sent to the model for training.

      Hide feature_values attribute Show feature_values attribute object
      • * string Additional properties
POST /_ml/data_frame/analytics/_preview
curl \
 -X POST http://api.example.com/_ml/data_frame/analytics/_preview \
 -H "Content-Type: application/json" \
 -d '{"config":{"source":{"index":"string","runtime_mappings":{"additionalProperty1":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"},"additionalProperty2":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"}},"_source":{"includes":["string"],"excludes":["string"]},"query":{}},"analysis":{"":{"alpha":42.0,"dependent_variable":"string","downsample_factor":42.0,"early_stopping_enabled":true,"eta":42.0,"eta_growth_rate_per_tree":42.0,"feature_bag_fraction":42.0,"feature_processors":[{"frequency_encoding":{},"multi_encoding":{},"n_gram_encoding":{},"one_hot_encoding":{},"target_mean_encoding":{}}],"gamma":42.0,"lambda":42.0,"max_optimization_rounds_per_hyperparameter":42.0,"max_trees":42.0,"num_top_feature_importance_values":42.0,"prediction_field_name":"string","randomize_seed":42.0,"soft_tree_depth_limit":42.0,"soft_tree_depth_tolerance":42.0,"":"string","loss_function":"string","loss_function_parameter":42.0},"outlier_detection":{"compute_feature_influence":true,"feature_influence_threshold":42.0,"method":"string","n_neighbors":42.0,"outlier_fraction":42.0,"standardization_enabled":true}},"model_memory_limit":"string","max_num_threads":42.0,"analyzed_fields":{"includes":["string"],"excludes":["string"]}}}'
Request examples
{
  "config": {
    "source": {
      "index": "string",
      "runtime_mappings": {
        "additionalProperty1": {
          "fields": {
            "additionalProperty1": {
              "type": "boolean"
            },
            "additionalProperty2": {
              "type": "boolean"
            }
          },
          "fetch_fields": [
            {
              "field": "string",
              "format": "string"
            }
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {
              "additionalProperty1": {},
              "additionalProperty2": {}
            },
            "": "painless",
            "options": {
              "additionalProperty1": "string",
              "additionalProperty2": "string"
            }
          },
          "type": "boolean"
        },
        "additionalProperty2": {
          "fields": {
            "additionalProperty1": {
              "type": "boolean"
            },
            "additionalProperty2": {
              "type": "boolean"
            }
          },
          "fetch_fields": [
            {
              "field": "string",
              "format": "string"
            }
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {
              "additionalProperty1": {},
              "additionalProperty2": {}
            },
            "": "painless",
            "options": {
              "additionalProperty1": "string",
              "additionalProperty2": "string"
            }
          },
          "type": "boolean"
        }
      },
      "_source": {
        "includes": [
          "string"
        ],
        "excludes": [
          "string"
        ]
      },
      "query": {}
    },
    "analysis": {
      "": {
        "alpha": 42.0,
        "dependent_variable": "string",
        "downsample_factor": 42.0,
        "early_stopping_enabled": true,
        "eta": 42.0,
        "eta_growth_rate_per_tree": 42.0,
        "feature_bag_fraction": 42.0,
        "feature_processors": [
          {
            "frequency_encoding": {},
            "multi_encoding": {},
            "n_gram_encoding": {},
            "one_hot_encoding": {},
            "target_mean_encoding": {}
          }
        ],
        "gamma": 42.0,
        "lambda": 42.0,
        "max_optimization_rounds_per_hyperparameter": 42.0,
        "max_trees": 42.0,
        "num_top_feature_importance_values": 42.0,
        "prediction_field_name": "string",
        "randomize_seed": 42.0,
        "soft_tree_depth_limit": 42.0,
        "soft_tree_depth_tolerance": 42.0,
        "": "string",
        "loss_function": "string",
        "loss_function_parameter": 42.0
      },
      "outlier_detection": {
        "compute_feature_influence": true,
        "feature_influence_threshold": 42.0,
        "method": "string",
        "n_neighbors": 42.0,
        "outlier_fraction": 42.0,
        "standardization_enabled": true
      }
    },
    "model_memory_limit": "string",
    "max_num_threads": 42.0,
    "analyzed_fields": {
      "includes": [
        "string"
      ],
      "excludes": [
        "string"
      ]
    }
  }
}
Response examples (200)
{
  "feature_values": [
    {
      "additionalProperty1": "string",
      "additionalProperty2": "string"
    }
  ]
}




Preview features used by data frame analytics Added in 7.13.0

POST /_ml/data_frame/analytics/{id}/_preview

Previews the extracted features used by a data frame analytics config.

Path parameters

  • id string Required

    Identifier for the data frame analytics job.

application/json

Body

  • config object

    Additional properties are allowed.

    Hide config attributes Show config attributes object
    • source object Required

      Additional properties are allowed.

      Hide source attributes Show source attributes object
      • index string | array[string] Required
      • Hide runtime_mappings attribute Show runtime_mappings attribute object
        • * object Additional properties

          Additional properties are allowed.

          Hide * attributes Show * attributes object
          • fields object

            For type composite

            Hide fields attribute Show fields attribute object
            • * object Additional properties

              Additional properties are allowed.

              Hide * attribute Show * attribute object
              • type string Required

                Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

          • fetch_fields array[object]

            For type lookup

            Hide fetch_fields attributes Show fetch_fields attributes object
            • field string Required

              Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

            • format string
          • format string

            A custom format for date type runtime fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • script object

            Additional properties are allowed.

            Hide script attributes Show script attributes object
            • source string

              The script source.

            • id string
            • params object

              Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

              Hide params attribute Show params attribute object
              • * object Additional properties

                Additional properties are allowed.

            • lang string

              Any of:

              Values are painless, expression, mustache, or java.

            • options object
              Hide options attribute Show options attribute object
              • * string Additional properties
          • type string Required

            Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

      • _source object

        Additional properties are allowed.

        Hide _source attributes Show _source attributes object
        • includes array[string] Required

          An array of strings that defines the fields that will be excluded from the analysis. You do not need to add fields with unsupported data types to excludes, these fields are excluded from the analysis automatically.

        • excludes array[string] Required

          An array of strings that defines the fields that will be included in the analysis.

      • query object

        The Elasticsearch query domain-specific language (DSL). This value corresponds to the query object in an Elasticsearch search POST body. All the options that are supported by Elasticsearch can be used, as this object is passed verbatim to Elasticsearch. By default, this property has the following value: {"match_all": {}}.

        Additional properties are allowed.

    • analysis object Required

      Additional properties are allowed.

      Hide analysis attributes Show analysis attributes object
      • Additional properties are allowed.

        Hide classification attributes Show classification attributes object
        • alpha number

          Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This parameter affects loss calculations by acting as a multiplier of the tree depth. Higher alpha values result in shallower trees and faster training times. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to zero.

        • dependent_variable string Required

          Defines which field of the document is to be predicted. It must match one of the fields in the index being used to train. If this field is missing from a document, then that document will not be used for training, but a prediction with the trained model will be generated for it. It is also known as continuous target variable. For classification analysis, the data type of the field must be numeric (integer, short, long, byte), categorical (ip or keyword), or boolean. There must be no more than 30 different values in this field. For regression analysis, the data type of the field must be numeric.

        • Advanced configuration option. Controls the fraction of data that is used to compute the derivatives of the loss function for tree training. A small value results in the use of a small fraction of the data. If this value is set to be less than 1, accuracy typically improves. However, too small a value may result in poor convergence for the ensemble and so require more trees. By default, this value is calculated during hyperparameter optimization. It must be greater than zero and less than or equal to 1.

        • Advanced configuration option. Specifies whether the training process should finish if it is not finding any better performing models. If disabled, the training process can take significantly longer and the chance of finding a better performing model is unremarkable.

        • eta number

          Advanced configuration option. The shrinkage applied to the weights. Smaller values result in larger forests which have a better generalization error. However, larger forests cause slower training. By default, this value is calculated during hyperparameter optimization. It must be a value between 0.001 and 1.

        • Advanced configuration option. Specifies the rate at which eta increases for each new tree that is added to the forest. For example, a rate of 1.05 increases eta by 5% for each extra tree. By default, this value is calculated during hyperparameter optimization. It must be between 0.5 and 2.

        • Advanced configuration option. Defines the fraction of features that will be used when selecting a random bag for each candidate split. By default, this value is calculated during hyperparameter optimization.

        • feature_processors array[object]

          Advanced configuration option. A collection of feature preprocessors that modify one or more included fields. The analysis uses the resulting one or more features instead of the original document field. However, these features are ephemeral; they are not stored in the destination index. Multiple feature_processors entries can refer to the same document fields. Automatic categorical feature encoding still occurs for the fields that are unprocessed by a custom processor or that have categorical values. Use this property only if you want to override the automatic feature encoding of the specified fields.

          Hide feature_processors attributes Show feature_processors attributes object
        • gamma number

          Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies a linear penalty associated with the size of individual trees in the forest. A high gamma value causes training to prefer small trees. A small gamma value results in larger individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

        • lambda number

          Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies an L2 regularization term which applies to leaf weights of the individual trees in the forest. A high lambda value causes training to favor small leaf weights. This behavior makes the prediction function smoother at the expense of potentially not being able to capture relevant relationships between the features and the dependent variable. A small lambda value results in large individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

        • Advanced configuration option. A multiplier responsible for determining the maximum number of hyperparameter optimization steps in the Bayesian optimization procedure. The maximum number of steps is determined based on the number of undefined hyperparameters times the maximum optimization rounds per hyperparameter. By default, this value is calculated during hyperparameter optimization.

        • Advanced configuration option. Defines the maximum number of decision trees in the forest. The maximum value is 2000. By default, this value is calculated during hyperparameter optimization.

        • Advanced configuration option. Specifies the maximum number of feature importance values per document to return. By default, no feature importance calculation occurs.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Defines the seed for the random generator that is used to pick training data. By default, it is randomly generated. Set it to a specific value to use the same training data each time you start a job (assuming other related parameters such as source and analyzed_fields are the same).

        • Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This soft limit combines with the soft_tree_depth_tolerance to penalize trees that exceed the specified depth; the regularized loss increases quickly beyond this depth. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.

        • Advanced configuration option. This option controls how quickly the regularized loss increases when the tree depth exceeds soft_tree_depth_limit. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.01.

        • Defines the number of categories for which the predicted probabilities are reported. It must be non-negative or -1. If it is -1 or greater than the total number of categories, probabilities are reported for all categories; if you have a large number of categories, there could be a significant effect on the size of your destination index. NOTE: To use the AUC ROC evaluation method, num_top_classes must be set to -1 or a value greater than or equal to the total number of categories.

      • Additional properties are allowed.

        Hide outlier_detection attributes Show outlier_detection attributes object
        • Specifies whether the feature influence calculation is enabled.

        • The minimum outlier score that a document needs to have in order to calculate its feature influence score. Value range: 0-1.

        • method string

          The method that outlier detection uses. Available methods are lof, ldof, distance_kth_nn, distance_knn, and ensemble. The default value is ensemble, which means that outlier detection uses an ensemble of different methods and normalises and combines their individual outlier scores to obtain the overall outlier score.

        • Defines the value for how many nearest neighbors each method of outlier detection uses to calculate its outlier score. When the value is not set, different values are used for different ensemble members. This default behavior helps improve the diversity in the ensemble; only override it if you are confident that the value you choose is appropriate for the data set.

        • The proportion of the data set that is assumed to be outlying prior to outlier detection. For example, 0.05 means it is assumed that 5% of values are real outliers and 95% are inliers.

        • If true, the following operation is performed on the columns before computing outlier scores: (x_i - mean(x_i)) / sd(x_i).

      • Additional properties are allowed.

        Hide regression attributes Show regression attributes object
        • alpha number

          Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This parameter affects loss calculations by acting as a multiplier of the tree depth. Higher alpha values result in shallower trees and faster training times. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to zero.

        • dependent_variable string Required

          Defines which field of the document is to be predicted. It must match one of the fields in the index being used to train. If this field is missing from a document, then that document will not be used for training, but a prediction with the trained model will be generated for it. It is also known as continuous target variable. For classification analysis, the data type of the field must be numeric (integer, short, long, byte), categorical (ip or keyword), or boolean. There must be no more than 30 different values in this field. For regression analysis, the data type of the field must be numeric.

        • Advanced configuration option. Controls the fraction of data that is used to compute the derivatives of the loss function for tree training. A small value results in the use of a small fraction of the data. If this value is set to be less than 1, accuracy typically improves. However, too small a value may result in poor convergence for the ensemble and so require more trees. By default, this value is calculated during hyperparameter optimization. It must be greater than zero and less than or equal to 1.

        • Advanced configuration option. Specifies whether the training process should finish if it is not finding any better performing models. If disabled, the training process can take significantly longer and the chance of finding a better performing model is unremarkable.

        • eta number

          Advanced configuration option. The shrinkage applied to the weights. Smaller values result in larger forests which have a better generalization error. However, larger forests cause slower training. By default, this value is calculated during hyperparameter optimization. It must be a value between 0.001 and 1.

        • Advanced configuration option. Specifies the rate at which eta increases for each new tree that is added to the forest. For example, a rate of 1.05 increases eta by 5% for each extra tree. By default, this value is calculated during hyperparameter optimization. It must be between 0.5 and 2.

        • Advanced configuration option. Defines the fraction of features that will be used when selecting a random bag for each candidate split. By default, this value is calculated during hyperparameter optimization.

        • feature_processors array[object]

          Advanced configuration option. A collection of feature preprocessors that modify one or more included fields. The analysis uses the resulting one or more features instead of the original document field. However, these features are ephemeral; they are not stored in the destination index. Multiple feature_processors entries can refer to the same document fields. Automatic categorical feature encoding still occurs for the fields that are unprocessed by a custom processor or that have categorical values. Use this property only if you want to override the automatic feature encoding of the specified fields.

          Hide feature_processors attributes Show feature_processors attributes object
        • gamma number

          Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies a linear penalty associated with the size of individual trees in the forest. A high gamma value causes training to prefer small trees. A small gamma value results in larger individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

        • lambda number

          Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies an L2 regularization term which applies to leaf weights of the individual trees in the forest. A high lambda value causes training to favor small leaf weights. This behavior makes the prediction function smoother at the expense of potentially not being able to capture relevant relationships between the features and the dependent variable. A small lambda value results in large individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

        • Advanced configuration option. A multiplier responsible for determining the maximum number of hyperparameter optimization steps in the Bayesian optimization procedure. The maximum number of steps is determined based on the number of undefined hyperparameters times the maximum optimization rounds per hyperparameter. By default, this value is calculated during hyperparameter optimization.

        • Advanced configuration option. Defines the maximum number of decision trees in the forest. The maximum value is 2000. By default, this value is calculated during hyperparameter optimization.

        • Advanced configuration option. Specifies the maximum number of feature importance values per document to return. By default, no feature importance calculation occurs.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Defines the seed for the random generator that is used to pick training data. By default, it is randomly generated. Set it to a specific value to use the same training data each time you start a job (assuming other related parameters such as source and analyzed_fields are the same).

        • Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This soft limit combines with the soft_tree_depth_tolerance to penalize trees that exceed the specified depth; the regularized loss increases quickly beyond this depth. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.

        • Advanced configuration option. This option controls how quickly the regularized loss increases when the tree depth exceeds soft_tree_depth_limit. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.01.

        • The loss function used during regression. Available options are mse (mean squared error), msle (mean squared logarithmic error), huber (Pseudo-Huber loss).

        • A positive number that is used as a parameter to the loss_function.

    • Additional properties are allowed.

      Hide analyzed_fields attributes Show analyzed_fields attributes object
      • includes array[string] Required

        An array of strings that defines the fields that will be excluded from the analysis. You do not need to add fields with unsupported data types to excludes, these fields are excluded from the analysis automatically.

      • excludes array[string] Required

        An array of strings that defines the fields that will be included in the analysis.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • feature_values array[object] Required

      An array of objects that contain feature name and value pairs. The features have been processed and indicate what will be sent to the model for training.

      Hide feature_values attribute Show feature_values attribute object
      • * string Additional properties
POST /_ml/data_frame/analytics/{id}/_preview
curl \
 -X POST http://api.example.com/_ml/data_frame/analytics/{id}/_preview \
 -H "Content-Type: application/json" \
 -d '{"config":{"source":{"index":"string","runtime_mappings":{"additionalProperty1":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"},"additionalProperty2":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"}},"_source":{"includes":["string"],"excludes":["string"]},"query":{}},"analysis":{"":{"alpha":42.0,"dependent_variable":"string","downsample_factor":42.0,"early_stopping_enabled":true,"eta":42.0,"eta_growth_rate_per_tree":42.0,"feature_bag_fraction":42.0,"feature_processors":[{"frequency_encoding":{},"multi_encoding":{},"n_gram_encoding":{},"one_hot_encoding":{},"target_mean_encoding":{}}],"gamma":42.0,"lambda":42.0,"max_optimization_rounds_per_hyperparameter":42.0,"max_trees":42.0,"num_top_feature_importance_values":42.0,"prediction_field_name":"string","randomize_seed":42.0,"soft_tree_depth_limit":42.0,"soft_tree_depth_tolerance":42.0,"":"string","loss_function":"string","loss_function_parameter":42.0},"outlier_detection":{"compute_feature_influence":true,"feature_influence_threshold":42.0,"method":"string","n_neighbors":42.0,"outlier_fraction":42.0,"standardization_enabled":true}},"model_memory_limit":"string","max_num_threads":42.0,"analyzed_fields":{"includes":["string"],"excludes":["string"]}}}'
Request examples
{
  "config": {
    "source": {
      "index": "string",
      "runtime_mappings": {
        "additionalProperty1": {
          "fields": {
            "additionalProperty1": {
              "type": "boolean"
            },
            "additionalProperty2": {
              "type": "boolean"
            }
          },
          "fetch_fields": [
            {
              "field": "string",
              "format": "string"
            }
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {
              "additionalProperty1": {},
              "additionalProperty2": {}
            },
            "": "painless",
            "options": {
              "additionalProperty1": "string",
              "additionalProperty2": "string"
            }
          },
          "type": "boolean"
        },
        "additionalProperty2": {
          "fields": {
            "additionalProperty1": {
              "type": "boolean"
            },
            "additionalProperty2": {
              "type": "boolean"
            }
          },
          "fetch_fields": [
            {
              "field": "string",
              "format": "string"
            }
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {
              "additionalProperty1": {},
              "additionalProperty2": {}
            },
            "": "painless",
            "options": {
              "additionalProperty1": "string",
              "additionalProperty2": "string"
            }
          },
          "type": "boolean"
        }
      },
      "_source": {
        "includes": [
          "string"
        ],
        "excludes": [
          "string"
        ]
      },
      "query": {}
    },
    "analysis": {
      "": {
        "alpha": 42.0,
        "dependent_variable": "string",
        "downsample_factor": 42.0,
        "early_stopping_enabled": true,
        "eta": 42.0,
        "eta_growth_rate_per_tree": 42.0,
        "feature_bag_fraction": 42.0,
        "feature_processors": [
          {
            "frequency_encoding": {},
            "multi_encoding": {},
            "n_gram_encoding": {},
            "one_hot_encoding": {},
            "target_mean_encoding": {}
          }
        ],
        "gamma": 42.0,
        "lambda": 42.0,
        "max_optimization_rounds_per_hyperparameter": 42.0,
        "max_trees": 42.0,
        "num_top_feature_importance_values": 42.0,
        "prediction_field_name": "string",
        "randomize_seed": 42.0,
        "soft_tree_depth_limit": 42.0,
        "soft_tree_depth_tolerance": 42.0,
        "": "string",
        "loss_function": "string",
        "loss_function_parameter": 42.0
      },
      "outlier_detection": {
        "compute_feature_influence": true,
        "feature_influence_threshold": 42.0,
        "method": "string",
        "n_neighbors": 42.0,
        "outlier_fraction": 42.0,
        "standardization_enabled": true
      }
    },
    "model_memory_limit": "string",
    "max_num_threads": 42.0,
    "analyzed_fields": {
      "includes": [
        "string"
      ],
      "excludes": [
        "string"
      ]
    }
  }
}
Response examples (200)
{
  "feature_values": [
    {
      "additionalProperty1": "string",
      "additionalProperty2": "string"
    }
  ]
}
















































Evaluate a trained model Added in 8.3.0

POST /_ml/trained_models/{model_id}/_infer

Path parameters

  • model_id string Required

    The unique identifier of the trained model.

Query parameters

  • timeout string

    Controls the amount of time to wait for inference results.

application/json

Body Required

  • docs array[object] Required

    An array of objects to pass to the model for inference. The objects should contain a fields matching your configured trained model input. Typically, for NLP models, the field name is text_field. Currently, for NLP models, only a single value is allowed.

    Hide docs attribute Show docs attribute object
    • * object Additional properties

      Additional properties are allowed.

  • Additional properties are allowed.

    Hide inference_config attributes Show inference_config attributes object
    • Additional properties are allowed.

      Hide regression attributes Show regression attributes object
    • Additional properties are allowed.

      Hide classification attributes Show classification attributes object
      • Specifies the number of top class predictions to return. Defaults to 0.

      • Specifies the maximum number of feature importance values per document.

      • Specifies the type of the predicted field to write. Acceptable values are: string, number, boolean. When boolean is provided 1.0 is transformed to true and 0.0 to false.

      • The field that is added to incoming documents to contain the inference prediction. Defaults to predicted_value.

      • Specifies the field to which the top classes are written. Defaults to top_classes.

    • Additional properties are allowed.

      Hide text_classification attributes Show text_classification attributes object
      • Specifies the number of top class predictions to return. Defaults to 0.

      • Additional properties are allowed.

        Hide tokenization attributes Show tokenization attributes object
        • truncate string

          Values are first, second, or none.

        • span number

          Span options to apply

      • The field that is added to incoming documents to contain the inference prediction. Defaults to predicted_value.

      • Classification labels to apply other than the stored labels. Must have the same deminsions as the default configured labels

    • Additional properties are allowed.

      Hide zero_shot_classification attributes Show zero_shot_classification attributes object
      • Additional properties are allowed.

        Hide tokenization attributes Show tokenization attributes object
        • truncate string

          Values are first, second, or none.

        • span number

          Span options to apply

      • The field that is added to incoming documents to contain the inference prediction. Defaults to predicted_value.

      • Update the configured multi label option. Indicates if more than one true label exists. Defaults to the configured value.

      • labels array[string] Required

        The labels to predict.

    • Additional properties are allowed.

      Hide fill_mask attributes Show fill_mask attributes object
      • Specifies the number of top class predictions to return. Defaults to 0.

      • Additional properties are allowed.

        Hide tokenization attributes Show tokenization attributes object
        • truncate string

          Values are first, second, or none.

        • span number

          Span options to apply

      • The field that is added to incoming documents to contain the inference prediction. Defaults to predicted_value.

    • ner object

      Additional properties are allowed.

      Hide ner attributes Show ner attributes object
      • Additional properties are allowed.

        Hide tokenization attributes Show tokenization attributes object
        • truncate string

          Values are first, second, or none.

        • span number

          Span options to apply

      • The field that is added to incoming documents to contain the inference prediction. Defaults to predicted_value.

    • Additional properties are allowed.

      Hide pass_through attributes Show pass_through attributes object
      • Additional properties are allowed.

        Hide tokenization attributes Show tokenization attributes object
        • truncate string

          Values are first, second, or none.

        • span number

          Span options to apply

      • The field that is added to incoming documents to contain the inference prediction. Defaults to predicted_value.

    • Additional properties are allowed.

      Hide text_embedding attributes Show text_embedding attributes object
      • Additional properties are allowed.

        Hide tokenization attributes Show tokenization attributes object
        • truncate string

          Values are first, second, or none.

        • span number

          Span options to apply

      • The field that is added to incoming documents to contain the inference prediction. Defaults to predicted_value.

    • Additional properties are allowed.

      Hide text_expansion attributes Show text_expansion attributes object
      • Additional properties are allowed.

        Hide tokenization attributes Show tokenization attributes object
        • truncate string

          Values are first, second, or none.

        • span number

          Span options to apply

      • The field that is added to incoming documents to contain the inference prediction. Defaults to predicted_value.

    • Additional properties are allowed.

      Hide question_answering attributes Show question_answering attributes object
      • question string Required

        The question to answer given the inference context

      • Specifies the number of top class predictions to return. Defaults to 0.

      • Additional properties are allowed.

        Hide tokenization attributes Show tokenization attributes object
        • truncate string

          Values are first, second, or none.

        • span number

          Span options to apply

      • The field that is added to incoming documents to contain the inference prediction. Defaults to predicted_value.

      • The maximum answer length to consider for extraction

Responses

POST /_ml/trained_models/{model_id}/_infer
curl \
 -X POST http://api.example.com/_ml/trained_models/{model_id}/_infer \
 -H "Content-Type: application/json" \
 -d '{"docs":[{"additionalProperty1":{},"additionalProperty2":{}}],"inference_config":{"regression":{"results_field":"string","num_top_feature_importance_values":42.0},"classification":{"num_top_classes":42.0,"num_top_feature_importance_values":42.0,"prediction_field_type":"string","results_field":"string","top_classes_results_field":"string"},"text_classification":{"num_top_classes":42.0,"tokenization":{"truncate":"first","span":42.0},"results_field":"string","classification_labels":["string"]},"zero_shot_classification":{"tokenization":{"truncate":"first","span":42.0},"results_field":"string","multi_label":true,"labels":["string"]},"fill_mask":{"num_top_classes":42.0,"tokenization":{"truncate":"first","span":42.0},"results_field":"string"},"ner":{"tokenization":{"truncate":"first","span":42.0},"results_field":"string"},"pass_through":{"tokenization":{"truncate":"first","span":42.0},"results_field":"string"},"text_embedding":{"tokenization":{"truncate":"first","span":42.0},"results_field":"string"},"text_expansion":{"tokenization":{"truncate":"first","span":42.0},"results_field":"string"},"question_answering":{"question":"string","num_top_classes":42.0,"tokenization":{"truncate":"first","span":42.0},"results_field":"string","max_answer_length":42.0}}}'
Request examples
{
  "docs": [
    {
      "additionalProperty1": {},
      "additionalProperty2": {}
    }
  ],
  "inference_config": {
    "regression": {
      "results_field": "string",
      "num_top_feature_importance_values": 42.0
    },
    "classification": {
      "num_top_classes": 42.0,
      "num_top_feature_importance_values": 42.0,
      "prediction_field_type": "string",
      "results_field": "string",
      "top_classes_results_field": "string"
    },
    "text_classification": {
      "num_top_classes": 42.0,
      "tokenization": {
        "truncate": "first",
        "span": 42.0
      },
      "results_field": "string",
      "classification_labels": [
        "string"
      ]
    },
    "zero_shot_classification": {
      "tokenization": {
        "truncate": "first",
        "span": 42.0
      },
      "results_field": "string",
      "multi_label": true,
      "labels": [
        "string"
      ]
    },
    "fill_mask": {
      "num_top_classes": 42.0,
      "tokenization": {
        "truncate": "first",
        "span": 42.0
      },
      "results_field": "string"
    },
    "ner": {
      "tokenization": {
        "truncate": "first",
        "span": 42.0
      },
      "results_field": "string"
    },
    "pass_through": {
      "tokenization": {
        "truncate": "first",
        "span": 42.0
      },
      "results_field": "string"
    },
    "text_embedding": {
      "tokenization": {
        "truncate": "first",
        "span": 42.0
      },
      "results_field": "string"
    },
    "text_expansion": {
      "tokenization": {
        "truncate": "first",
        "span": 42.0
      },
      "results_field": "string"
    },
    "question_answering": {
      "question": "string",
      "num_top_classes": 42.0,
      "tokenization": {
        "truncate": "first",
        "span": 42.0
      },
      "results_field": "string",
      "max_answer_length": 42.0
    }
  }
}
Response examples (200)
{
  "inference_results": [
    {
      "entities": [
        {
          "class_name": "string",
          "class_probability": 42.0,
          "entity": "string",
          "start_pos": 42.0,
          "end_pos": 42.0
        }
      ],
      "is_truncated": true,
      "": 42.0,
      "predicted_value_sequence": "string",
      "prediction_probability": 42.0,
      "prediction_score": 42.0,
      "top_classes": [
        {
          "class_name": "string",
          "class_probability": 42.0,
          "class_score": 42.0
        }
      ],
      "warning": "string",
      "feature_importance": [
        {
          "feature_name": "string",
          "importance": 42.0,
          "classes": [
            {}
          ]
        }
      ]
    }
  ]
}




Create part of a trained model definition Added in 8.0.0

PUT /_ml/trained_models/{model_id}/definition/{part}

Path parameters

  • model_id string Required

    The unique identifier of the trained model.

  • part number Required

    The definition part number. When the definition is loaded for inference the definition parts are streamed in the order of their part number. The first part must be 0 and the final part must be total_parts - 1.

application/json

Body Required

  • definition string Required

    The definition part for the model. Must be a base64 encoded string.

  • The total uncompressed definition length in bytes. Not base64 encoded.

  • total_parts number Required

    The total number of parts that will be uploaded. Must be greater than 0.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • acknowledged boolean Required

      For a successful response, this value is always true. On failure, an exception is returned instead.

PUT /_ml/trained_models/{model_id}/definition/{part}
curl \
 -X PUT http://api.example.com/_ml/trained_models/{model_id}/definition/{part} \
 -H "Content-Type: application/json" \
 -d '{"definition":"string","total_definition_length":42.0,"total_parts":42.0}'
Request examples
{
  "definition": "string",
  "total_definition_length": 42.0,
  "total_parts": 42.0
}
Response examples (200)
{
  "acknowledged": true
}


































Send monitoring data Added in 6.3.0

PUT /_monitoring/bulk

This API is used by the monitoring features to send monitoring data.

Query parameters

application/json

Body object Required

One of:
  • index object

    Additional properties are allowed.

    Hide index attributes Show index attributes object
    • _id string
    • _index string
    • routing string
    • version number
    • Values are internal, external, external_gte, or force.

    • A map from the full name of fields to the name of dynamic templates. It defaults to an empty map. If a name matches a dynamic template, that template will be applied regardless of other match predicates defined in the template. If a field is already defined in the mapping, then this parameter won't be used.

      Hide dynamic_templates attribute Show dynamic_templates attribute object
      • * string Additional properties
    • pipeline string

      The ID of the pipeline to use to preprocess incoming documents. If the index has a default ingest pipeline specified, setting the value to _none turns off the default ingest pipeline for this request. If a final pipeline is configured, it will always run regardless of the value of this parameter.

    • If true, the request's actions must target an index alias.

  • create object

    Additional properties are allowed.

    Hide create attributes Show create attributes object
    • _id string
    • _index string
    • routing string
    • version number
    • Values are internal, external, external_gte, or force.

    • A map from the full name of fields to the name of dynamic templates. It defaults to an empty map. If a name matches a dynamic template, that template will be applied regardless of other match predicates defined in the template. If a field is already defined in the mapping, then this parameter won't be used.

      Hide dynamic_templates attribute Show dynamic_templates attribute object
      • * string Additional properties
    • pipeline string

      The ID of the pipeline to use to preprocess incoming documents. If the index has a default ingest pipeline specified, setting the value to _none turns off the default ingest pipeline for this request. If a final pipeline is configured, it will always run regardless of the value of this parameter.

    • If true, the request's actions must target an index alias.

  • update object

    Additional properties are allowed.

    Hide update attributes Show update attributes object
  • delete object

    Additional properties are allowed.

    Hide delete attributes Show delete attributes object

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • error object

      Additional properties are allowed.

      Hide error attributes Show error attributes object
      • type string Required

        The type of error

      • reason string

        A human-readable explanation of the error, in English.

      • The server stack trace. Present only if the error_trace=true parameter was sent with the request.

      • Additional properties are allowed.

      • root_cause array[object]

        Additional properties are allowed.

      • suppressed array[object]

        Additional properties are allowed.

    • errors boolean Required

      True if there is was an error

    • ignored boolean Required

      Was collection disabled?

    • took number Required
PUT /_monitoring/bulk
curl \
 -X PUT http://api.example.com/_monitoring/bulk?system_id=string&system_api_version=string&interval=string \
 -H "Content-Type: application/json" \
 -d '[{"":{"_id":"string","_index":"string","routing":"string","if_primary_term":42.0,"if_seq_no":42.0,"version":42.0,"version_type":"internal"}}]'
Request examples
[
  {
    "": {
      "_id": "string",
      "_index": "string",
      "routing": "string",
      "if_primary_term": 42.0,
      "if_seq_no": 42.0,
      "version": 42.0,
      "version_type": "internal"
    }
  }
]
Response examples (200)
{
  "error": {
    "type": "string",
    "reason": "string",
    "stack_trace": "string",
    "caused_by": {},
    "root_cause": [
      {}
    ],
    "suppressed": [
      {}
    ]
  },
  "errors": true,
  "ignored": true,
  "took": 42.0
}



















































































Get the rollup job capabilities Deprecated Technical preview

GET /_rollup/data

Get the capabilities of any rollup jobs that have been configured for a specific index or index pattern.

This API is useful because a rollup job is often configured to rollup only a subset of fields from the source index. Furthermore, only certain aggregations can be configured for various fields, leading to a limited subset of functionality depending on that configuration. This API enables you to inspect an index and determine:

  1. Does this index have associated rollup data somewhere in the cluster?
  2. If yes to the first question, what fields were rolled up, what aggregations can be performed, and where does the data live?

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • * object Additional properties

      Additional properties are allowed.

      Hide * attribute Show * attribute object
      • rollup_jobs array[object] Required

        There can be multiple, independent jobs configured for a single index or index pattern. Each of these jobs may have different configurations, so the API returns a list of all the various configurations available.

        Hide rollup_jobs attributes Show rollup_jobs attributes object
GET /_rollup/data
curl \
 -X GET http://api.example.com/_rollup/data
Response examples (200)
{
  "additionalProperty1": {
    "rollup_jobs": [
      {
        "fields": {
          "additionalProperty1": [
            {}
          ],
          "additionalProperty2": [
            {}
          ]
        },
        "index_pattern": "string",
        "job_id": "string",
        "rollup_index": "string"
      }
    ]
  },
  "additionalProperty2": {
    "rollup_jobs": [
      {
        "fields": {
          "additionalProperty1": [
            {}
          ],
          "additionalProperty2": [
            {}
          ]
        },
        "index_pattern": "string",
        "job_id": "string",
        "rollup_index": "string"
      }
    ]
  }
}