› › ›

Ingest data through API

The Elasticsearch APIs enable you to ingest data through code. You can use the APIs of one of the language clients or the Elasticsearch HTTP APIs. The examples on this page use the HTTP APIs to demonstrate how ingesting works in Elasticsearch through APIs. If you want to ingest timestamped data or have a more complex ingestion use case, check out Beats or Logstash.

Using the bulk API

edit

You can index multiple JSON documents to an index and make it searchable using the bulk API.

The following example uses the bulk API to ingest book-related data into an index called books. The API call creates the index if it doesn’t exist already.

curl -X POST "${ES_URL}/_bulk?pretty" \
-H "Authorization: ApiKey ${API_KEY}" \
-H "Content-Type: application/json" \
-d'
{ "index" : { "_index" : "books" } }
{"title": "Snow Crash", "author": "Neal Stephenson", "release_date": "1992-06-01", "page_count": 470}
{ "index" : { "_index" : "books" } }
{"title": "Revelation Space", "author": "Alastair Reynolds", "release_date": "2000-03-15", "page_count": 585}
{ "index" : { "_index" : "books" } }
{"title": "1984", "author": "George Orwell", "release_date": "1985-06-01", "page_count": 328}
{ "index" : { "_index" : "books" } }
{"title": "Fahrenheit 451", "author": "Ray Bradbury", "release_date": "1953-10-15", "page_count": 227}
{ "index" : { "_index" : "books" } }
{"title": "Brave New World", "author": "Aldous Huxley", "release_date": "1932-06-01", "page_count": 268}
{ "index" : { "_index" : "books" } }
{"title": "The Blind Assassin", "author": "Margaret Atwood", "release_date": "2000-09-02", "page_count": 536}
'

The API returns a response similar to this:

{
  "errors": false,
  "took": 902,
  "items": [
    {
      "index": {
        "_index": "books",
        "_id": "MCYbQooByucZ6Gimx2BL",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 1,
          "successful": 1,
          "failed": 0
        },
        "_seq_no": 0,
        "_primary_term": 1,
        "status": 201
      }
    },
    ...
  ]
}

Under the hood, the bulk request creates a data schema, called "mappings" for the books index. To review the mappings and ensure the JSON body matches the index mappings, navigate to Content → Index management, select the index you want to ingest the data into, and click the Mappings tab.

The API call creates an index called books and adds six documents to it. All those documents have the title, author, release_date, and page_count fields with associated values. This data is now searchable.

You can check if a book is in the index by calling the search API and specifying either of the properties of the book in a match query, for example:

curl "${ES_URL}/books/_search?pretty" \
-H "Authorization: ApiKey ${API_KEY}" \
-H "Content-Type: application/json" \
-d'
{
  "query": {
    "match": {
      "title": "Snow Crash"
    }
  }
}
'

The API response contains an array of hits. Each hit represents a document that matches the query. The response contains the whole document. Only one document matches this query.

Using the index API

edit

Use the index API to ingest a single document to an index. Following the previous example, a new document will be added to the books index.

curl -X POST "${ES_URL}/books/_doc/" \
-H "Authorization: ApiKey ${API_KEY}" \
-H "Content-Type: application/json" \
-d'
{
  "title": "Neuromancer",
  "author": "William Gibson",
  "release_date": "1984-07-01",
  "page_count": "271"
}
'

The API call indexes the new document into the books index. Now you can search for it!

« Ingest your data Connector clients »