IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« What’s new in 8.0 Set up Elasticsearch »

›

Quick start

edit

Quick start

edit

This guide helps beginners learn how to:

Install and run Elasticsearch in a test environment
Add data to Elasticsearch
Search and sort data
Extract fields from unstructured content during a search

Run Elasticsearch

edit

The simplest way to set up Elasticsearch is to create a managed deployment with Elasticsearch Service on Elastic Cloud. If you prefer to manage your own test environment, you can install and run Elasticsearch using Docker.

Get a free trial.
Log into Elastic Cloud.
Click Create deployment.
Give your deployment a name.
Click Create deployment and download the password for the elastic user.
Click Continue to open Kibana.
Click Explore on my own.

Install and run Elasticsearch

Install and start Docker Desktop.
Run:
```
docker network create elastic
docker pull docker.elastic.co/elasticsearch/elasticsearch:8.0.1
docker run --name es01 --net elastic -p 9200:9200 -p 9300:9300 -it docker.elastic.co/elasticsearch/elasticsearch:8.0.1
```
When you start Elasticsearch for the first time, the following security configuration occurs automatically:
- Certificates and keys are generated for the transport and HTTP layers.
- The Transport Layer Security (TLS) configuration settings are written to elasticsearch.yml.
- A password is generated for the elastic user.
- An enrollment token is generated for Kibana.
You might need to scroll back a bit in the terminal to view the password and enrollment token.
Copy the generated password and enrollment token and save them in a secure location. These values are shown only when you start Elasticsearch for the first time. You’ll use these to enroll Kibana with your Elasticsearch cluster and log in.

If you need to reset the password for the elastic user or other built-in users, run the elasticsearch-reset-password tool. To generate new enrollment tokens for Kibana or Elasticsearch nodes, run the elasticsearch-create-enrollment-token tool. These tools are available in the Elasticsearch bin directory.

Install and run Kibana

To analyze, visualize, and manage Elasticsearch data using an intuitive UI, install Kibana.

In a new terminal session, run:

docker pull docker.elastic.co/kibana/kibana:8.0.1
docker run --name kibana --net elastic -p 5601:5601 docker.elastic.co/kibana/kibana:8.0.1

When you start Kibana, a unique link is output to your terminal.

To access Kibana, click the generated link in your terminal.
1. In your browser, paste the enrollment token that you copied and click the button to connect your Kibana instance with Elasticsearch.
2. Log in to Kibana as the elastic user with the password that was generated when you started Elasticsearch.

Send requests to Elasticsearch

edit

You send data and other requests to Elasticsearch using REST APIs. This lets you interact with Elasticsearch using any client that sends HTTP requests, such as curl. You can also use Kibana’s console to send requests to Elasticsearch.

Use Kibana

Open Kibana’s main menu and go to Dev Tools > Console.

Run the following example API request in the console:

$response = $client->info();

resp = client.info()
print(resp)

response = client.info
puts response

res, err := es.Info()
fmt.Println(res, err)

const response = await client.info()
console.log(response)

GET /

Use curl

To communicate with Elasticsearch using curl or another client, you need your cluster’s endpoint.

Open Kibana’s main menu and click Manage this deployment.
From your deployment menu, go to the Elasticsearch page. Click Copy endpoint.
To submit an example API request, run the following curl command in a new terminal session. Replace <password> with the password for the elastic user. Replace <elasticsearch_endpoint> with your endpoint.
```
curl -u elastic:<password> <elasticsearch_endpoint>/
```

Use Kibana

Open Kibana’s main menu and go to Dev Tools > Console.

Run the following example API request in the console:

$response = $client->info();

resp = client.info()
print(resp)

response = client.info
puts response

res, err := es.Info()
fmt.Println(res, err)

const response = await client.info()
console.log(response)

GET /

Use curl

To submit an example API request, run the following curl command in a new terminal session.

Copy the http_ca.crt security certificate from your Docker container to your local machine.
```
docker cp es01:/usr/share/elasticsearch/config/certs/http_ca.crt .
```
Open a new terminal and verify that you can connect to your Elasticsearch cluster by making an authenticated call, using the http_ca.crt file that you copied from your Docker container. Enter the password for the elastic user when prompted.
```
curl --cacert http_ca.crt -u elastic https://localhost:9200
```

Add data

edit

You add data to Elasticsearch as JSON objects called documents. Elasticsearch stores these documents in searchable indices.

For time series data, such as logs and metrics, you typically add documents to a data stream made up of multiple auto-generated backing indices.

A data stream requires an index template that matches its name. Elasticsearch uses this template to configure the stream’s backing indices. Documents sent to a data stream must have a @timestamp field.

Add a single document

edit

Submit the following indexing request to add a single log entry to the logs-my_app-default data stream. Since logs-my_app-default doesn’t exist, the request automatically creates it using the built-in logs-*-* index template.

POST logs-my_app-default/_doc
{
  "@timestamp": "2099-05-06T16:21:15.000Z",
  "event": {
    "original": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736"
  }
}

The response includes metadata that Elasticsearch generates for the document:

The backing _index that contains the document. Elasticsearch automatically generates the names of backing indices.
A unique _id for the document within the index.

{
  "_index": ".ds-logs-my_app-default-2099-05-06-000001",
  "_id": "gl5MJXMBMk1dGnErnBW8",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 0,
  "_primary_term": 1
}

Add multiple documents

edit

Use the _bulk endpoint to add multiple documents in one request. Bulk data must be newline-delimited JSON (NDJSON). Each line must end in a newline character (\n), including the last line.

PUT logs-my_app-default/_bulk
{ "create": { } }
{ "@timestamp": "2099-05-07T16:24:32.000Z", "event": { "original": "192.0.2.242 - - [07/May/2020:16:24:32 -0500] \"GET /images/hm_nbg.jpg HTTP/1.0\" 304 0" } }
{ "create": { } }
{ "@timestamp": "2099-05-08T16:25:42.000Z", "event": { "original": "192.0.2.255 - - [08/May/2099:16:25:42 +0000] \"GET /favicon.ico HTTP/1.0\" 200 3638" } }

Search data

edit

Indexed documents are available for search in near real-time. The following search matches all log entries in logs-my_app-default and sorts them by @timestamp in descending order.

GET logs-my_app-default/_search
{
  "query": {
    "match_all": { }
  },
  "sort": [
    {
      "@timestamp": "desc"
    }
  ]
}

By default, the hits section of the response includes up to the first 10 documents that match the search. The _source of each hit contains the original JSON object submitted during indexing.

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": null,
    "hits": [
      {
        "_index": ".ds-logs-my_app-default-2099-05-06-000001",
        "_id": "PdjWongB9KPnaVm2IyaL",
        "_score": null,
        "_source": {
          "@timestamp": "2099-05-08T16:25:42.000Z",
          "event": {
            "original": "192.0.2.255 - - [08/May/2099:16:25:42 +0000] \"GET /favicon.ico HTTP/1.0\" 200 3638"
          }
        },
        "sort": [
          4081940742000
        ]
      },
      ...
    ]
  }
}

Get specific fields

edit

Parsing the entire _source is unwieldy for large documents. To exclude it from the response, set the _source parameter to false. Instead, use the fields parameter to retrieve the fields you want.

GET logs-my_app-default/_search
{
  "query": {
    "match_all": { }
  },
  "fields": [
    "@timestamp"
  ],
  "_source": false,
  "sort": [
    {
      "@timestamp": "desc"
    }
  ]
}

The response contains each hit’s fields values as a flat array.

{
  ...
  "hits": {
    ...
    "hits": [
      {
        "_index": ".ds-logs-my_app-default-2099-05-06-000001",
        "_id": "PdjWongB9KPnaVm2IyaL",
        "_score": null,
        "fields": {
          "@timestamp": [
            "2099-05-08T16:25:42.000Z"
          ]
        },
        "sort": [
          4081940742000
        ]
      },
      ...
    ]
  }
}

Search a date range

edit

To search across a specific time or IP range, use a range query.

GET logs-my_app-default/_search
{
  "query": {
    "range": {
      "@timestamp": {
        "gte": "2099-05-05",
        "lt": "2099-05-08"
      }
    }
  },
  "fields": [
    "@timestamp"
  ],
  "_source": false,
  "sort": [
    {
      "@timestamp": "desc"
    }
  ]
}

You can use date math to define relative time ranges. The following query searches for data from the past day, which won’t match any log entries in logs-my_app-default.

GET logs-my_app-default/_search
{
  "query": {
    "range": {
      "@timestamp": {
        "gte": "now-1d/d",
        "lt": "now/d"
      }
    }
  },
  "fields": [
    "@timestamp"
  ],
  "_source": false,
  "sort": [
    {
      "@timestamp": "desc"
    }
  ]
}

Extract fields from unstructured content

edit

You can extract runtime fields from unstructured content, such as log messages, during a search.

Use the following search to extract the source.ip runtime field from event.original. To include it in the response, add source.ip to the fields parameter.

GET logs-my_app-default/_search
{
  "runtime_mappings": {
    "source.ip": {
      "type": "ip",
      "script": """
        String sourceip=grok('%{IPORHOST:sourceip} .*').extract(doc[ "event.original" ].value)?.sourceip;
        if (sourceip != null) emit(sourceip);
      """
    }
  },
  "query": {
    "range": {
      "@timestamp": {
        "gte": "2099-05-05",
        "lt": "2099-05-08"
      }
    }
  },
  "fields": [
    "@timestamp",
    "source.ip"
  ],
  "_source": false,
  "sort": [
    {
      "@timestamp": "desc"
    }
  ]
}

Combine queries

edit

You can use the bool query to combine multiple queries. The following search combines two range queries: one on @timestamp and one on the source.ip runtime field.

GET logs-my_app-default/_search
{
  "runtime_mappings": {
    "source.ip": {
      "type": "ip",
      "script": """
        String sourceip=grok('%{IPORHOST:sourceip} .*').extract(doc[ "event.original" ].value)?.sourceip;
        if (sourceip != null) emit(sourceip);
      """
    }
  },
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "@timestamp": {
              "gte": "2099-05-05",
              "lt": "2099-05-08"
            }
          }
        },
        {
          "range": {
            "source.ip": {
              "gte": "192.0.2.0",
              "lte": "192.0.2.240"
            }
          }
        }
      ]
    }
  },
  "fields": [
    "@timestamp",
    "source.ip"
  ],
  "_source": false,
  "sort": [
    {
      "@timestamp": "desc"
    }
  ]
}

Aggregate data

edit

Use aggregations to summarize data as metrics, statistics, or other analytics.

The following search uses an aggregation to calculate the average_response_size using the http.response.body.bytes runtime field. The aggregation only runs on documents that match the query.

GET logs-my_app-default/_search
{
  "runtime_mappings": {
    "http.response.body.bytes": {
      "type": "long",
      "script": """
        String bytes=grok('%{COMMONAPACHELOG}').extract(doc[ "event.original" ].value)?.bytes;
        if (bytes != null) emit(Integer.parseInt(bytes));
      """
    }
  },
  "aggs": {
    "average_response_size":{
      "avg": {
        "field": "http.response.body.bytes"
      }
    }
  },
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "@timestamp": {
              "gte": "2099-05-05",
              "lt": "2099-05-08"
            }
          }
        }
      ]
    }
  },
  "fields": [
    "@timestamp",
    "http.response.body.bytes"
  ],
  "_source": false,
  "sort": [
    {
      "@timestamp": "desc"
    }
  ]
}

The response’s aggregations object contains aggregation results.

{
  ...
  "aggregations" : {
    "average_response_size" : {
      "value" : 12368.0
    }
  }
}

Explore more search options

edit

To keep exploring, index more data to your data stream and check out Common search options.

Clean up

edit

When you’re done, delete your test data stream and its backing indices.

DELETE _data_stream/logs-my_app-default

You can also delete your test deployment.

Click Delete deployment from the deployment overview page and follow the prompts.

To stop your Elasticsearch and Kibana Docker containers, run:

docker stop es01
docker stop kibana

To remove the containers and their network, run:

docker network rm elastic
docker rm es01
docker rm kibana

What’s next?

edit

Get the most out of your time series data by setting up data tiers and ILM. See Use Elasticsearch for time series data.
Use Fleet and Elastic Agent to collect logs and metrics directly from your data sources and send them to Elasticsearch. See the Ingest logs, metrics, and uptime data with Elastic Agent.
Use Kibana to explore, visualize, and manage your Elasticsearch data. See the Kibana quick start guide.

« What’s new in 8.0 Set up Elasticsearch »