WARNING: Version 5.4 of the Elastic Stack has passed its EOL date.

This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.

« Getting Started Creating Jobs »
Elastic Docs ›X-Pack for the Elastic Stack [5.4] ›Machine Learning in the Elastic Stack ›Getting Started

Identifying Data for Analysis

edit

Identifying Data for Analysis

edit

For the purposes of this tutorial, we provide sample data that you can play with and search in Elasticsearch. When you consider your own data, however, it’s important to take a moment and think about where the X-Pack machine learning features will be most impactful.

The first consideration is that it must be time series data. The machine learning features are designed to model and detect anomalies in time series data.

The second consideration, especially when you are first learning to use machine learning, is the importance of the data and how familiar you are with it. Ideally, it is information that contains key performance indicators (KPIs) for the health, security, or success of your business or system. It is information that you need to monitor and act on when anomalous behavior occurs. You might even have Kibana dashboards that you’re already using to watch this data. The better you know the data, the quicker you will be able to create machine learning jobs that generate useful insights.

The final consideration is where the data is located. This tutorial assumes that your data is stored in Elasticsearch. It guides you through the steps required to create a datafeed that passes data to a job. If your own data is outside of Elasticsearch, analysis is still possible by using a post data API.

If you want to create machine learning jobs in Kibana, you must use datafeeds. That is to say, you must store your input data in Elasticsearch. When you create a job, you select an existing index pattern and Kibana configures the datafeed for you under the covers.

Obtaining a Sample Data Set

edit

In this step we will upload some sample data to Elasticsearch. This is standard Elasticsearch functionality, and is needed to set the stage for using machine learning.

The sample data for this tutorial contains information about the requests that are received by various applications and services in a system. A system administrator might use this type of information to track the total number of requests across all of the infrastructure. If the number of requests increases or decreases unexpectedly, for example, this might be an indication that there is a problem or that resources need to be redistributed. By using the X-Pack machine learning features to model the behavior of this data, it is easier to identify anomalies and take appropriate action.

Download this sample data and scripts by clicking here: server_metrics.tar.gz

Use the following commands to extract the files:

tar -zxvf server_metrics.tar.gz

Each document in the server-metrics data set has the following schema:

{
  "index":
  {
    "_index":"server-metrics",
    "_type":"metric",
    "_id":"1177"
  }
}
{
  "@timestamp":"2017-03-23T13:00:00",
  "accept":36320,
  "deny":4156,
  "host":"server_2",
  "response":2.4558210155,
  "service":"app_3",
  "total":40476
}

The sample data sets include summarized data. For example, the total value is a sum of the requests that were received by a specific service at a particular time. If your data is stored in Elasticsearch, you can generate this type of sum or average by using aggregations. One of the benefits of summarizing data this way is that Elasticsearch automatically distributes these calculations across your cluster. You can then feed this summarized data into X-Pack machine learning instead of raw results, which reduces the volume of data that must be considered while detecting anomalies. For the purposes of this tutorial, however, these summary values are stored in Elasticsearch. For more information, see Aggregating Data For Faster Performance.

Before you load the data set, you need to set up mappings for the fields. Mappings divide the documents in the index into logical groups and specify a field’s characteristics, such as the field’s searchability or whether or not it’s tokenized, or broken up into separate words.

You can use scripts to create the mappings and load the data set. If X-Pack security is enabled, use the upload_server_metrics.sh script. Before you run it, however, you must edit the USERNAME and PASSWORD variables with your actual user ID and password. If X-Pack security is not enabled, use the upload_server_metrics_noauth.sh script.

The scripts run a curl command that makes the following create index API request:

PUT server-metrics
{
   "settings":{
      "number_of_shards":1,
      "number_of_replicas":0
   },
   "mappings":{
      "metric":{
         "properties":{
            "@timestamp":{
               "type":"date"
            },
            "accept":{
               "type":"long"
            },
            "deny":{
               "type":"long"
            },
            "host":{
               "type":"keyword"
            },
            "response":{
               "type":"float"
            },
            "service":{
               "type":"keyword"
            },
            "total":{
               "type":"long"
            }
         }
      }
   }
}

To learn more about mappings and data types, see Mapping.

You can then use the bulk API to load the sample data set. The scripts run commands similar to the following example:

curl -X POST -H "Content-Type: application/json"
http://localhost:9200/server-metrics/_bulk --data-binary "@server-metrics_1.json"

There are twenty data files. These commands might take some time to run, depending on the computing resources available.

You can verify that the data was loaded successfully by running the cat indices API:

GET _cat/indices?v

You should see output similar to the following:

health status index ... pri rep docs.count  docs.deleted  store.size ...
green  open   server-metrics ... 1 0 905940  0  120.5mb  ...

Next, you must define an index pattern for this data set:

  1. Open Kibana in your web browser. If you are running Kibana locally, go to http://localhost:5601/.
  2. Click the Management tab, then Index Patterns.
  3. If you already have index patterns, click the plus sign (+) to define a new one. Otherwise, the Configure an index pattern wizard is already open.
  4. For this tutorial, any pattern that matches the name of the index you’ve loaded will work. For example, enter server-metrics* as the index pattern.
  5. Verify that the Index contains time-based events is checked.
  6. Select the @timestamp field from the Time-field name list.
  7. Click Create.

This data set can now be analyzed in machine learning jobs in Kibana.

« Getting Started Creating Jobs »