Predicting delayed flights with classification analysis

Each document in the sample flight data set contains details for a single flight, so this data is ready for analysis; it is already in a two-dimensional entity-based data structure. In general, you often need to transform the data into an entity-centric index before you can analyze the data.

In order to be analyzed, a document must contain at least one field with a supported data type (numeric, boolean, text, keyword or ip) and must not contain arrays with more than one item. If your source data consists of some documents that contain the dependent variable and some that do not, the model is trained on the subset of documents that contain it.

{
  "_index": "kibana_sample_data_flights",
  "_type": "_doc",
  "_id": "S-JS1W0BJ7wufFIaPAHe",
  "_version": 1,
  "_seq_no": 3356,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "FlightNum": "N32FE9T",
    "DestCountry": "JP",
    "OriginWeather": "Thunder & Lightning",
    "OriginCityName": "Adelaide",
    "AvgTicketPrice": 499.08518599798685,
    "DistanceMiles": 4802.864932998549,
    "FlightDelay": false,
    "DestWeather": "Sunny",
    "Dest": "Chubu Centrair International Airport",
    "FlightDelayType": "No Delay",
    "OriginCountry": "AU",
    "dayOfWeek": 3,
    "DistanceKilometers": 7729.461862731618,
    "timestamp": "2019-10-17T11:12:29",
    "DestLocation": {
      "lat": "34.85839844",
      "lon": "136.8049927"
    },
    "DestAirportID": "NGO",
    "Carrier": "ES-Air",
    "Cancelled": false,
    "FlightTimeMin": 454.6742272195069,
    "Origin": "Adelaide International Airport",
    "OriginLocation": {
      "lat": "-34.945",
      "lon": "138.531006"
    },
    "DestRegion": "SE-BD",
    "OriginAirportID": "ADL",
    "OriginRegion": "SE-BD",
    "DestCityName": "Tokoname",
    "FlightTimeHour": 7.577903786991782,
    "FlightDelayMin": 0
  }
}

To predict whether a specific flight is delayed:

Now you have a new index that contains a copy of your source data with predictions for your dependent variable.

When you view the classification results in Kibana, it shows contents of the destination index in a tabular format:

In this example, the table shows a column for the dependent variable (FlightDelay), which contains the ground truth values that you are trying to predict. It also shows a column for the predicted values (ml.FlightDelay_prediction), which were generated by the classification analysis. The ml.is_training column indicates whether the document was used in the training or testing data set. You can use this information to filter the table and the confusion matrix such that they contain only testing or training data.

If you want to understand how certain the model is about each prediction, you can examine its probability and score (ml.prediction_probability and ml.prediction_score). The higher these values are, the more confident the model is that the data point belongs to the named class. If you examine the destination index more closely in the Discover app in Kibana or use the standard Elasticsearch search command, you can see that the analysis predicts the probability of all possible classes for the dependent variable. The top_classes object contains the predicted classes with the highest scores.

GET df-flight-delayed/_search

          ...
          "FlightDelay" : false,
          ...
          "ml" : {
            "top_classes" : [ 
              {
                "class_probability" : 0.9198146781161334,
                "class_score" : 0.36964390728677926,
                "class_name" : "false"
              },
              {
                "class_probability" : 0.08018532188386665,
                "class_score" : 0.08018532188386665,
                "class_name" : "true"
              }
            ],
            "prediction_score" : 0.36964390728677926,
            "FlightDelay_prediction" : false,
            "prediction_probability" : 0.9198146781161334,
            "is_training" : false
          }

Though you can look at individual results and compare the predicted value (ml.FlightDelay_prediction) to the actual value (FlightDelay), you typically need to evaluate the success of your classification model as a whole.

Kibana provides a normalized confusion matrix that contains the percentage of occurrences where the analysis classified data points correctly with their actual class and the percentage of occurrences where it misclassified them.

If you want to see the exact number of occurrences, select a quadrant in the matrix. You can optionally filter the table to contain only testing data so you can see how well the model performs on previously unseen data. In this example, there are 2952 documents in the testing data that have the true class. 899 of them are predicted as false; this is called a false negative. 2053 are predicted correctly as true; this is called a true positive. The confusion matrix therefore shows us that 70% of the actual true values were correctly predicted and 30% were incorrectly predicted in the test data set.

Likewise if you select other quadrants in the matrix, it shows the number of documents that have the false class as their actual value in the testing data set. In this example, the model labeled 7223 documents out of 8801 correctly as false; this is called a true negative. 1578 documents are predicted incorrectly as true; this is called a false positive. Thus 82% of the actual false values were correctly predicted and 18% were incorrectly predicted in the test data set.

For more information about interpreting the evaluation metrics, see Classification evaluation.

You can also generate these metrics with the data frame analytics evaluate API.

POST _ml/data_frame/_evaluate
{
 "index": "df-flight-delayed",
   "query": {
    "term": {
      "ml.is_training": {
        "value": true  
      }
    }
  },
 "evaluation": {
   "classification": {
     "actual_field": "FlightDelay",
     "predicted_field": "ml.FlightDelay_prediction",
     "metrics": {
       "multiclass_confusion_matrix" : {}
     }
   }
 }
}

POST _ml/data_frame/_evaluate
{
 "index": "df-flight-delayed",
   "query": {
    "term": {
      "ml.is_training": {
        "value": false  
      }
    }
  },
 "evaluation": {
   "classification": {
     "actual_field": "FlightDelay",
     "predicted_field": "ml.FlightDelay_prediction",
     "metrics": {
       "multiclass_confusion_matrix" : {}
     }
   }
 }
}

{
  "classification" : {
    "multiclass_confusion_matrix" : {
      "confusion_matrix" : [
        {
          "actual_class" : "false", 
          "actual_class_doc_count" : 8801, 
          "predicted_classes" : [
            {
              "predicted_class" : "false", 
              "count" : 7223 
            },
            {
              "predicted_class" : "true",
              "count" : 1578
            }
          ],
          "other_predicted_class_doc_count" : 0
        },
        {
          "actual_class" : "true",
          "actual_class_doc_count" : 2952,
          "predicted_classes" : [
            {
              "predicted_class" : "false",
              "count" : 899
            },
            {
              "predicted_class" : "true",
              "count" : 2053
            }
          ],
          "other_predicted_class_doc_count" : 0
        }
      ],
      "other_actual_class_count" : 0
    }
  }
}

If you don’t want to keep the data frame analytics job, you can delete it by using the delete data frame analytics job API. When you delete data frame analytics jobs, the destination indices remain intact.