Predicting flight delays with regression analysis

Each document in the data set contains details for a single flight, so this data is ready for analysis; it is already in a two-dimensional entity-based data structure. In general, you often need to transform the data into an entity-centric index before you analyze the data.

In order to be analyzed, a document must contain at least one field with a supported data type (numeric, boolean, text, keyword or ip) and must not contain arrays with more than one item. If your source data consists of some documents that contain the dependent variable and some that do not, the model is trained on the subset of the documents that contain it.

{
  "_index": "kibana_sample_data_flights",
  "_type": "_doc",
  "_id": "S-JS1W0BJ7wufFIaPAHe",
  "_version": 1,
  "_seq_no": 3356,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "FlightNum": "N32FE9T",
    "DestCountry": "JP",
    "OriginWeather": "Thunder & Lightning",
    "OriginCityName": "Adelaide",
    "AvgTicketPrice": 499.08518599798685,
    "DistanceMiles": 4802.864932998549,
    "FlightDelay": false,
    "DestWeather": "Sunny",
    "Dest": "Chubu Centrair International Airport",
    "FlightDelayType": "No Delay",
    "OriginCountry": "AU",
    "dayOfWeek": 3,
    "DistanceKilometers": 7729.461862731618,
    "timestamp": "2019-10-17T11:12:29",
    "DestLocation": {
      "lat": "34.85839844",
      "lon": "136.8049927"
    },
    "DestAirportID": "NGO",
    "Carrier": "ES-Air",
    "Cancelled": false,
    "FlightTimeMin": 454.6742272195069,
    "Origin": "Adelaide International Airport",
    "OriginLocation": {
      "lat": "-34.945",
      "lon": "138.531006"
    },
    "DestRegion": "SE-BD",
    "OriginAirportID": "ADL",
    "OriginRegion": "SE-BD",
    "DestCityName": "Tokoname",
    "FlightTimeHour": 7.577903786991782,
    "FlightDelayMin": 0
  }
}

To predict the number of minutes delayed for each flight:

Now you have a new index that contains a copy of your source data with predictions for your dependent variable.

When you view the regression results in Kibana, it shows the contents of the destination index in a tabular format:

In this example, the table shows a column for the dependent variable (FlightDelayMin), which contains the ground truth values that we are trying to predict with the regression analysis. It also shows a column for the prediction values (ml.FlightDelayMin_prediction) and a column that indicates whether the document was used in the training set (ml.is_training). You can filter the table to show only testing or training data and you can select which fields are shown in the table. You can also enable histogram charts to get a better understanding of the distribution of values in your data.

If you chose to calculate feature importance, the destination index also contains ml.feature_importance objects. Every field that is included in the regression analysis (known as a feature of the data point) is assigned a feature importance value. This value has both a magnitude and a direction (positive or negative), which indicates how each field affects a particular prediction. Only the most significant values (in this case, the top 5) are stored in the index. However, the trained model metadata also contains the average magnitude of the feature importance values for each field across all the training data. You can view this summarized information in Kibana:

You can also see the feature importance values for each individual prediction in the form of a decision plot:

The decision path starts at a baseline, which is the average of the predictions for all the data points in the training data set. From there, the feature importance values are added to the decision path until it arrives at its final prediction. The features with the most significant positive or negative impact appear at the top. Thus in this example, the features related to the flight distance had the most significant influence on this particular predicted flight delay. This type of information can help you to understand how models arrive at their predictions. It can also indicate which aspects of your data set are most influential or least useful when you are training and tuning your model.

If you do not use Kibana, you can see summarized feature importance values by using the get trained model API and the individual values by searching the destination index.

GET _ml/inference/model-flight-delays*?include=total_feature_importance,feature_importance_baseline

{
  "count" : 1,
  "trained_model_configs" : [
    {
      "model_id" : "model-flight-delays-1601312043770",
      ...
      "metadata" : {
        ...
        "feature_importance_baseline" : {
          "baseline" : 47.43643652716527 
        },
        "total_feature_importance" : [
          {
            "feature_name" : "dayOfWeek",
            "importance" : {
              "mean_magnitude" : 0.38674590521018903, 
              "min" : -9.42823116446923, 
              "max" : 8.707461689065173 
            }
          },
          {
            "feature_name" : "OriginWeather",
            "importance" : {
            "mean_magnitude" : 0.18548393012368913,
            "min" : -9.079576266629092,
            "max" : 5.142479101907649
          }
          ...

GET model-flight-delays/_search

          ...
          "DestCountry" : "CH",
          "DestRegion" : "CH-ZH",
          "OriginAirportID" : "VIE",
          "DestCityName" : "Zurich",
          "ml": {
            "FlightDelayMin_prediction": 277.5392150878906,
            "feature_importance": [
            {
              "feature_name": "DestCityName",
              "importance": 0.6285966753441136
            },
            {
              "feature_name": "DistanceKilometers",
              "importance": 84.4982943868267
            },
            {
              "feature_name": "DistanceMiles",
              "importance": 103.90011847132116
            },
            {
              "feature_name": "FlightTimeHour",
              "importance": 3.7119156097309345
            },
            {
              "feature_name": "FlightTimeMin",
              "importance": 38.700587425831365
            }
            ],
            "is_training": true
          }
          ...

Though you can look at individual results and compare the predicted value (ml.FlightDelayMin_prediction) to the actual value (FlightDelayMins), you typically need to evaluate the success of the regression model as a whole.

Kibana provides training error metrics, which represent how well the model performed on the training data set. It also provides generalization error metrics, which represent how well the model performed on testing data.

A mean squared error (MSE) of zero means that the models predicts the dependent variable with perfect accuracy. This is the ideal, but is typically not possible. Likewise, an R-squared value of 1 indicates that all of the variance in the dependent variable can be explained by the feature variables. Typically, you compare the MSE and R-squared values from multiple regression models to find the best balance or fit for your data.

For more information about the interpreting the evaluation metrics, see Regression evaluation.

You can alternatively generate these metrics with the data frame analytics evaluate API.

POST _ml/data_frame/_evaluate
{
 "index": "model-flight-delays",
  "query": {
      "bool": {
        "filter": [{ "term":  { "ml.is_training": true } }]  
      }
    },
 "evaluation": {
   "regression": {
     "actual_field": "FlightDelayMin",   
     "predicted_field": "ml.FlightDelayMin_prediction", 
     "metrics": {
       "r_squared": {},
       "mse": {},
       "msle": {},
       "huber": {}
     }
   }
 }
}

{
  "regression" : {
    "huber" : {
      "value" : 30.216037330465102
    },
    "mse" : {
      "value" : 2847.2211476422967
    },
    "msle" : {
      "value" : "NaN"
    },
    "r_squared" : {
      "value" : 0.6956530017255125
    }
  }
}

POST _ml/data_frame/_evaluate
{
 "index": "model-flight-delays",
  "query": {
      "bool": {
        "filter": [{ "term":  { "ml.is_training": false } }] 
      }
    },
 "evaluation": {
   "regression": {
     "actual_field": "FlightDelayMin",
     "predicted_field": "ml.FlightDelayMin_prediction",
     "metrics": {
       "r_squared": {},
       "mse": {},
       "msle": {},
       "huber": {}
     }
   }
 }
}

When you have trained a satisfactory model, you can deploy it to make predictions about new data. Those steps are not covered in this example. See Inference.

If you don’t want to keep the data frame analytics job, you can delete it. For example, use Kibana or the delete data frame analytics job API. When you delete data frame analytics jobs in Kibana, you have the option to also remove the destination indices and index patterns.