IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Build a map to compare metrics by country or region Map custom regions with reverse geocoding »
Elastic Docs ›Kibana Guide [8.2] ›Maps

Track, visualize, and alert on assets in real time

edit

Track, visualize, and alert on assets in real time

edit

Are you interested in asset tracking? Good news! Visualizing and analyzing data that moves is easy with Maps. You can track the location of an IoT device and monitor a package or vehicle in transit.

In this tutorial, you’ll look at live urban transit data from the city of Portland, Oregon. You’ll watch the city buses, use the data to visualize congestion, and notify a dispatch team when a bus enters a construction zone.

You’ll learn to:

  • Use Filebeat to ingest the TriMet REST API into Elasticsearch.
  • Create a map with layers that visualize asset tracks and last-known locations.
  • Use symbols and colors to style data values and show which direction an asset is heading.
  • Set up tracking containment alerts to monitor moving vehicles.

When you complete this tutorial, you’ll have a map that looks like this:

construction zones

Prerequisites

edit

Part 1: Ingest the Portland bus data

edit

To get to the fun of visualizing and alerting on Portland buses, you must first create a Filebeat input to ingest the TriMet Portland bus data into Elasticsearch.

Step 1: Set up an Elasticsearch index

edit
  1. In Kibana, open the main menu, then click Dev Tools.

  2. In Console, create the tri_met_tracks index lifecyle policy. This policy will keep the events in the hot data phase for 7 days. The data then moves to the warm phase. After 365 days in the warm phase, the data is deleted.

    PUT _ilm/policy/tri_met_tracks
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb",
            "max_age": "7d"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "0d",
        "actions": {
          "set_priority": {
            "priority": 50
          }
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

  3. In Console, create the tri_met_tracks index template, which is configured to use datastreams:

    PUT _index_template/tri_met_tracks
{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "tri_met_tracks"
        }
      }
    },
    "mappings": {
      "_routing": {
        "required": false
      },
      "numeric_detection": false,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "dynamic": true,
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "dynamic_templates": [],
      "date_detection": true,
      "properties": {
        "trimet": {
          "type": "object",
          "properties": {
            "expires": {
              "type": "date"
            },
            "signMessage": {
              "type": "text"
            },
            "serviceDate": {
              "type": "date"
            },
            "loadPercentage": {
              "type": "float"
            },
            "nextStopSeq": {
              "type": "integer"
            },
            "source": {
              "type": "keyword"
            },
            "type": {
              "type": "keyword"
            },
            "blockID": {
              "type": "integer"
            },
            "signMessageLong": {
              "type": "text"
            },
            "lastLocID": {
              "type": "keyword"
            },
            "nextLocID": {
              "type": "keyword"
            },
            "locationInScheduleDay": {
              "type": "integer"
            },
            "newTrip": {
              "type": "boolean"
            },
            "direction": {
              "type": "integer"
            },
            "inCongestion": {
              "type": "boolean"
            },
            "routeNumber": {
              "type": "integer"
            },
            "bearing": {
              "type": "integer"
            },
            "garage": {
              "type": "keyword"
            },
            "tripID": {
              "type": "keyword"
            },
            "delay": {
              "type": "integer"
            },
            "extraBlockID": {
              "type": "keyword"
            },
            "messageCode": {
              "type": "integer"
            },
            "lastStopSeq": {
              "type": "integer"
            },
            "location": {
              "type": "geo_point"
            },
            "time": {
              "index": true,
              "ignore_malformed": false,
              "store": false,
              "type": "date",
              "doc_values": true
            },
            "vehicleID": {
              "type": "keyword"
            },
            "offRoute": {
              "type": "boolean"
            }
          }
        }
      }
    }
  },
  "index_patterns": [
    "tri_met_tracks*"
  ],
  "data_stream": {
    "hidden": false,
    "allow_custom_routing": false
  },
  "composed_of": []
}

  4. In Console, add the tri_met_track ingest pipeline.

    PUT _ingest/pipeline/tri_met_tracks
{
  "processors": [
    {
      "set": {
        "field": "trimet.inCongestion",
        "value": "false",
        "if": "ctx?.trimet?.inCongestion == null"
      }
    },
    {
      "convert": {
        "field": "trimet.bearing",
        "type": "float"
      }
    },
    {
      "convert": {
        "field": "trimet.inCongestion",
        "type": "boolean"
      }
    },
    {
      "script": {
        "source": "ctx['trimet']['location'] = ctx['trimet']['latitude'] + \",\" + ctx['trimet']['longitude']"
      }
    },
    {
      "script": {
        "source": "ctx['_id'] = ctx['trimet']['vehicleID'] + \"_\" + ctx['trimet']['time']",
        "description": "Generate documentID"
      }
    },
    {
      "remove": {
        "field": [
          "message",
          "input",
          "agent",
          "ecs",
          "host",
          "event",
          "trimet.longitude",
          "trimet.latitude"
        ]
      }
    },
    {
      "set": {
        "field": "_index",
        "value": "tri_met_tracks"
      }
    }
  ]
}

Step 2: Start Filebeat

edit

  1. Replace the contents in your filebeat.yml file with the following:

    filebeat.inputs:
# Fetch trimet bus data every minute.
- type: httpjson
  interval: 1m
  request.url: "https://developer.trimet.org/ws/v2/vehicles?appID=<tri_met_app_id>"
  response.split:
    target: body.resultSet.vehicle
  processors:
    - decode_json_fields:
        fields: ["message"]
        target: "trimet"

  pipeline: "tri_met_tracks"


# ---------------------------- Elastic Cloud Output ----------------------------
cloud.id: <cloud_id>
cloud.auth: <username:password>
  2. Replace <tri_met_app_id> with your TriMet application id.
  3. Replace <username:password> with your Elastic Cloud deployment credentials.
  4. Replace <cloud_id> with your Elastic Cloud Enterprise/ece-cloud-id.html[elastic cloud id].
  5. Open a terminal window, and then navigate to the Filebeat folder.

  6. In your filebeat folder, run Filebeat with the edited config:

    /bin/filebeat -c filebeat.yml
  7. Wait for Filebeat to start shipping data to Elastic Cloud. Filebeat should not produce any output to stdout.
  8. Leave the terminal window open and Filebeat running throughout this tutorial.

Step 3: Create a data view for the tri_met_tracks Elasticsearch index

edit
  1. In Kibana, open the main menu, and click Stack Management > Data Views.
  2. Click Create data view.
  3. Give the data view a name: tri_met_tracks*.
  4. Click Next step.
  5. Set the Time field to trimet.time.
  6. Click Create data view.

Kibana shows the fields in your data view.

data view

Step 4: Explore the Portland bus data

edit
  1. Open the main menu, and click Discover.
  2. Set the data view to tri_met_tracks*.
  3. Open the time filter, and set the time range to the last 15 minutes.
  4. Expand a document and explore some of the fields that you will use later in this tutorial: trimet.bearing, trimet.inCongestion, trimet.location, and trimet.vehicleID.
discover

Part 2: Build an operational map

edit

It’s hard to get an overview of Portland buses by looking at individual events. Let’s create a map to show the bus routes and current location for each bus, along with the direction the buses are heading.

Step 1: Create your map

edit

Create your map and set the theme for the default layer to dark mode.

  1. Open the main menu, and click Maps.
  2. Click Create map.
  3. In the Layers list, click Road map, and then click Edit layer settings.
  4. Open the Tile service dropdown, and select Road map - dark.
  5. Click Save & close.

Step 2. Add a tracks layer

edit

Add a layer to show the bus routes for the last 15 minutes.

  1. Click Add layer.
  2. Click Tracks.
  3. Select the tri_met_tracks* data view.

  4. Define the tracks:

    1. Set Entity to trimet.vehicleID.
    2. Set Sort to trimet.time.
  5. Click Add layer.

  6. In Layer settings:

    1. Set Name to Buses.
    2. Set Opacity to 80%.
  7. Scroll to Layer Style, and set Border color to pink.
  8. Click Save & close.
  9. In the Layers list, click Buses, and then click Fit to data.

At this point, you have a map with lines that represent the routes of the buses as they move around the city.

tracks layer

Step 3. Indicate the direction of the bus tracks

edit

Add a layer that uses attributes in the data to set the style and orientation of the buses. You’ll see the direction buses are headed and what traffic is like.

  1. Click Add layer, and then select Top Hits per entity.
  2. Select the tri_met_tracks* data view.

  3. To display the most recent location per bus:

    1. Set Entity to trimet.vehicleID.
    2. Set Documents per entity to 1.
    3. Set Sort field to trimet.time.
    4. Set Sort order to descending.
  4. Click Add layer.

  5. Scroll to Layer Style.

    1. Set Symbol type to icon.
    2. Set Icon to arrow-es.

    3. Set the Fill color:

      1. Select By value styling, and set the field to trimet.inCongestion.
      2. Use a Custom color palette.
      3. Set the Other color to black.
      4. Add a green class for false, meaning the bus is not in traffic.
      5. Add a red class for true, meaning the bus is in congestion.
    4. Set Border width to 0.

    5. Change Symbol orientation to use By value and the trimet.bearing field.

      top hits layer style
  6. Click Save & close.
  7. Open the time filter, and set Refresh every to 10 seconds, and click Start.

Your map should automatically refresh every 10 seconds to show the latest bus positions and tracks.

tracks and top hits

Part 3: Setup geo-fencing alerts

edit

Let’s make TriMet Portland bus data actionable and alert when buses enter construction zones.

Step 1. Add a construction zone

edit

Add a layer for construction zones, which you will draw on the map. The construction zones will be used as your geofence boundary or threshold that serves as the basis for triggering alerts.

  1. Click Add layer.
  2. Click Create index.
  3. Set Index name to trimet_construction_zones.
  4. Click Create index.

  5. Draw 2 or 3 construction zones on your map:

    1. In the toolbar on left side of the map, select the bounding box icon bounding box icon.
    2. To draw a construction zone, click a start point on the map and drag.
    3. Click an endpoint to finish.
  6. When you finish drawing the construction zones, click Exit under the layer name in the legend.
  7. In Layer settings, set Name to Construction zones.
  8. Scroll to Layer Style, and set Fill color to yellow.
  9. Click Save & close.

  10. Save the map.

    1. Give the map a title.
    2. Under Add to dashboard, select None.
    3. Click Save and add to library.

The map now represents an operational view of live bus traffic. You’ll see the direction that the buses are traveling, and whether they are near or have entered a construction zone.

Your map is now complete.

construction zones

Step 2. Configure an alert

edit

Create a new alert by defining a rule and a connector. The rule includes the conditions that will trigger the alert, and the connector defines what action takes place once the alert is triggered. In this case, each alert will log a message to the Kibana log.

  1. Open Stack Management, and then click Rules and Connectors.
  2. Click Create rule.
  3. Name the rule Bus Alerts.
  4. Set Check every to 5 seconds.

  5. Notify Only on status change.

    rule configuration
  6. Select the Tracking containment rule type.

  7. Set Select entity:

    1. Set INDEX to tri_met_tracks*.
    2. Set BY to trimet.vehicleID.

  8. Set Select boundary INDEX to trimet_construction_zones.

    tracking containment configuration
  9. Under Actions, select the Server log connector type.
  10. Click Create a connector.

  11. In the Server log connector:

    1. Set Connector name to Bus alert connector.
    2. Click Save.

  12. Complete the Actions configuration.

    1. Set Message to :

      {
  "entityId": "{{context.entityId}}",
  "entityDateTime": "{{context.entityDateTime}}",
  "entityDocumentId": "{{context.entityDocumentId}}",
  "detectionDateTime": "{{context.detectionDateTime}}",
  "entityLocation": "{{context.entityLocation}}",
  "containingBoundaryId": "{{context.containingBoundaryId}}",
  "containingBoundaryName": "{{context.containingBoundaryName}}"
}
  13. Click Save.

The Bus Alert connector is added to the Rules and Connectors page. For more information on common connectors, refer to the Slack and Email connectors.

rules and connectors

Congratulations! You have completed the tutorial and have the recipe for tracking assets. You can now try replicating this same analysis with your own data.

« Build a map to compare metrics by country or region Map custom regions with reverse geocoding »