Implementation patterns

Normalizing data provides a more consistent view of events from various data sources. Following these conventions will help to better describe, discover, identify, and categorize events.

The group of individual fields residing outside any field set at the top-level of ECS are known as base fields.

ECS events follow these conventions with the base fields:

In ECS, the host is the computing instance where the event happened. A host can be a physical device, virtual machine, container, or cloud instance.

The host.* field set contains common attributes for different computing instances. Certain host types have more fields to capture specific details, like cloud.* or container.*.

An agent is software that collects, observes, measures, or detects the event. The agent.* fields capture details about which agent entity captured the event, including the agent’s version. Examples of agents are Beats and Elastic Agent.

An observer is an external monitoring or intermediary device, like a firewall, APM server, or web proxy. These devices monitor and detect network, security, application events. Capture the details for these device types in the observer.* field set.

ECS requires the @timestamp field on every event. Some events also contain extra timestamps to capture.

These three timestamps should typically follow a chronological order:

@timestamp < event.created < event.ingested

event.duration = event.end - event.start

Specific event.* fields exist to capture where an event originated.

The event categorization fields group similar events using allowed values for four fields:

Using the Categorization Fields covers more details on using these four fields together to categorize events.

A monitoring agent or ingest pipeline can add more details to the original event. ECS has many fields to hold these enrichment details.

{
  "source": {
    "address": "8.8.8.8",
    "ip": "8.8.8.8",
    "geo": {
      "continent_name": "North America",
      "country_name": "United States",
      "country_iso_code": "US",
      "location": { "lat": 37.751, "lon": -97.822 }
    }
  }
}

{
  "user_agent": {
    "user_agent": {
      "name": "Chrome",
      "original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36",
      "version": "51.0.2704.103",
      "os": {
        "name": "Mac OS X",
        "version": "10.10.5",
        "full": "Mac OS X 10.10.5",
        "platform": "darwin",
        "type": "macos"
      },
      "device" : {
        "name" : "Mac"
      }
    }
  }
}

{
  "original" : "http://myusername:mypassword@www.example.com:80/foo.gif?key1=val1&key2=val2#fragment",
  "url" : {
    "path" : "/foo.gif",
    "fragment" : "fragment",
    "extension" : "gif",
    "password" : "mypassword",
    "original" : "http://myusername:mypassword@www.example.com:80/foo.gif?key1=val1&key2=val2#fragment",
    "scheme" : "http",
    "port" : 80,
    "user_info" : "myusername:mypassword",
    "domain" : "www.example.com",
    "query" : "key1=val1&key2=val2",
    "username" : "myusername"
  }
}

{
  "fqdn": "www.example.ac.uk",
  "url": {
    "subdomain": "www",
    "registered_domain": "example.ac.uk",
    "top_level_domain": "ac.uk",
    "domain": "www.example.ac.uk"
}

Many events have similar content populating different fields: IP addresses, file hashes, hostnames. Pivot between these events using the related.* fields.

For example, IP addresses found under the host.*, source.*, destination.*, client.*, and server.* fields sets and the network.forwarded_ip field. By adding all IP addresses in an event to the related.ip field, there is now a single field to search for a given IP regardless of what field it appeared:

related.ip: ["10.42.42.42"]