New

The executive guide to generative AI

Read more
About usPartnersSupport|Login
« GCP Vertex AI Hadoop »
Elastic Docs ›Elastic integrations

GoFlow2

edit

GoFlow2

edit

Version

0.4.0 [beta] This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features. (View all)

Compatible Kibana version(s)

8.11.0 or higher
9.0.0 or higher

Supported Serverless project types
What’s this?

Security
Observability

Subscription level
What’s this?

Basic

Level of support
What’s this?

Community

The GoFlow2 integration allows you to import logs generated by goflow2.

The only protocol/normalisation of goflow2 that is supported in this integration is sFlow. The normalisation of IPFIX and/or NetFlow is not yet support.

Data streams

edit

sflow

edit

The Goflow2 sFlow integration collects one type of data streams: logs

Sample Event

edit
Example

An example event for sflow looks as following:

{
    "@timestamp": "2024-07-31T00:00:59.314Z",
    "destination": {
        "address": [
            "216.160.83.58"
        ],
        "as": {
            "number": 209
        },
        "geo": {
            "city_name": "Milton",
            "continent_name": "North America",
            "country_iso_code": "US",
            "country_name": "United States",
            "location": {
                "lat": 47.2513,
                "lon": -122.3149
            },
            "region_iso_code": "US-WA",
            "region_name": "Washington"
        },
        "ip": "216.160.83.58",
        "port": 443
    },
    "ecs": {
        "version": "8.17.0"
    },
    "event": {
        "action": "SFLOW_5",
        "category": [
            "network"
        ],
        "kind": "event",
        "original": "{\"type\":\"SFLOW_5\",\"time_flow_start_ns\":1722384059314899647,\"sampler_address\":\"67.43.156.1\",\"sequence_num\":44555,\"in_if\":563,\"out_if\":573,\"src_addr\":\"216.160.83.57\",\"dst_addr\":\"216.160.83.58\",\"etype\":\"IPv4\",\"proto\":\"TCP\",\"src_port\":10876,\"dst_port\":443,\"src_vlan\":1500,\"dst_vlan\":1500,\"sampling_rate\":1000,\"bytes\":70}",
        "timezone": "+00:00",
        "type": [
            "connection"
        ]
    },
    "network": {
        "bytes": 70000,
        "packets": 1000,
        "transport": "tcp",
        "type": "ipv4"
    },
    "observer": {
        "egress": {
            "interface": {
                "id": "573"
            },
            "vlan": {
                "id": "1500"
            }
        },
        "ingress": {
            "interface": {
                "id": "563"
            },
            "vlan": {
                "id": "1500"
            }
        },
        "ip": [
            "67.43.156.1"
        ]
    },
    "related": {
        "ip": [
            "216.160.83.57",
            "216.160.83.58"
        ]
    },
    "sflow": {
        "bytes": 70,
        "sample_rate": 1000,
        "sequence_num": 44555
    },
    "source": {
        "address": [
            "216.160.83.57"
        ],
        "as": {
            "number": 209
        },
        "geo": {
            "city_name": "Milton",
            "continent_name": "North America",
            "country_iso_code": "US",
            "country_name": "United States",
            "location": {
                "lat": 47.2513,
                "lon": -122.3149
            },
            "region_iso_code": "US-WA",
            "region_name": "Washington"
        },
        "ip": "216.160.83.57",
        "port": 10876
    },
    "tags": [
        "preserve_original_event",
        "forwarded",
        "sflow"
    ]
}

Requirements

edit

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.

You need GoFlow2 to create log files for sFlow traffic. https://github.com/netsampler/goflow2

Setup

edit
  • Install integration and role out elastic agent
  • Install GoFlow2 for sFlow logging

Please use the following GoFlow2 mapping.yaml file:

# File: /etc/goflow2/mapping.yaml
formatter:
    fields: # list of fields to format in JSON
        - type
        - time_flow_start_ns
        - sampler_address
        - sequence_num
        - in_if
        - out_if
        - src_addr
        - dst_addr
        - etype
        - proto
        - src_port
        - dst_port
        - src_vlan
        - dst_vlan
        - sampling_rate
        - bytes

The output sflow transport files must be stored in the directory /var/log/sflow/goflow2/

Full command to run GoFlow2 for sflow traffic:

goflow2 -format json -listen "sflow://:6343" -mapping /etc/goflow2/mapping.yaml -transport.file /var/log/sflow/goflow2/goflow2.log

Fields

edit
Exported fields
Field Description Type

@timestamp

Event timestamp.

date

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

input.type

Input type

keyword

log.file.device_id

Device Id of the log file this event came from.

keyword

log.file.inode

Inode of the log file this event came from.

keyword

log.offset

Log offset

long

sflow.bytes

Original size in bytes of the sample packet.

long

sflow.sample_rate

sample rate.

long

sflow.sequence_num

flow sequence number.

long

Changelog

edit
Changelog
Version Details Kibana version(s)

0.4.0

Enhancement (View pull request)
Support stack version 9.0.

—

0.3.0

Enhancement (View pull request)
ECS version updated to 8.17.0.

—

0.2.0

Enhancement (View pull request)
Add configuration for custom filestream options. Ensure filestream fingerprint is disabled in system test.

—

0.1.1

Bug fix (View pull request)
Use triple-brace Mustache templating when referencing variables in ingest pipelines.

—

0.1.0

Enhancement (View pull request)
Initial version of the package

—
« GCP Vertex AI Hadoop »

On this page

Was this helpful?
Feedback