›

GoFlow2

Version	0.1.1 [beta] This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features. (View all)
Compatible Kibana version(s)	8.11.0 or higher
Supported Serverless project types What’s this?	Security Observability
Subscription level What’s this?	Basic
Level of support What’s this?	Community

The GoFlow2 integration allows you to import logs generated by goflow2.

The only protocol/normalisation of goflow2 that is supported in this integration is sFlow. The normalisation of IPFIX and/or NetFlow is not yet support.

Data streams

edit

sflow

edit

The Goflow2 sFlow integration collects one type of data streams: logs

Sample Event

edit

Example

An example event for sflow looks as following:

{
    "@timestamp": "2024-07-31T00:00:59.314Z",
    "destination": {
        "address": [
            "216.160.83.58"
        ],
        "as": {
            "number": 209
        },
        "geo": {
            "city_name": "Milton",
            "continent_name": "North America",
            "country_iso_code": "US",
            "country_name": "United States",
            "location": {
                "lat": 47.2513,
                "lon": -122.3149
            },
            "region_iso_code": "US-WA",
            "region_name": "Washington"
        },
        "ip": "216.160.83.58",
        "port": 443
    },
    "ecs": {
        "version": "8.11.0"
    },
    "event": {
        "action": "SFLOW_5",
        "category": [
            "network"
        ],
        "kind": "event",
        "original": "{\"type\":\"SFLOW_5\",\"time_flow_start_ns\":1722384059314899647,\"sampler_address\":\"67.43.156.1\",\"sequence_num\":44555,\"in_if\":563,\"out_if\":573,\"src_addr\":\"216.160.83.57\",\"dst_addr\":\"216.160.83.58\",\"etype\":\"IPv4\",\"proto\":\"TCP\",\"src_port\":10876,\"dst_port\":443,\"src_vlan\":1500,\"dst_vlan\":1500,\"sampling_rate\":1000,\"bytes\":70}",
        "timezone": "+00:00",
        "type": [
            "connection"
        ]
    },
    "network": {
        "bytes": 70000,
        "packets": 1000,
        "transport": "tcp",
        "type": "ipv4"
    },
    "observer": {
        "egress": {
            "interface": {
                "id": "573"
            },
            "vlan": {
                "id": "1500"
            }
        },
        "ingress": {
            "interface": {
                "id": "563"
            },
            "vlan": {
                "id": "1500"
            }
        },
        "ip": [
            "67.43.156.1"
        ]
    },
    "related": {
        "ip": [
            "216.160.83.57",
            "216.160.83.58"
        ]
    },
    "sflow": {
        "bytes": 70,
        "sample_rate": 1000,
        "sequence_num": 44555
    },
    "source": {
        "address": [
            "216.160.83.57"
        ],
        "as": {
            "number": 209
        },
        "geo": {
            "city_name": "Milton",
            "continent_name": "North America",
            "country_iso_code": "US",
            "country_name": "United States",
            "location": {
                "lat": 47.2513,
                "lon": -122.3149
            },
            "region_iso_code": "US-WA",
            "region_name": "Washington"
        },
        "ip": "216.160.83.57",
        "port": 10876
    },
    "tags": [
        "preserve_original_event",
        "forwarded",
        "sflow"
    ]
}

Requirements

edit

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.

You need GoFlow2 to create log files for sFlow traffic. https://github.com/netsampler/goflow2

Setup

edit

Install integration and role out elastic agent
Install GoFlow2 for sFlow logging

Please use the following GoFlow2 mapping.yaml file:

# File: /etc/goflow2/mapping.yaml
formatter:
    fields: # list of fields to format in JSON
        - type
        - time_flow_start_ns
        - sampler_address
        - sequence_num
        - in_if
        - out_if
        - src_addr
        - dst_addr
        - etype
        - proto
        - src_port
        - dst_port
        - src_vlan
        - dst_vlan
        - sampling_rate
        - bytes

The output sflow transport files must be stored in the directory /var/log/sflow/goflow2/

Full command to run GoFlow2 for sflow traffic:

goflow2 -format json -listen "sflow://:6343" -mapping /etc/goflow2/mapping.yaml -transport.file /var/log/sflow/goflow2/goflow2.log

Fields

edit

Exported fields

Field	Description	Type
@timestamp	Event timestamp.	date
data_stream.dataset	Data stream dataset.	constant_keyword
data_stream.namespace	Data stream namespace.	constant_keyword
data_stream.type	Data stream type.	constant_keyword
input.type	Input type	keyword
log.file.device_id	Device Id of the log file this event came from.	keyword
log.file.inode	Inode of the log file this event came from.	keyword
log.offset	Log offset	long
sflow.bytes	Original size in bytes of the sample packet.	long
sflow.sample_rate	sample rate.	long
sflow.sequence_num	flow sequence number.	long

Changelog

edit

Changelog

Version	Details	Kibana version(s)
0.1.1	Bug fix (View pull request) Use triple-brace Mustache templating when referencing variables in ingest pipelines.	—
0.1.0	Enhancement (View pull request) Initial version of the package	—

« GCP Vertex AI Hadoop »