GoFlow2 logs Integration for Elastic

The GoFlow2 logs integration for Elastic enables you to ingest and normalize network flow data, specifically focusing on the sFlow protocol. By using this integration, you can gain deep visibility into network traffic patterns, bandwidth usage, and potential security threats.

This integration is compatible with the following:

• GoFlow2 binary versions that support the -mapping and -transport.file flags, such as version 2.1.2 or later.
• Network infrastructure devices (switches, routers, firewalls) capable of exporting sFlow (versions 2, 4, or 5).

This integration collects network flow data by monitoring JSON log files generated by the GoFlow2 collector. GoFlow2 acts as a flow collector that receives sFlow packets from network devices and converts them into a structured JSON format.

The Elastic Agent monitors the configured log path (defaulting to /var/log/sflow/goflow2/*.log) to ingest these formatted JSON logs. The integration then maps the data to the Elastic Common Schema (ECS), ensuring that fields like source and destination IP addresses, ports, and VLAN tags are normalized for consistent analysis across your Elastic deployment.

The GoFlow2 logs integration collects network flow data by monitoring JSON log files generated by the GoFlow2 collector. The data is ingested into the goflow2.sflow dataset.

The integration collects the following types of data:

• sFlow logs: Detailed records of sampled network traffic, capturing packet headers and flow metadata.
• Network Metadata: Information including source and destination IP addresses, source and destination ports, and VLAN tags.
• Flow Statistics: Metrics regarding the volume of traffic, such as the number of bytes and packets transferred in a specific flow.

Integrating GoFlow2 logs with Elastic provides visibility into network traffic patterns and performance. Key use cases include:

• Network traffic analysis: Analyze traffic flows to understand bandwidth usage, identify top talkers, and optimize network performance. Gain real-time visibility into high-volume network traffic by collecting sFlow data from routers and switches, allowing for detailed analysis of source and destination patterns.
• Security analysis and monitoring: Detect anomalous traffic patterns, potential DDoS attacks, or unauthorized data transfers by monitoring source and destination IPs.
• Bandwidth management: Track bandwidth consumption across different interfaces and VLANs to identify top talkers and optimize network resource allocation. Use flow statistics to track long-term trends in network usage and plan for future infrastructure needs.
• Troubleshooting: Investigate network connectivity issues by examining specific flow records and metadata.
• Compliance and auditing: Maintain long-term records of network flows for regulatory compliance and forensic investigations following security incidents.

To use this integration, you need:

• Elastic Stack version: Ensure you are running an Elastic Stack version that supports Elastic Agent and Fleet (version 8.11.0 or later is required).
• Elastic Agent: The agent must be installed and enrolled in a policy via Fleet. It requires network access to the Elasticsearch cluster to ship logs.
• Administrative access: You must have root or sudo privileges on the Linux server where GoFlow2 and the Elastic Agent will be installed.
• GoFlow2 binary: The GoFlow2 binary must be downloaded and placed in the system execution path (for example, /usr/local/bin/).
• Directory permissions: The directory /var/log/sflow/goflow2/ must exist. It must be writable by the GoFlow2 process and readable by the Elastic Agent.
• Network connectivity: Port 6343/UDP must be open on the host firewall to receive sFlow traffic from network devices.
• Network device configuration: Access to your network switches or routers is required to configure them to export sFlow data to the collector's IP address.

Elastic Agent must be installed. For more details, check the Elastic Agent installation instructions. You can install only one Elastic Agent per host.

Elastic Agent is required to stream data from the syslog or log file receiver and ship the data to Elastic, where the events will then be processed via the integration's ingest pipelines.

Follow these steps to install and configure GoFlow2 to collect sFlow data and write it to a log file that Elastic Agent can read.

Download the latest release and move it to your bin directory:

wget https://github.com/netsampler/goflow2/releases/download/v2.2.6/goflow2_2.2.6_linux_amd64.tar.gz
tar -xzvf goflow2_2.2.6_linux_amd64.tar.gz
sudo mv goflow2 /usr/local/bin/
sudo chmod +x /usr/local/bin/goflow2
		

Create the configuration and log directories:

sudo mkdir -p /etc/goflow2
sudo mkdir -p /var/log/sflow/goflow2
		

Create /etc/goflow2/mapping.yaml to ensure JSON fields match the integration's expectations:

formatter:
    fields:
        - type
        - time_flow_start_ns
        - sampler_address
        - sequence_num
        - in_if
        - out_if
        - src_addr
        - dst_addr
        - etype
        - proto
        - src_port
        - dst_port
        - src_vlan
        - dst_vlan
        - sampling_rate
        - bytes
		

Create /etc/systemd/system/goflow2.service with the following configuration:

[Unit]
Description=GoFlow2 sFlow Collector
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/goflow2 \
    -format json \
    -listen "sflow://:6343" \
    -mapping /etc/goflow2/mapping.yaml \
    -transport.file /var/log/sflow/goflow2/goflow2.log
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
		

Reload systemd, enable, and start the service:

sudo systemctl daemon-reload
sudo systemctl enable --now goflow2
		

Log into your network devices (switches/routers) and configure the sFlow destination to the IP of this server on UDP port 6343.

• GoFlow2 GitHub Repository
• GoFlow2 Configuration Guide - DeepWiki

1. In Kibana, navigate to Management > Integrations.
2. Search for and select GoFlow2.
3. Click Add GoFlow2.
4. Configure the Collecting logs via log file input:
   • Paths: Specify the list of paths to the log files that contain sFlow logs. Default: ['/var/log/sflow/goflow2/*.log'].
   • Preserve original event: Toggle this to preserve a raw copy of the original event in the event.original field. Default: False.
   • Ignore events older than: Set the duration to ignore events older than a specific time. Valid units: ns, us, ms, s, m, h. Default: 72h.
   • Exclude files: Add RE2 syntax regular expressions to match files that should be excluded. See exclude_files for details. Default: ['\\.gz$'].
   • Tags: Add custom tags to append to the events. Default: ['sflow', 'forwarded'].
   • Custom Filestream Options: Provide additional configuration options for the filestream input. See filestream input for details.
   • Processors: Define processors to enhance or filter data at the agent level. See Processors for details. Default: - add_locale: ~.
5. Provide a Name and Description for the integration instance.
6. Select the Agent Policy to which you want to add the integration.
7. Click Save and continue.

After configuration is complete, verify that data is flowing correctly.

1. Generate Network Traffic: Initiate traffic through a network device configured to export sFlow (e.g., run a ping sweep or a large file transfer through the switch).
2. Verify Log Generation: Check that GoFlow2 is actively writing to the log file by running the following command:

   tail -f /var/log/sflow/goflow2/goflow2.log
   		

3. Restart Service: Restart the GoFlow2 service to generate initial startup logs:

   sudo systemctl restart goflow2
   		

4. Check UDP Reception: Confirm the server is receiving traffic on the sFlow port:

   sudo tcpdump -i any udp port 6343
   		

1. Navigate to Analytics > Discover.
2. Select the logs-* data view.
3. Enter the KQL filter: data_stream.dataset : "goflow2.sflow"
4. Verify logs appear. Expand a log entry and confirm these fields are populated:
   • event.dataset (should be goflow2.sflow)
   • source.ip
   • destination.ip
   • event.action or event.outcome
   • message
5. Navigate to Analytics > Dashboards and search for "GoFlow2" to view pre-built visualizations.

For help with Elastic ingest tools, check Common problems.

• Log file permission issues:
  • Ensure the user running the Elastic Agent has read access to /var/log/sflow/goflow2/ and that the GoFlow2 process has write permissions.
• UDP port conflict:
  • If the GoFlow2 service fails to start, check if another process is already bound to port 6343/UDP using the command sudo netstat -tulpn | grep 6343.
• Mapping.yaml formatting errors:
  • YAML is sensitive to indentation. If GoFlow2 fails to parse the mapping file, ensure it matches the exact structure provided in the setup steps.
• Mismatched file paths:
  • Ensure the Paths variable provided in the Kibana integration configuration matches the -transport.file path in the GoFlow2 systemd service.
• Parsing failures:
  • If logs appear in Kibana but fields are not correctly mapped, check the error.message field in the log entry. This often indicates that the GoFlow2 output does not match the integration's expected JSON structure.
• Empty log files:
  • If the log file exists but is empty, verify that the network devices are successfully reaching the collector on port 6343/UDP using the command tcpdump -i any udp port 6343.
• Unsupported protocols:
  • If you attempt to send NetFlow or IPFIX data, GoFlow2 may log errors or output data that the integration pipeline cannot process, as only sFlow is currently supported.

To ensure optimal performance in high-volume environments, consider the following strategies:

• Transport and collection: The integration uses the filestream input to monitor local log files generated by the GoFlow2 binary. Ensure that the disk subsystem has sufficient I/O performance to handle simultaneous writes from the GoFlow2 collector and reads by the Elastic Agent. Use a robust log rotation strategy (e.g., logrotate) to prevent disk exhaustion, ensuring that the Elastic Agent has sufficient time to harvest logs before they are purged.
• Data volume management: In high-density network environments, it is critical to tune the sFlow sampling rate at the source (network hardware). A sampling rate of 1 in 1000 or 1 in 5000 is often sufficient for traffic analysis while significantly reducing the CPU load on the GoFlow2 collector and the event volume processed by the Elastic Agent.
• Elastic Agent scaling: For high-throughput environments, deploy the Elastic Agent on a dedicated host with multiple CPU cores to handle JSON parsing. If a single collector becomes a bottleneck, distribute your network devices across multiple GoFlow2 collectors and Elastic Agents to load-balance the ingest pipeline.

For more information on architectures that can be used for scaling this integration, check the Ingest Architectures documentation.

These inputs can be used with this integration:

The sflow data stream collects sFlow network traffic data processed by GoFlow2.

{
    "@timestamp": "2024-07-31T00:00:59.314Z",
    "destination": {
        "address": [
            "216.160.83.58"
        ],
        "as": {
            "number": 209
        },
        "geo": {
            "city_name": "Milton",
            "continent_name": "North America",
            "country_iso_code": "US",
            "country_name": "United States",
            "location": {
                "lat": 47.2513,
                "lon": -122.3149
            },
            "region_iso_code": "US-WA",
            "region_name": "Washington"
        },
        "ip": "216.160.83.58",
        "port": 443
    },
    "ecs": {
        "version": "8.17.0"
    },
    "event": {
        "action": "SFLOW_5",
        "category": [
            "network"
        ],
        "kind": "event",
        "original": "{\"type\":\"SFLOW_5\",\"time_flow_start_ns\":1722384059314899647,\"sampler_address\":\"67.43.156.1\",\"sequence_num\":44555,\"in_if\":563,\"out_if\":573,\"src_addr\":\"216.160.83.57\",\"dst_addr\":\"216.160.83.58\",\"etype\":\"IPv4\",\"proto\":\"TCP\",\"src_port\":10876,\"dst_port\":443,\"src_vlan\":1500,\"dst_vlan\":1500,\"sampling_rate\":1000,\"bytes\":70}",
        "timezone": "+00:00",
        "type": [
            "connection"
        ]
    },
    "network": {
        "bytes": 70000,
        "packets": 1000,
        "transport": "tcp",
        "type": "ipv4"
    },
    "observer": {
        "egress": {
            "interface": {
                "id": "573"
            },
            "vlan": {
                "id": "1500"
            }
        },
        "ingress": {
            "interface": {
                "id": "563"
            },
            "vlan": {
                "id": "1500"
            }
        },
        "ip": [
            "67.43.156.1"
        ]
    },
    "related": {
        "ip": [
            "216.160.83.57",
            "216.160.83.58"
        ]
    },
    "sflow": {
        "bytes": 70,
        "sample_rate": 1000,
        "sequence_num": 44555
    },
    "source": {
        "address": [
            "216.160.83.57"
        ],
        "as": {
            "number": 209
        },
        "geo": {
            "city_name": "Milton",
            "continent_name": "North America",
            "country_iso_code": "US",
            "country_name": "United States",
            "location": {
                "lat": 47.2513,
                "lon": -122.3149
            },
            "region_iso_code": "US-WA",
            "region_name": "Washington"
        },
        "ip": "216.160.83.57",
        "port": 10876
    },
    "tags": [
        "preserve_original_event",
        "forwarded",
        "sflow"
    ]
}
		

See the following resources for more information:

• GoFlow2 GitHub Repository