- Packetbeat Reference: other versions:
- Overview
- Get started
- Set up and run
- Upgrade Packetbeat
- Configure
- Traffic sniffing
- Network flows
- Protocols
- Processes
- General settings
- Project paths
- Output
- Kerberos
- SSL
- Index lifecycle management (ILM)
- Elasticsearch index template
- Kibana endpoint
- Kibana dashboards
- Processors
- Define processors
- add_cloud_metadata
- add_cloudfoundry_metadata
- add_docker_metadata
- add_fields
- add_host_metadata
- add_id
- add_kubernetes_metadata
- add_labels
- add_locale
- add_observer_metadata
- add_process_metadata
- add_tags
- community_id
- convert
- copy_fields
- decode_base64_field
- decode_json_fields
- decompress_gzip_field
- dissect
- dns
- drop_event
- drop_fields
- extract_array
- fingerprint
- include_fields
- registered_domain
- rename
- translate_sid
- truncate_fields
- urldecode
- Internal queue
- Logging
- HTTP endpoint
- packetbeat.reference.yml
- How to guides
- Exported fields
- AMQP fields
- Beat fields
- Cassandra fields
- Cloud provider metadata fields
- Common fields
- DHCPv4 fields
- DNS fields
- Docker fields
- ECS fields
- Flow Event fields
- Host fields
- HTTP fields
- ICMP fields
- Jolokia Discovery autodiscover provider fields
- Kubernetes fields
- Memcache fields
- MongoDb fields
- MySQL fields
- NFS fields
- PostgreSQL fields
- Process fields
- Raw fields
- Redis fields
- Thrift-RPC fields
- Detailed TLS fields
- Transaction Event fields
- Measurements (Transactions) fields
- Monitor
- Secure
- Visualize Packetbeat data in Kibana
- Troubleshoot
- Get help
- Debug
- Record a trace
- Common problems
- Dashboard in Kibana is breaking up data fields incorrectly
- Packetbeat doesn’t see any packets when using mirror ports
- Packetbeat can’t capture traffic from Windows loopback interface
- Packetbeat is missing long running transactions
- Packetbeat isn’t capturing MySQL performance data
- Packetbeat uses too much bandwidth
- Error loading config file
- Found unexpected or unknown characters
- Logstash connection doesn’t work
- @metadata is missing in Logstash
- Not sure whether to use Logstash or Beats
- SSL client fails to connect to Logstash
- Monitoring UI shows fewer Beats than expected
- Dashboard could not locate the index-pattern
- Fields show up as nested JSON in Kibana
- Contribute to Beats
Use Metricbeat to send monitoring data
editUse Metricbeat to send monitoring data
editIn 7.3 and later, you can use Metricbeat to collect data about Packetbeat and ship it to the monitoring cluster. The benefit of using Metricbeat instead of internal collection is that the monitoring agent remains active even if the Packetbeat instance dies.
To collect and ship monitoring data:
Configure the shipper you want to monitor
edit-
Enable the HTTP endpoint to allow external collection of monitoring data:
Add the following setting in the Packetbeat configuration file (
packetbeat.yml
):http.enabled: true
By default, metrics are exposed on port 5066. If you need to monitor multiple Beats shippers running on the same server, set
http.port
to expose metrics for each shipper on a different port number:http.port: 5067
-
Disable the default collection of Packetbeat monitoring metrics.
Add the following setting in the Packetbeat configuration file (
packetbeat.yml
):monitoring.enabled: false
For more information, see Monitoring configuration options.
- Start Packetbeat.
Install and configure Metricbeat to collect monitoring data
edit- Install Metricbeat on the same server as Packetbeat. If you already have Metricbeat installed on the server, skip this step.
-
Enable the
beat-xpack
module in Metricbeat.For example, to enable the default configuration in the
modules.d
directory, run the following command, using the correct command syntax for your OS:metricbeat modules enable beat-xpack
For more information, see Configure modules and beat module.
-
Configure the
beat-xpack
module in Metricbeat.The
modules.d/beat-xpack.yml
file contains the following settings:- module: beat metricsets: - stats - state period: 10s hosts: ["http://localhost:5066"] #username: "user" #password: "secret" xpack.enabled: true
Set the
hosts
,username
, andpassword
settings as required by your environment. For other module settings, it’s recommended that you accept the defaults.By default, the module collects Packetbeat monitoring data from
localhost:5066
. If you exposed the metrics on a different host or port when you enabled the HTTP endpoint, update thehosts
setting.To monitor multiple Beats agents, specify a list of hosts, for example:
hosts: ["http://localhost:5066","http://localhost:5067","http://localhost:5068"]
If you configured Packetbeat to use encrypted communications, you must access it via HTTPS. For example, use a
hosts
setting likehttps://localhost:5066
.If the Elastic security features are enabled, you must also provide a user ID and password so that Metricbeat can collect metrics successfully:
-
Create a user on the Elasticsearch cluster that has the
remote_monitoring_collector
built-in role. Alternatively, if it’s available in your environment, use theremote_monitoring_user
built-in user. -
Add the
username
andpassword
settings to the beat module configuration file.
-
Create a user on the Elasticsearch cluster that has the
-
Optional: Disable the system module in the Metricbeat.
By default, the system module is enabled. The information it collects, however, is not shown on the Stack Monitoring page in Kibana. Unless you want to use that information for other purposes, run the following command:
metricbeat modules disable system
-
Identify where to send the monitoring data.
In production environments, we strongly recommend using a separate cluster (referred to as the monitoring cluster) to store the data. Using a separate monitoring cluster prevents production cluster outages from impacting your ability to access your monitoring data. It also prevents monitoring activities from impacting the performance of your production cluster.
For example, specify the Elasticsearch output information in the Metricbeat configuration file (
metricbeat.yml
):output.elasticsearch: # Array of hosts to connect to. hosts: ["http://es-mon-1:9200", "http://es-mon2:9200"] # Optional protocol and basic auth credentials. #protocol: "https" #api_key: "id:api_key" #username: "elastic" #password: "changeme"
In this example, the data is stored on a monitoring cluster with nodes
es-mon-1
andes-mon-2
.Specify one of
api_key
orusername
/password
.If you configured the monitoring cluster to use encrypted communications, you must access it via HTTPS. For example, use a
hosts
setting likehttps://es-mon-1:9200
.The Elasticsearch monitoring features use ingest pipelines, therefore the cluster that stores the monitoring data must have at least one ingest node.
If the Elasticsearch security features are enabled on the monitoring cluster, you must provide a valid user ID and password so that Metricbeat can send metrics successfully:
-
Create a user on the monitoring cluster that has the
remote_monitoring_agent
built-in role. Alternatively, if it’s available in your environment, use theremote_monitoring_user
built-in user.If you’re using index lifecycle management, the remote monitoring user requires additional privileges to create and read indices. For more information, see Grant users access to secured resources.
-
Add the
username
andpassword
settings to the Elasticsearch output information in the Metricbeat configuration file.
For more information about these configuration options, see Configure the Elasticsearch output.
-
- Start Metricbeat to begin collecting monitoring data.
- View the monitoring data in Kibana.
On this page
ElasticON events are back!
Learn about the Elastic Search AI Platform from the experts at our live events.
Register now