Elastic SIEM for home and small business: Beats on CentOS
Note: The “SIEM for home and small business” blog series contains configurations relevant to the beta release of Elastic SIEM using Elastic Stack 7.4. We recommend using Elastic Stack 7.6 and newer, as Elastic SIEM was made generally available in 7.6. Please also note the Elastic SIEM solution mentioned in this post is now referred to as Elastic Security.
Hey, there. This is part five of the Elastic SIEM for home and small business blog series. If you haven’t read the first, second, and third blogs, you may want to before going any further. In the Getting started blog, we created our Elasticsearch Service deployment and started collecting data from one of our computers using Winlogbeat. In the Securing cluster access blog, we secured access to our cluster by restricting privileges for users and Beats. In the GeoIP data and Beats config blog, we created an ingest pipeline for GeoIP data and reviewed our Beats configurations.
Identifying our data collection needs on CentOS
In the first blog, we determined that we need to use Auditbeat, Filebeat, Packetbeat, and Winlogbeat to collect log files, activities of users and processes, and network data. We do not need to install all of those Beats applications on our Linux devices, only the Beats we need to collect data relevant to us. In this blog, we will install and configure Auditbeat, Filebeat, and Packetbeat on CentOS.
For data collection from our Linux systems, we will focus on activities of users and processes, log files, and network data. We will use Auditbeat to collect file integrity and system information, Filebeat to collect auditd and log data, and Packetbeat to collect system-specific network data. In this blog, we will review the Repositories for YUM guide in order to install Beats with Yum. In order to get started we need administrative access to our CentOS system, then we need to add the Beats repository for Yum, and a copy of our Beats common configuration settings.
Determining the event data we want to collect
We will primarily collect system information from Auditbeat since the System module for Auditbeat is fully implemented for Linux — meaning most of our information will come from Auditbeat. Also, while we have the option to use the Auditd module for Auditbeat to collect audit events, we will instead use the Auditd module for Filebeat as an example of how to configure modules in Filebeat. Regarding log collection with Filebeat, we can review the log files that exist in /var/log on our CentOS system and decide if there are any additional logs we want to collect.
The CentOS system we will collect data from is primarily used for Pi-hole, so we will leverage Packetbeat to capture DNS traffic. While there are many different ways to configure event collection from Pi-hole (a few examples are using Logstash with elk-hole or DNS-ELK, or by using an ECS compliant ingest pipeline), this blog will focus on data collection from CentOS while highlighting some example configuration options.
Using the GeoIP ingest pipeline
Before we start installing and configuring Beats on our CentOS system, I want to cover the GeoIP ingest processor issue in more detail. There is a workaround that will allow us to enrich data with GeoIP information, but it depends on many variables. Essentially, you will have to enable logging for each respective Beat, then look at the logs to confirm data is being sent to your Elasticsearch Service deployment.
In the GeoIP data and Beats config blog, we created an ingest pipeline for GeoIP data, then we commented out the output.elasticsearch.pipeline: geoip-info setting. The reason I disabled use of the GeoIP ingest processor was to reduce (or prevent) frustration. Quite simply, you can read through (and follow) Elasticsearch GitHub issue 46193 for more detail around why I have currently chosen to disable these settings.
At a high level, the GeoIP ingest processor does not allow an array of IP addresses. This is a problem for systems running both IPv4 and IPv6, as well as systems with multiple network adapters (both physical and virtual). Any time Beats passes an array of IPs to Elasticsearch while using the GeoIP ingest processor a Failed to publish events: temporary bulk send failure error message will be generated when netinfo.enabled: true is set on devices with multiple interfaces or addresses, Beats will include IP addresses and MAC addresses as fields host.ip and host.mac. This means the respective Beat will attempt to send an array of values for host.ip that causes the error.
While the example Beats common configurations have netinfo.enabled: false (instead of true) and commented out the output.elasticsearch.pipeline: geoip-info setting, the way to tell Beats to use the GeoIP ingest pipeline is to uncomment the output.elasticsearch.pipeline: geoip-info setting in your Beats configuration file.
If you decide to use the GeoIP ingest pipeline, you should also configure logging for each Beat. The reason we need to configure logging is that the Beat will run even though data is not being sent to Elasticsearch. For this blog, we will store our logs in the /var/log/elastic directory, but you will need to look for an Failed to publish events: temporary bulk send failure error message that will look similar to this:
ERROR pipeline/output.go:121 Failed to publish events: temporary bulk send failure
Again, this has to do with passing an array of IPs to Elasticsearch using the GeoIP ingest processor. With that said, feel free to follow Elasticsearch GitHub issue 46193.
Configuring the Beats repository for YUM
The Beats repository for YUM guide provides instructions on how to add the Beats repository for YUM on our CentOS server. The guide tells us we need to download and install the public signing key, then create an elastic.repo file in our /etc/yum.repos.d/ directory.
[myuser@centos7pi ~]$ sudo rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch [sudo] password for myuser: [myuser@centos7pi ~]$ echo "Now we will create the elastic.repo file as defined in the YUM repo guide" # Now we will create the elastic.repo file as defined in the YUM repo guide [myuser@centos7pi ~]$ sudo vi /etc/yum.repos.d/elastic.repo [myuser@centos7pi ~]$ echo "Now we will validate the elastic.repo file" # Now we will validate the elastic.repo file [myuser@centos7pi ~]$ sudo cat /etc/yum.repos.d/elastic.repo [elastic-7.x] name=Elastic repository for 7.x packages baseurl=https://artifacts.elastic.co/packages/7.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md [myuser@centos7pi ~]$
We can continue now that the repository is ready to use.
Auditbeat on CentOS
Installing Auditbeat
Since we have our Beats repository for YUM added, we will download and install Auditbeat by issuing the sudo yum install auditbeat command. Once we have our configuration file ready, we will also configure Auditbeat to start automatically during boot. After installing Auditbeat, do not start the service until we finish initial configuration.
[myuser@centos7pi ~]$ sudo yum install auditbeat [sudo] password for myuser: Is this ok [y/d/N]: y Downloading packages: auditbeat-7.4.0-x86_64.rpm | 21 MB 00:00:03 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : auditbeat-7.4.0-1.x86_64 1/1 Verifying : auditbeat-7.4.0-1.x86_64 1/1 Installed: auditbeat.x86_64 0:7.4.0-1 Complete! [myuser@centos7pi ~]$
Remember, do not start the Auditbeat service until we finish initial configuration.
Configuring Auditbeat
Now that Auditbeat is installed, we need to update our auditbeat.yml configuration file. Since we have already determined our common Beats configurations, we only need to finalize our Auditbeat-specific options, which will be stored in the first section of the file. Make sure to read through the auditbeat.reference.yml page or the auditbeat.reference.yml file in the /etc/auditbeat directory. For clarity, the Beats common configuration sections will be stored below the Auditbeat-specific options.
If you plan on using the GeoIP ingest processor, you should configure logging for Auditbeat. Since we will store our log files in the /var/log/elastic directory, our logging configuration will look like this:
#=== Auditbeat logging === # Initially use `info` for the logging.level, set logging.level to `debug` if you see # an `Failed to publish events: temporary bulk send failure` error message in the logs logging.level: info logging.to_files: true logging.files: path: /var/log/elastic name: auditbeat keepfiles: 7 permissions: 0644
We will add the Auditbeat logging section to our configuration, but leave it commented out for now. With that said, after issuing sudo cat /etc/auditbeat/auditbeat.yml here is what I have set in auditbeat.yml:
##=== Auditbeat specific options ===
#=== Modules configuration ===
auditbeat.modules:
- module: file_integrity
paths:
- /bin
- /usr/bin
- /sbin
- /usr/sbin
- /etc
- module: system
datasets:
- user
- login
user.detect_password_changes: true
period: 10s
state.period: 12h
- module: system
datasets:
- package
- host
period: 30m
state.period: 12h
- module: system
datasets:
- socket
socket.include_localhost: false
period: 3s
- module: system
datasets:
- process
processors:
- drop_event.when:
or:
- contains.event.action: "existing_process"
- contains.event.action: "process_error"
- add_process_metadata:
match_pids: [process.ppid]
target: process.parent
period: 5s
#=== Auditbeat logging ===
# Configure logging for Auditbeat if you plan on using the GeoIP ingest processor
# Initially use `info` for the logging.level, set logging.level to `debug` if you see
# an `Failed to publish events: temporary bulk send failure` error message in the logs
#logging.level: info
#logging.to_files: true
#logging.files:
# path: /var/log/elastic
# name: auditbeat
# keepfiles: 7
# permissions: 0644
#=== Beats Common Configs Here ===
You may have noticed I have added two drop_event.when conditions as an example using the drop_event processor. You may have also noticed I specified state.period and period in the system module section. The period parameter controls the poll frequency while state.period controls the frequency of state updates. I also included the add process metadata processor to enrich events collected by the system module. For reference, this example configuration can be found in the examples GitHub repo under Security Analytics/SIEM-at-Home.
Next, we will add the settings from our Beats common configuration settings to our auditbeat.yml configuration file. We will use the settings from the General, Top Level Processor, Elastic Cloud, Xpack Monitoring, and Queue sections and add them to the end of our configuration file. Our completed auditbeat.yml file now looks like this:
#=== Auditbeat specific options ===
#=== Modules configuration ===
auditbeat.modules:
- module: file_integrity
paths:
- /bin
- /usr/bin
- /sbin
- /usr/sbin
- /etc
- module: system
datasets:
- user
- login
user.detect_password_changes: true
period: 10s
state.period: 12h
- module: system
datasets:
- package
- host
period: 30m
state.period: 12h
- module: system
datasets:
- socket
socket.include_localhost: false
period: 3s
- module: system
datasets:
- process
processors:
- drop_event.when:
or:
- contains.event.action: "existing_process"
- contains.event.action: "process_error"
- add_process_metadata:
match_pids: [process.ppid]
target: process.parent
period: 5s
#=== Auditbeat logging ===
# Configure logging for Auditbeat if you plan on using the GeoIP ingest processor
# Initially use `info` for the logging.level, set logging.level to `debug` if you see
# an `Failed to publish events: temporary bulk send failure` error message in the logs
#logging.level: info
#logging.to_files: true
#logging.files:
# path: /var/log/elastic
# name: auditbeat
# keepfiles: 7
# permissions: 0644
# Configuration applicable for all beats on a specific device
#=== General ===
name: C7PiHole
tags: ["myTag", "C7PiHole"]
fields:
env: myEnv
version: 12-18-2019
#=== Top Level Processor ===
processors:
- add_host_metadata:
# netinfo.enabled should be set to `false` until GitHub issue
# https://github.com/elastic/elasticsearch/issues/46193 is resolved
netinfo.enabled: false
Geo: # These Geo configurations are optional
location: 40.7128, -74.0060
continent_name: North America
country_iso_code: US
region_name: New York
region_iso_code: US-NY
city_name: New York City
name: myHomeLocation
- add_locale: ~
# - add_cloud_metadata: ~
- add_fields:
#when.network.source.ip: 10.101.101.0/24
when.network.source.ip: private
fields:
source.geo.location:
lat: 40.7128
lon: -74.0060
source.geo.continent_name: North America
source.geo.country_iso_code: US
source.geo.region_name: New York
source.geo.region_iso_code: US-NY
source.geo.city_name: New York City
source.geo.name: myHomeLocation
target: ''
- add_fields:
#when.network.destination.ip: 10.101.101.0/24
when.network.destination.ip: private
fields:
destination.geo.location:
lat: 40.7128
lon: -74.0060
destination.geo.continent_name: North America
destination.geo.country_iso_code: US
destination.geo.region_name: New York
destination.geo.region_iso_code: US-NY
destination.geo.city_name: New York City
destination.geo.name: myHomeLocation
target: ''
#=== Elastic Cloud ===
# These settings simplify using beats with the Elastic Cloud (https://cloud.elastic.co/).
cloud.id: "My_Elastic_Cloud_Deployment:abcdefghijklmnopqrstuvwxyz1234567890ABCDEFGHIJKLMNOPQRSTUVWXYZ"
cloud.auth: "home_beats:0987654321abcDEF"
# The geoip-info pipeline is used to enrich GeoIP information in Elasticsearch
# You must configure the pipeline in Elasticsearch before enabling the pipeline in Beats.
# The `output.elasticsearch.pipeline: geoip-info` setting should be commented out until
# until GitHub issue https://github.com/elastic/elasticsearch/issues/46193 is resolved
#output.elasticsearch.pipeline: geoip-info
# The `max_retries` setting is the number of times to retry publishing an event after
# a publishing failure. After the specified number of retries, the events are typically dropped.
output.elasticsearch.max_retries: 5
# When deploying beats to multiple systems or locations, uncomment the following
# setup.template.* and setup.ilm.* configurations after running the beats setup command
# Otherwise, uncomment the following configurations for beats that will only publish data
#setup.template.enabled: false
#setup.ilm.check_exists: false
#setup.ilm.overwrite: false
#=== Xpack Monitoring ===
# When monitoring is enabled, the beat will ship monitoring data to the cluster
monitoring.enabled: true
#=== Queue ===
# See the 'Configure the internal queue' documentation for each Beat before
# configuring the queue. Note that only one queue type can be configured.
# You need to uncomment the specific queue type you decide to use.
# The `queue.mem` settings will cache events to memory in case access to the
# Elasticsearch cluster, via the internet, is unavailable (internet outage)
#queue.mem:
# events: 4096
# flush.min_events: 512
# flush.timeout: 1s
# The `queue.spool` settings will cache events to disk in case the system is offline
# NOTE: The file spool queue is a beta functionality in 7.4
queue.spool:
file:
size: 512MiB
page_size: 16KiB
prealloc: ~
write:
buffer_size: 10MiB
flush.timeout: 1s
flush.min.events: 1024
Once the file is updated, don’t forget to save the changes made to auditbeat.yml before proceeding.
Starting Auditbeat
Now that the configuration is updated, it's time to set up Auditbeat with our Elasticsearch Service deployment. Running the setup command only needs to be performed once for each beat (for instance only once for Auditbeat 7.4.0). In our console, we will navigate to the /usr/share/auditbeat directory, then issue the sudo bin/auditbeat -c /etc/auditbeat/auditbeat.yml -path.home /usr/share/auditbeat setup command. Then we will confirm we receive “Index setup finished” and “Loaded dashboards” messages (shown below).
[myuser@centos7pi ~]$ cd /usr/share/auditbeat [myuser@centos7pi auditbeat]$ sudo bin/auditbeat -c /etc/auditbeat/auditbeat.yml -path.home /usr/share/auditbeat setup Index setup finished. Loading dashboards (Kibana must be running and reachable) Loaded dashboards [myuser@centos7pi auditbeat]$
Now that we’ve set up the index and loaded the dashboards, we'll enable the Auditbeat service then start the Auditbeat service.
[myuser@centos7pi auditbeat]$ sudo systemctl enable auditbeat
Created symlink from /etc/systemd/system/multi-user.target.wants/auditbeat.service to /usr/lib/systemd/system/auditbeat.service.
[myuser@centos7pi auditbeat]$ sudo systemctl start auditbeat
[myuser@centos7pi auditbeat]$ sudo systemctl status auditbeat
● auditbeat.service - Audit the activities of users and processes on your system.
Loaded: loaded (/usr/lib/systemd/system/auditbeat.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2019-12-11 16:39:34 CST; 22s ago
Docs: https://www.elastic.co/products/beats/auditbeat
Main PID: 12558 (auditbeat)
CGroup: /system.slice/auditbeat.service
└─12558 /usr/share/auditbeat/bin/auditbeat -e -c /etc/auditbeat/auditbeat.yml -path.home /...
Dec 11 16:39:54 centos7pi.my.domain auditbeat[12558]: 2019-12-11T16:39:54.338-0600 IN...se
Hint: Some lines were ellipsized, use -l to show in full.
[myuser@centos7pi auditbeat]$ cd ~
[myuser@centos7pi ~]$
We will navigate back to our home directory in our console so we can install and configure Filebeat and Packetbeat on this CentOS system. If you are using the GeoIP ingest pipeline and do not see data coming from Auditbeat for this device in your Elastic cluster, you should uncomment the logging section and restart Auditbeat.
Filebeat on CentOS
Installing Filebeat
Since we have our Beats repository for YUM added, we will download and install Filebeat by issuing the sudo yum install filebeat command. Once we have our configuration file ready, we will also configure Filebeat to start automatically during boot. After installing Filebeat, do not start the service until we finish initial configuration.
[myuser@centos7pi ~]$ sudo yum install filebeat [sudo] password for myuser: Is this ok [y/d/N]: y Downloading packages: filebeat-7.4.0-x86_64.rpm | 23 MB 00:00:04 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : filebeat-7.4.0-1.x86_64 1/1 Verifying : filebeat-7.4.0-1.x86_64 1/1 Installed: filebeat.x86_64 0:7.4.0-1 Complete! [myuser@centos7pi ~]$
Remember, do not start the Filebeat service until we finish initial configuration.
Configuring Filebeat
Now that Filebeat is installed, we need to update our filebeat.yml configuration file. Since we have already determined our common Beats configurations, we only need to finalize our Filebeat-specific options, which will be stored in the first section of the file. Make sure to read through the filebeat.reference.yml page or the filebeat.reference.yml file in the /etc/filebeat directory. For clarity, the Beats common configuration sections will be stored below the Filebeat specific options.
If you plan on using the GeoIP ingest processor, you should configure logging for Filebeat. Since we will store our log files in the /var/log/elastic directory, our logging configuration will look like this:
#=== Filebeat logging === # Initially use `info` for the logging.level, set logging.level to `debug` if you see # an `Failed to publish events: temporary bulk send failure` error message in the logs logging.level: info logging.to_files: true logging.files: path: /var/log/elastic name: filebeat keepfiles: 7 permissions: 0644
We will add the Filebeat logging section to our configuration but leave it commented out for now. With that said, after issuing sudo cat /etc/filebeat/filebeat.yml here is what I have set in filebeat.yml:
#=== Filebeat specific options ===
#=== Filebeat modules ===
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
reload.period: 30s
#=== Filebeat inputs ===
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/yum.log
close_inactive: 1m
- type: log
enabled: true
paths:
- /var/log/pihole.log
fields:
app: pihole
name: pihole
- type: log
enabled: true
paths:
- /var/log/pihole-FTL.log
fields:
app: pihole
name: pihole-FTL
- type: log
enabled: true
paths:
- /var/log/pihole_updateGravity.log
fields:
app: pihole
name: pihole-updateGravity
- type: log
enabled: true
paths:
- /var/log/lighttpd/*.log
fields:
app: pihole
name: pihole-lighttpd
#=== Filebeat logging ===
# Configure logging for Filebeat if you plan on using the GeoIP ingest processor
# Initially use `info` for the logging.level, set logging.level to `debug` if you see
# an `Failed to publish events: temporary bulk send failure` error message in the logs
#logging.level: info
#logging.to_files: true
#logging.files:
# path: /var/log/elastic
# name: filebeat
# keepfiles: 7
# permissions: 0644
#=== Beats Common Configs Here ===
You may have noticed the entry for Yum logs contains a modified close_inactive setting — this configuration example tells Filebeat to close the file handle if the yum log has not been harvested for one minute. You may also have noticed the four entries for Pi-hole logs that will enrich our Pi-hole data with the app and name fields for the individual log inputs. For reference, this example configuration can be found in the examples GitHub repo under Security Analytics/SIEM-at-Home.
Next, we will add the settings from our Beats common configuration settings to our filebeat.yml configuration file. We will use the settings from the General, Top Level Processor, Elastic Cloud, Xpack Monitoring, and Queue sections and add them to the end of our configuration file. Our completed filebeat.yml file now looks like this:
#=== Filebeat specific options ===
#=== Filebeat modules ===
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
reload.period: 30s
#=== Filebeat inputs ===
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/yum.log
close_inactive: 1m
- type: log
enabled: true
paths:
- /var/log/pihole.log
fields:
app: pihole
name: pihole
- type: log
enabled: true
paths:
- /var/log/pihole-FTL.log
fields:
app: pihole
name: pihole-FTL
- type: log
enabled: true
paths:
- /var/log/pihole_updateGravity.log
fields:
app: pihole
name: pihole-updateGravity
- type: log
enabled: true
paths:
- /var/log/lighttpd/*.log
fields:
app: pihole
name: pihole-lighttpd
#=== Filebeat logging ===
# Configure logging for Filebeat if you plan on using the GeoIP ingest processor
# Initially use `info` for the logging.level, set logging.level to `debug` if you see
# an `Failed to publish events: temporary bulk send failure` error message in the logs
#logging.level: info
#logging.to_files: true
#logging.files:
# path: /var/log/elastic
# name: filebeat
# keepfiles: 7
# permissions: 0644
# Configuration applicable for all beats on a specific device
#=== General ===
name: C7PiHole
tags: ["myTag", "C7PiHole"]
fields:
env: myEnv
version: 12-18-2019
#=== Top Level Processor ===
processors:
- add_host_metadata:
# netinfo.enabled should be set to `false` until GitHub issue
# https://github.com/elastic/elasticsearch/issues/46193 is resolved
netinfo.enabled: false
Geo: # These Geo configurations are optional
location: 40.7128, -74.0060
continent_name: North America
country_iso_code: US
region_name: New York
region_iso_code: US-NY
city_name: New York City
name: myHomeLocation
- add_locale: ~
# - add_cloud_metadata: ~
- add_fields:
#when.network.source.ip: 10.101.101.0/24
when.network.source.ip: private
fields:
source.geo.location:
lat: 40.7128
lon: -74.0060
source.geo.continent_name: North America
source.geo.country_iso_code: US
source.geo.region_name: New York
source.geo.region_iso_code: US-NY
source.geo.city_name: New York City
source.geo.name: myHomeLocation
target: ''
- add_fields:
#when.network.destination.ip: 10.101.101.0/24
when.network.destination.ip: private
fields:
destination.geo.location:
lat: 40.7128
lon: -74.0060
destination.geo.continent_name: North America
destination.geo.country_iso_code: US
destination.geo.region_name: New York
destination.geo.region_iso_code: US-NY
destination.geo.city_name: New York City
destination.geo.name: myHomeLocation
target: ''
#=== Elastic Cloud ===
# These settings simplify using beats with the Elastic Cloud (https://cloud.elastic.co/).
cloud.id: "My_Elastic_Cloud_Deployment:abcdefghijklmnopqrstuvwxyz1234567890ABCDEFGHIJKLMNOPQRSTUVWXYZ"
cloud.auth: "home_beats:0987654321abcDEF"
# The geoip-info pipeline is used to enrich GeoIP information in Elasticsearch
# You must configure the pipeline in Elasticsearch before enabling the pipeline in Beats.
# The `output.elasticsearch.pipeline: geoip-info` setting should be commented out until
# until GitHub issue https://github.com/elastic/elasticsearch/issues/46193 is resolved
#output.elasticsearch.pipeline: geoip-info
# The `max_retries` setting is the number of times to retry publishing an event after
# a publishing failure. After the specified number of retries, the events are typically dropped.
output.elasticsearch.max_retries: 5
# When deploying beats to multiple systems or locations, uncomment the following
# setup.template.* and setup.ilm.* configurations after running the beats setup command
# Otherwise, uncomment the following configurations for beats that will only publish data
#setup.template.enabled: false
#setup.ilm.check_exists: false
#setup.ilm.overwrite: false
#=== Xpack Monitoring ===
# When monitoring is enabled, the beat will ship monitoring data to the cluster
monitoring.enabled: true
#=== Queue ===
# See the 'Configure the internal queue' documentation for each Beat before
# configuring the queue. Note that only one queue type can be configured.
# You need to uncomment the specific queue type you decide to use.
# The `queue.mem` settings will cache events to memory in case access to the
# Elasticsearch cluster, via the internet, is unavailable (internet outage)
#queue.mem:
# events: 4096
# flush.min_events: 512
# flush.timeout: 1s
# The `queue.spool` settings will cache events to disk in case the system is offline
# NOTE: The file spool queue is a beta functionality in 7.4
queue.spool:
file:
size: 512MiB
page_size: 16KiB
prealloc: ~
write:
buffer_size: 10MiB
flush.timeout: 1s
flush.min.events: 1024
Once the file is updated, don’t forget to save the changes made to filebeat.yml before proceeding.
Configuring Filebeat Modules
Next, we need to enable a few Filebeat Modules, which will simplify the collection, parsing, and visualization of common log formats. Since we installed Filebeat with YUM, the default modules are disabled but available in the /etc/filebeat/modules.d/ directory. For your awareness, the file name of the modules that are disabled end in .disabled. On this system we will use the auditd and system modules.
[myuser@centos7pi ~]$ sudo filebeat modules list Enabled: Disabled: #[...] [myuser@centos7pi ~]$ sudo filebeat modules enable auditd Enabled auditd [myuser@centos7pi ~]$ sudo filebeat modules enable system Enabled system [myuser@centos7pi ~]$ sudo filebeat modules list Enabled: auditd system Disabled: #[...] [myuser@centos7pi ~]$
We can continue now that the auditd and system modules are enabled.
Starting Filebeat
Now that the configuration is updated, it is time to set up Filebeat with our Elasticsearch Service deployment. Running the setup command only needs to be performed once for each beat (for instance, only once for Filebeat 7.4.0). In our console, we will navigate to the /usr/share/filebeat directory then issue the sudo bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat setup command. Then we will confirm we receive “Index setup finished”, “Loaded dashboards”, “Loaded machine learning job configurations”, and “Loaded Ingest pipelines” messages (shown below).
[myuser@centos7pi ~]$ cd /usr/share/filebeat [myuser@centos7pi filebeat]$ sudo bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat setup Index setup finished. Loading dashboards (Kibana must be running and reachable) Loaded dashboards Loaded machine learning job configurations Loaded Ingest pipelines [myuser@centos7pi filebeat]$
Now that we’ve set up the index and loaded the dashboards, we can enable the Filebeat service and then start the Filebeat service.
[myuser@centos7pi filebeat]$ sudo systemctl enable filebeat
Created symlink from /etc/systemd/system/multi-user.target.wants/filebeat.service to /usr/lib/systemd/system/filebeat.service.
[myuser@centos7pi filebeat]$ sudo systemctl start filebeat
[myuser@centos7pi filebeat]$ sudo systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2019-12-11 22:34:36 CST; 5s ago
Docs: https://www.elastic.co/products/beats/filebeat
Main PID: 12869 (filebeat)
CGroup: /system.slice/filebeat.service
└─12869 /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/...
Dec 11 22:34:40 centos7pi.my.domain filebeat[12869]: 2019-12-11T22:34:40.351-0600 INF...ed
Hint: Some lines were ellipsized, use -l to show in full.
[myuser@centos7pi filebeat]$ cd ~
[myuser@centos7pi ~]$
We will navigate back to our home directory in our console so that we can install and configure Packetbeat on this CentOS system. If you are using the GeoIP ingest pipeline and do not see data coming from Filebeat for this device in your Elastic cluster, you should uncomment the logging section and restart Filebeat.
Packetbeat on CentOS
While it is ideal to deploy Packetbeat to either existing application servers or dedicated servers (to get traffic from mirror ports or tap devices), we will leverage Packetbeat to collect DNS traffic since we are running Pi-hole on this system.
Installing Packetbeat
Since we have our Beats repository for YUM added, we will download and install Packetbeat by issuing the sudo yum install packetbeat command. Once we have our configuration file ready, we will also configure Packetbeat to start automatically during boot. After installing Packetbeat, do not start the service until we finish initial configuration.
[myuser@centos7pi ~]$ sudo yum install packetbeat [sudo] password for myuser: Is this ok [y/d/N]: y Downloading packages: packetbeat-7.4.0-x86_64.rpm | 20 MB 00:00:04 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : packetbeat-7.4.0-1.x86_64 1/1 Verifying : packetbeat-7.4.0-1.x86_64 1/1 Installed: packetbeat.x86_64 0:7.4.0-1 Complete! [myuser@centos7pi ~]$
Remember, do not start the Packetbeat service until we finish initial configuration.
Configuring Packetbeat
Now that Packetbeat is installed, we need to update our packetbeat.yml configuration file. Since we have already determined our common Beats configurations, we only need to finalize our Packetbeat-specific options, which will be stored in the first section of the file. Make sure to read through the packetbeat.reference.yml page or the packetbeat.reference.yml file in the /etc/packetbeat directory. For clarity, the Beats common configuration sections will be stored below the Packetbeat-specific options.
If you plan on using the GeoIP ingest processor, you should configure logging for Packetbeat. Since we will store our log files in the /var/log/elastic directory, our logging configuration will look like this:
#=== Packetbeat logging === # Initially use `info` for the logging.level, set logging.level to `debug` if you see # an `Failed to publish events: temporary bulk send failure` error message in the logs logging.level: info logging.to_files: true logging.files: path: /var/log/elastic name: packetbeat keepfiles: 7 permissions: 0644
We will add the Packetbeat logging section to our configuration but leave it commented out for now. With that said, after issuing sudo cat /etc/packetbeat/packetbeat.yml here is what I have set in packetbeat.yml:
#=== Packetbeat specific options ===
#=== Network device ===
# Select the network interface to sniff the data. On Linux, you can use the
# "any" keyword to sniff on all connected interfaces.
packetbeat.interfaces.device: any
#=== Flows ===
packetbeat.flows:
timeout: 30s
period: 10s
#=== Transaction protocols ===
# For more information on the transaction protocols, see
# https://www.elastic.co/guide/en/beats/packetbeat/7.4/configuration-protocols.html
packetbeat.protocols:
- type: icmp
# Enable ICMPv4 and ICMPv6 monitoring. Default: false
enabled: true
- type: dhcpv4
# Configure the DHCP for IPv4 ports.
ports: [67, 68]
send_request: true
send_response: true
- type: dns
# Configure the ports where to listen for DNS traffic. You can disable
# the DNS protocol by commenting out the list of ports.
ports: [53]
include_authorities: true
include_additionals: true
send_request: true
send_response: true
- type: http
# Configure the ports where to listen for HTTP traffic. You can disable
# the HTTP protocol by commenting out the list of ports.
ports: [80, 8080, 8000, 5000, 8002]
- type: tls
# Configure the ports where to listen for TLS traffic. You can disable
# the TLS protocol by commenting out the list of ports.
ports:
- 443 # HTTPS
- 993 # IMAPS
- 995 # POP3S
- 5223 # XMPP over SSL
- 8443
- 8883 # Secure MQTT
- 9243 # Elasticsearch
#=== Packetbeat logging ===
# Configure logging for Packetbeat if you plan on using the GeoIP ingest processor
# Initially use `info` for the logging.level, set logging.level to `debug` if you see
# an `Failed to publish events: temporary bulk send failure` error message in the logs
#logging.level: info
#logging.to_files: true
#logging.files:
# path: /var/log/elastic
# name: packetbeat
# keepfiles: 7
# permissions: 0644
#=== Beats Common Configs Here ===
You may have noticed the include_authorities: true and include_additionals: true entries in the DNS transaction protocol section, which is an example using DNS-specific configuration options. You may have also noticed the send_request: true and send_response: true entries in both the DHCP and DNS transaction protocol sections, which are examples using common protocol options. For reference, this example configuration can be found in the examples GitHub repo under Security Analytics/SIEM-at-Home.
Next, we will add the settings from our Beats common configuration settings to our packetbeat.yml configuration file. We will use the settings from the General, Top Level Processor, Elastic Cloud, Xpack Monitoring, and Queue sections and add them to the end of our configuration file. Our completed packetbeat.yml file now looks like this:
#=== Packetbeat specific options ===
#=== Network device ===
# Select the network interface to sniff the data. On Linux, you can use the
# "any" keyword to sniff on all connected interfaces.
packetbeat.interfaces.device: any
#=== Flows ===
packetbeat.flows:
timeout: 30s
period: 10s
#=== Transaction protocols ===
# For more information on the transaction protocols, see
# https://www.elastic.co/guide/en/beats/packetbeat/7.4/configuration-protocols.html
packetbeat.protocols:
- type: icmp
# Enable ICMPv4 and ICMPv6 monitoring. Default: false
enabled: true
- type: dhcpv4
# Configure the DHCP for IPv4 ports.
ports: [67, 68]
send_request: true
send_response: true
- type: dns
# Configure the ports where to listen for DNS traffic. You can disable
# the DNS protocol by commenting out the list of ports.
ports: [53]
include_authorities: true
include_additionals: true
send_request: true
send_response: true
- type: http
# Configure the ports where to listen for HTTP traffic. You can disable
# the HTTP protocol by commenting out the list of ports.
ports: [80, 8080, 8000, 5000, 8002]
- type: tls
# Configure the ports where to listen for TLS traffic. You can disable
# the TLS protocol by commenting out the list of ports.
ports:
- 443 # HTTPS
- 993 # IMAPS
- 995 # POP3S
- 5223 # XMPP over SSL
- 8443
- 8883 # Secure MQTT
- 9243 # Elasticsearch
#=== Packetbeat logging ===
# Configure logging for Packetbeat if you plan on using the GeoIP ingest processor
# Initially use `info` for the logging.level, set logging.level to `debug` if you see
# an `Failed to publish events: temporary bulk send failure` error message in the logs
#logging.level: info
#logging.to_files: true
#logging.files:
# path: /var/log/elastic
# name: packetbeat
# keepfiles: 7
# permissions: 0644
# Configuration applicable for all beats on a specific device
#=== General ===
name: C7PiHole
tags: ["myTag", "C7PiHole"]
fields:
env: myEnv
version: 12-18-2019
#=== Top Level Processor ===
processors:
- add_host_metadata:
# netinfo.enabled should be set to `false` until GitHub issue
# https://github.com/elastic/elasticsearch/issues/46193 is resolved
netinfo.enabled: false
Geo: # These Geo configurations are optional
location: 40.7128, -74.0060
continent_name: North America
country_iso_code: US
region_name: New York
region_iso_code: US-NY
city_name: New York City
name: myHomeLocation
- add_locale: ~
# - add_cloud_metadata: ~
- add_fields:
#when.network.source.ip: 10.101.101.0/24
when.network.source.ip: private
fields:
source.geo.location:
lat: 40.7128
lon: -74.0060
source.geo.continent_name: North America
source.geo.country_iso_code: US
source.geo.region_name: New York
source.geo.region_iso_code: US-NY
source.geo.city_name: New York City
source.geo.name: myHomeLocation
target: ''
- add_fields:
#when.network.destination.ip: 10.101.101.0/24
when.network.destination.ip: private
fields:
destination.geo.location:
lat: 40.7128
lon: -74.0060
destination.geo.continent_name: North America
destination.geo.country_iso_code: US
destination.geo.region_name: New York
destination.geo.region_iso_code: US-NY
destination.geo.city_name: New York City
destination.geo.name: myHomeLocation
target: ''
#=== Elastic Cloud ===
# These settings simplify using beats with the Elastic Cloud (https://cloud.elastic.co/).
cloud.id: "My_Elastic_Cloud_Deployment:abcdefghijklmnopqrstuvwxyz1234567890ABCDEFGHIJKLMNOPQRSTUVWXYZ"
cloud.auth: "home_beats:0987654321abcDEF"
# The geoip-info pipeline is used to enrich GeoIP information in Elasticsearch
# You must configure the pipeline in Elasticsearch before enabling the pipeline in Beats.
# The `output.elasticsearch.pipeline: geoip-info` setting should be commented out until
# until GitHub issue https://github.com/elastic/elasticsearch/issues/46193 is resolved
#output.elasticsearch.pipeline: geoip-info
# The `max_retries` setting is the number of times to retry publishing an event after
# a publishing failure. After the specified number of retries, the events are typically dropped.
output.elasticsearch.max_retries: 5
# When deploying beats to multiple systems or locations, uncomment the following
# setup.template.* and setup.ilm.* configurations after running the beats setup command
# Otherwise, uncomment the following configurations for beats that will only publish data
#setup.template.enabled: false
#setup.ilm.check_exists: false
#setup.ilm.overwrite: false
#=== Xpack Monitoring ===
# When monitoring is enabled, the beat will ship monitoring data to the cluster
monitoring.enabled: true
#=== Queue ===
# See the 'Configure the internal queue' documentation for each Beat before
# configuring the queue. Note that only one queue type can be configured.
# You need to uncomment the specific queue type you decide to use.
# The `queue.mem` settings will cache events to memory in case access to the
# Elasticsearch cluster, via the internet, is unavailable (internet outage)
#queue.mem:
# events: 4096
# flush.min_events: 512
# flush.timeout: 1s
# The `queue.spool` settings will cache events to disk in case the system is offline
# NOTE: The file spool queue is a beta functionality in 7.4
queue.spool:
file:
size: 512MiB
page_size: 16KiB
prealloc: ~
write:
buffer_size: 10MiB
flush.timeout: 1s
flush.min.events: 1024
Once the file is updated, don’t forget to save the changes made to packetbeat.yml before proceeding.
Starting Packetbeat
Now that the configuration is updated, it's time to set up Packetbeat with our Elasticsearch Service deployment. Running the setup command only needs to be performed once for each beat (for instance, only once for Packetbeat 7.4.0). In our console, we will navigate to the /usr/share/packetbeat directory, then issue the sudo bin/packetbeat -c /etc/packetbeat/packetbeat.yml -path.home /usr/share/packetbeat setup command. Then we will confirm we receive “Index setup finished” and “Loaded dashboards” messages (shown below).
[myuser@centos7pi ~]$ cd /usr/share/packetbeat [myuser@centos7pi packetbeat]$ sudo bin/packetbeat -c /etc/packetbeat/packetbeat.yml -path.home /usr/share/packetbeat setup Index setup finished. Loading dashboards (Kibana must be running and reachable) Loaded dashboards [myuser@centos7pi packetbeat]$
Now that we’ve set up the index and loaded the dashboards, it is time to enable the Packetbeat service, then start the Packetbeat service.
[myuser@centos7pi packetbeat]$ sudo systemctl enable packetbeat
Created symlink from /etc/systemd/system/multi-user.target.wants/packetbeat.service to /usr/lib/systemd/system/packetbeat.service.
[myuser@centos7pi packetbeat]$ sudo systemctl start packetbeat
[myuser@centos7pi packetbeat]$ sudo systemctl status packetbeat
● packetbeat.service - Packetbeat analyzes network traffic and sends the data to Elasticsearch.
Loaded: loaded (/usr/lib/systemd/system/packetbeat.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2019-12-11 23:01:16 CST; 5s ago
Docs: https://www.elastic.co/products/beats/packetbeat
Main PID: 12990 (packetbeat)
CGroup: /system.slice/packetbeat.service
└─12990 /usr/share/packetbeat/bin/packetbeat -e -c /etc/packetbeat/packetbeat.yml -path.ho...
Dec 11 23:01:20 centos7pi.my.domain packetbeat[12990]: 2019-12-11T23:01:20.733-0600 I...ed
Hint: Some lines were ellipsized, use -l to show in full.
[myuser@centos7pi packetbeat]$ cd ~
[myuser@centos7pi ~]$
If you are using the GeoIP ingest pipeline and do not see data coming from Packetbeat for this device in your Elastic cluster, uncomment the logging section and restart Packetbeat.
Wrapping up with Beats on CentOS
Hurray! Now we have finished installing and configuring Auditbeat, Filebeat, and Packetbeat on this CentOS system. If you run into any issues, make sure to enable logging for the respective Beat (uncomment the logging options in the configuration file) then look through the log files in the /var/log/elastic directory.
Next steps: Data collection with Beats
In the next blog, we will install and configure Auditbeat, Filebeat, and Packetbeat on MacOS. While the next few blogs will focus on data collection from different systems, we will also cover some other use cases, including visualizing GeoIP data in Elastic Maps.
Follow along with this Elastic SIEM for home and small business blog series as we develop a powerful, yet simple, security solution at home (or for your small business). Remember that once we install and configure Beats on our systems, we can go to the SIEM app to see what data is available.
- Part 1: Getting started
- Part 2: Securing cluster access
- Part 3: GeoIP data and Beats config
- Part 4: Beats on Windows
- Part 5: Beats on CentOS
- Part 6: Beats on macOS
- Part 7: SIEM overview
A few last things...
If you run into any issues, the first place we’d recommend turning is to our documentation. It can help with many common issues. If you still have outstanding questions, check out our Elastic forums for additional help. Or, if you want to talk to the Elastic Support team directly, you have direct access to a team of experts if you’ve deployed on Elasticsearch Service. If you are self-hosting, you can start an Elastic subscription today and have direct access to a team of experts. Be safe out there!