Elastic SIEM for home and small business: Beats on Mac
Note: The “SIEM for home and small business” blog series contains configurations relevant to the beta release of Elastic SIEM using Elastic Stack 7.4. We recommend using Elastic Stack 7.6 and newer, as Elastic SIEM was made generally available in 7.6. Please also note the Elastic SIEM solution mentioned in this post is now referred to as Elastic Security.
Hey, there. This is part six of the Elastic SIEM for home and small business blog series. If you haven’t read the first, second, and third blogs, you may want to before going any further. In the Getting started blog, we created our Elasticsearch Service deployment and started collecting data from one of our computers using Winlogbeat. In the Securing cluster access blog, we secured access to our cluster by restricting privileges for users and Beats. In the GeoIP data and Beats config blog, we created an ingest pipeline for GeoIP data and reviewed our Beats configurations.
Identifying our data collection needs on macOS
In the first blog, we determined that we need to use Auditbeat, Filebeat, Packetbeat, and Winlogbeat to collect log files, activities of users and processes, and network data. We do not need to install all of those Beats applications on our macOS devices, only the Beats we need to collect data relevant to us. In this blog, we will install and configure Auditbeat and Packetbeat on macOS.
For data collection from our macOS systems, we will focus on activities of users and processes, as well as network data. We will use Auditbeat to collect file integrity and system information, and Packetbeat to collect system-specific network data. In this blog, we will install Beats using Homebrew, so we will review the brew section of the Beats installation guide. In order to get started, we need administrative access to our macOS system. Then we need to add the Elastic Homebrew Tap and a copy of our Beats common configuration settings.
Using the GeoIP ingest pipeline
Before we start installing and configuring Beats on our macOS system, I want to cover the GeoIP ingest processor issue in more detail. There is a workaround that will allow us to enrich data with GeoIP information, but it depends on many variables. Essentially, you will have to enable logging for each respective Beat, then look at the logs to confirm data is being sent to your Elasticsearch Service deployment.
In the GeoIP data and Beats config blog, we created an ingest pipeline for GeoIP data, then we commented out the output.elasticsearch.pipeline: geoip-info setting. The reason I disabled use of the GeoIP ingest processor was to reduce (or prevent) frustration. Quite simply, you can read through (and follow) Elasticsearch GitHub issue 46193 for more detail around why I have currently chosen to disable these settings.
At a high level, the GeoIP ingest processor does not allow an array of IP addresses. This is a problem for systems running both IPv4 and IPv6, as well as systems with multiple network adapters (both physical and virtual). Any time Beats passes an array of IPs to Elasticsearch while using the GeoIP ingest processor, a Failed to publish events: temporary bulk send failure error message will be generated.
When netinfo.enabled: true is set on devices with multiple interfaces or addresses, Beats will include IP addresses and MAC addresses as fields host.ip and host.mac. This means the respective Beat will attempt to send an array of values for host.ip that causes the error.
While the example Beats common configurations have netinfo.enabled: false (instead of true) and commented out the output.elasticsearch.pipeline: geoip-info setting, the way to tell Beats to use the GeoIP ingest pipeline is to uncomment the output.elasticsearch.pipeline: geoip-info setting in your Beats configuration file.
If you decide to use the GeoIP ingest pipeline, you should also configure logging for each Beat. The reason we need to configure logging is that the Beat will run even though data is not being sent to Elasticsearch. For this blog, we will store our logs in the /var/log/elastic directory, but you will need to look for an Failed to publish events: temporary bulk send failure error message that will look similar to this:
ERROR pipeline/output.go:121 Failed to publish events: temporary bulk send failure
Again, this has to do with passing an array of IPs to Elasticsearch using the GeoIP ingest processor. With that said, feel free to follow Elasticsearch GitHub issue 46193 for updates.
Determining our installation method
In this blog, we are going to assume that the macOS system will be a shared device. On this shared device, we want to prevent non-admin (non-root) users from having access (or the ability to modify) our configuration. We also want to prevent non-root users from stopping any of the Beats services. In the steps outlined below, we will set the ownership to all of the Beats files to root, then we will run Beats as root. While this takes a few more steps, the intention is to prevent non-root users to access or modify Beats configurations or services on our macOS system.
If you would prefer to run Beats as a non-root user on your Mac, feel free to exclude using sudo in with the brew commands (for example, use brew services start instead of sudo brew services start). You will also need to skip the sudo chown commands in the examples below. Lastly, you will also need to configure a separate logging directory as the examples below are used by root.
Configuring the Elastic Homebrew Tap
The Beats installation guide provides instructions on how to configure the Elastic Homebrew Tap on our Mac. The guide tells us we need to add the Elastic Homebrew Tap by issuing brew tap elastic/tap in our Mac console.
mymac:~ rob$ brew tap elastic/tap ==> Tapping elastic/tap Cloning into '/usr/local/Homebrew/Library/Taps/elastic/homebrew-tap'... remote: Enumerating objects: 24, done. remote: Counting objects: 100% (24/24), done. remote: Compressing objects: 100% (24/24), done. remote: Total 24 (delta 10), reused 6 (delta 0), pack-reused 0 Unpacking objects: 100% (24/24), done. Tapped 18 formulae (68 files, 117.0KB). mymac:~ rob$
We can continue now that the Elastic Homebrew Tap is ready to use.
Auditbeat on macOS
Installing Auditbeat
Since we have our Elastic Homebrew Tap added, we will download and install Auditbeat by issuing the brew install elastic/tap/auditbeat-full command (or the brew install elastic/tap/auditbeat-oss command if you want to use the OSS distribution). Once we have our configuration file ready, we will also configure Auditbeat to start automatically during boot. After installing Auditbeat, do not start the service until we finish initial configuration.
mymac:~ rob$ brew install elastic/tap/auditbeat-full ==> Installing auditbeat-full from elastic/tap ==> Downloading https://artifacts.elastic.co/downloads/beats/auditbeat/auditbeat-7.4.0-darwin-x86_64.tar.gz ######################################################################## 100.0% ==> Caveats To have launchd start elastic/tap/auditbeat-full now and restart at login: brew services start elastic/tap/auditbeat-full Or, if you don't want/need a background service you can just run: auditbeat ==> Summary /usr/local/Cellar/auditbeat-full/7.4.0: 20 files, 65.1MB, built in 7 seconds mymac:~ rob$
Remember, do not start the Auditbeat service until we finish initial configuration.
Configuring Auditbeat
Now that Auditbeat is installed, we need to update our auditbeat.yml configuration file. Since we have already determined our common Beats configurations, we only need to finalize our Auditbeat-specific options, which will be stored in the first section of the file. Make sure to read through the auditbeat.reference.yml page or the auditbeat.reference.yml file in the /usr/local/etc/auditbeat directory. For clarity, the Beats common configuration sections will be stored below the Auditbeat specific options.
If you plan on using the GeoIP ingest processor, you should configure logging for Auditbeat. Since we will store our log files in the /var/log/elastic directory, our logging configuration will look like this:
#=== Auditbeat logging === # Initially use `info` for the logging.level, set logging.level to `debug` if you see # an `Failed to publish events: temporary bulk send failure` error message in the logs logging.level: info logging.to_files: true logging.files: path: /var/log/elastic name: auditbeat keepfiles: 7 permissions: 0644
We will add the Auditbeat logging section to our configuration but leave it commented out for now. With that said, after issuing cat /usr/local/etc/auditbeat/auditbeat.yml, here is what I have set in auditbeat.yml:
#=== Auditbeat specific options ===
#=== Modules configuration ===
auditbeat.modules:
- module: file_integrity
paths:
- /bin
- /usr/bin
- /usr/local
- /sbin
- /usr/sbin
- /Applications
- /Library
- /System
- module: system
datasets:
- package
- host
period: 30m
state.period: 12h
- module: system
datasets:
- process
processors:
- drop_event.when:
or:
- contains.event.action: "existing_process"
- contains.event.action: "process_error"
- add_process_metadata:
match_pids: [process.ppid]
target: process.parent
period: 5s
#=== Auditbeat logging ===
# Configure logging for Auditbeat if you plan on using the GeoIP ingest processor
# Initially use `info` for the logging.level, set logging.level to `debug` if you see
# an `Failed to publish events: temporary bulk send failure` error message in the logs
#logging.level: info
#logging.to_files: true
#logging.files:
# path: /var/log/elastic
# name: auditbeat
# keepfiles: 7
# permissions: 0644
#=== Beats Common Configs Here ===
You may have noticed I have added two drop_event.when conditions as an example using the drop_event processor. You may have also noticed I specified state.period and period in the system module section. The period parameter controls the poll frequency while state.period controls the frequency of state updates. I also included the add process metadata processor to enrich events collected by the system module. For reference, this example configuration can be found in the examples GitHub repo under Security Analytics/SIEM-at-Home.
Next, we will add the settings from our Beats common configuration settings to our auditbeat.yml configuration file. We will use the settings from the General, Top Level Processor, Elastic Cloud, Xpack Monitoring, and Queue sections and add them to the end of our configuration file. Our completed auditbeat.yml file now looks like this:
#=== Auditbeat specific options ===
#=== Modules configuration ===
auditbeat.modules:
- module: file_integrity
paths:
- /bin
- /usr/bin
- /usr/local
- /sbin
- /usr/sbin
- /Applications
- /Library
- /System
- module: system
datasets:
- package
- host
period: 30m
state.period: 12h
- module: system
datasets:
- process
processors:
- drop_event.when:
or:
- contains.event.action: "existing_process"
- contains.event.action: "process_error"
- add_process_metadata:
match_pids: [process.ppid]
target: process.parent
period: 5s
#=== Auditbeat logging ===
# Configure logging for Auditbeat if you plan on using the GeoIP ingest processor
# Initially use `info` for the logging.level, set logging.level to `debug` if you see
# an `Failed to publish events: temporary bulk send failure` error message in the logs
#logging.level: info
#logging.to_files: true
#logging.files:
# path: /var/log/elastic
# name: auditbeat
# keepfiles: 7
# permissions: 0644
# Configuration applicable for all beats on a specific device
#=== General ===
name: myMac
tags: ["myTag", "myMac"]
fields:
env: myEnv
version: 12-18-2019
#=== Top Level Processor ===
processors:
- add_host_metadata:
# netinfo.enabled should be set to `false` until GitHub issue
# https://github.com/elastic/elasticsearch/issues/46193 is resolved
netinfo.enabled: false
Geo: # These Geo configurations are optional
location: 40.7128, -74.0060
continent_name: North America
country_iso_code: US
region_name: New York
region_iso_code: US-NY
city_name: New York City
name: myHomeLocation
- add_locale: ~
# - add_cloud_metadata: ~
- add_fields:
#when.network.source.ip: 10.101.101.0/24
when.network.source.ip: private
fields:
source.geo.location:
lat: 40.7128
lon: -74.0060
source.geo.continent_name: North America
source.geo.country_iso_code: US
source.geo.region_name: New York
source.geo.region_iso_code: US-NY
source.geo.city_name: New York City
source.geo.name: myHomeLocation
target: ''
- add_fields:
#when.network.destination.ip: 10.101.101.0/24
when.network.destination.ip: private
fields:
destination.geo.location:
lat: 40.7128
lon: -74.0060
destination.geo.continent_name: North America
destination.geo.country_iso_code: US
destination.geo.region_name: New York
destination.geo.region_iso_code: US-NY
destination.geo.city_name: New York City
destination.geo.name: myHomeLocation
target: ''
#=== Elastic Cloud ===
# These settings simplify using beats with the Elastic Cloud (https://cloud.elastic.co/).
cloud.id: "My_Elastic_Cloud_Deployment:abcdefghijklmnopqrstuvwxyz1234567890ABCDEFGHIJKLMNOPQRSTUVWXYZ"
cloud.auth: "home_beats:0987654321abcDEF"
# The geoip-info pipeline is used to enrich GeoIP information in Elasticsearch
# You must configure the pipeline in Elasticsearch before enabling the pipeline in Beats.
# The `output.elasticsearch.pipeline: geoip-info` setting should be commented out until
# until GitHub issue https://github.com/elastic/elasticsearch/issues/46193 is resolved
#output.elasticsearch.pipeline: geoip-info
# The `max_retries` setting is the number of times to retry publishing an event after
# a publishing failure. After the specified number of retries, the events are typically dropped.
output.elasticsearch.max_retries: 5
# When deploying beats to multiple systems or locations, uncomment the following
# setup.template.* and setup.ilm.* configurations after running the beats setup command
# Otherwise, uncomment the following configurations for beats that will only publish data
#setup.template.enabled: false
#setup.ilm.check_exists: false
#setup.ilm.overwrite: false
#=== Xpack Monitoring ===
# When monitoring is enabled, the beat will ship monitoring data to the cluster
monitoring.enabled: true
#=== Queue ===
# See the 'Configure the internal queue' documentation for each Beat before
# configuring the queue. Note that only one queue type can be configured.
# You need to uncomment the specific queue type you decide to use.
# The `queue.mem` settings will cache events to memory in case access to the
# Elasticsearch cluster, via the internet, is unavailable (internet outage)
#queue.mem:
# events: 4096
# flush.min_events: 512
# flush.timeout: 1s
# The `queue.spool` settings will cache events to disk in case the system is offline
# NOTE: The file spool queue is a beta functionality in 7.4
queue.spool:
file:
size: 512MiB
page_size: 16KiB
prealloc: ~
write:
buffer_size: 10MiB
flush.timeout: 1s
flush.min.events: 1024
Once the file is updated, don’t forget to save the changes made to auditbeat.yml before proceeding.
Root ownership of Auditbeat files
Before we continue, we need to set all of the files to root ownership. If you are running Beats as a non-root user, you can skip this section. We need to change the owner of all files in both the /usr/local/Cellar/auditbeat-full and the /usr/local/etc/auditbeat directories. For each directory, we will issue the sudo chown -Rv root:wheel <directory> command. Here is the output from my MacBook when updating ownership of the /usr/local/Cellar/auditbeat-full directory:
mymac:~ rob$ sudo chown -Rv root:wheel /usr/local/Cellar/auditbeat-full Password: /usr/local/Cellar/auditbeat-full/7.4.0/homebrew.mxcl.auditbeat-full.plist /usr/local/Cellar/auditbeat-full/7.4.0/INSTALL_RECEIPT.json /usr/local/Cellar/auditbeat-full/7.4.0/bin/auditbeat /usr/local/Cellar/auditbeat-full/7.4.0/bin /usr/local/Cellar/auditbeat-full/7.4.0/.brew/auditbeat-full.rb /usr/local/Cellar/auditbeat-full/7.4.0/.brew /usr/local/Cellar/auditbeat-full/7.4.0/libexec/kibana/7/dashboard/auditbeat-system-overview-dashboard.json /usr/local/Cellar/auditbeat-full/7.4.0/libexec/kibana/7/dashboard/auditbeat-kernel-overview.json /usr/local/Cellar/auditbeat-full/7.4.0/libexec/kibana/7/dashboard/auditbeat-file-integrity.json /usr/local/Cellar/auditbeat-full/7.4.0/libexec/kibana/7/dashboard/auditbeat-kernel-sockets.json /usr/local/Cellar/auditbeat-full/7.4.0/libexec/kibana/7/dashboard/auditbeat-system-process-dashboard.json /usr/local/Cellar/auditbeat-full/7.4.0/libexec/kibana/7/dashboard/auditbeat-system-package-dashboard.json /usr/local/Cellar/auditbeat-full/7.4.0/libexec/kibana/7/dashboard/auditbeat-system-user-dashboard.json /usr/local/Cellar/auditbeat-full/7.4.0/libexec/kibana/7/dashboard/auditbeat-system-socket-dashboard.json /usr/local/Cellar/auditbeat-full/7.4.0/libexec/kibana/7/dashboard/auditbeat-system-host-dashboard.json /usr/local/Cellar/auditbeat-full/7.4.0/libexec/kibana/7/dashboard/auditbeat-kernel-executions.json /usr/local/Cellar/auditbeat-full/7.4.0/libexec/kibana/7/dashboard/auditbeat-system-login-dashboard.json /usr/local/Cellar/auditbeat-full/7.4.0/libexec/kibana/7/dashboard /usr/local/Cellar/auditbeat-full/7.4.0/libexec/kibana/7 /usr/local/Cellar/auditbeat-full/7.4.0/libexec/kibana /usr/local/Cellar/auditbeat-full/7.4.0/libexec/bin/auditbeat /usr/local/Cellar/auditbeat-full/7.4.0/libexec/bin /usr/local/Cellar/auditbeat-full/7.4.0/libexec/fields.yml /usr/local/Cellar/auditbeat-full/7.4.0/libexec /usr/local/Cellar/auditbeat-full/7.4.0/README.md /usr/local/Cellar/auditbeat-full/7.4.0/NOTICE.txt /usr/local/Cellar/auditbeat-full/7.4.0/LICENSE.txt /usr/local/Cellar/auditbeat-full/7.4.0 /usr/local/Cellar/auditbeat-full/ mymac:~ rob$
Here is the output from my MacBook when updating ownership of the /usr/local/etc/auditbeat directory:
mymac:~ rob$ sudo chown -Rv root:wheel /usr/local/etc/auditbeat Password: /usr/local/etc/auditbeat/auditbeat.yml /usr/local/etc/auditbeat/auditbeat.yml.default /usr/local/etc/auditbeat/auditbeat.reference.yml /usr/local/etc/auditbeat/fields.yml /usr/local/etc/auditbeat/fields.yml.default /usr/local/etc/auditbeat/ mymac:~ rob$
Now that root owns the Auditbeat files, we can proceed.
Starting Auditbeat
Now that the configuration is updated, it’s time to set up Auditbeat with our Elasticsearch Service deployment. Running the setup command only needs to be performed once for each beat (for instance, only once for Auditbeat 7.4.0). In our console, we will issue the sudo auditbeat setup command. If you are running Auditbeat as a non-root user, you will need to exclude sudo from the commands below. Then we will confirm we receive “Index setup finished” and “Loaded dashboards” messages (shown below).
mymac:~ rob$ sudo auditbeat setup Index setup finished. Loading dashboards (Kibana must be running and reachable) Loaded dashboards mymac:~ rob$
Now that we’ve set up the index and loaded the dashboards, it’s time to enable and start the Auditbeat service.
mymac:~ rob$ sudo brew services start elastic/tap/auditbeat-full
We will verify Auditbeat is running by issuing the brew services list command:
mymac:~ rob$ brew services list Name Status User Plist auditbeat-full started root /Library/LaunchDaemons/homebrew.mxcl.auditbeat-full.plist mymac:~ rob$
If you are using the GeoIP ingest pipeline and do not see data coming from Auditbeat for this device in your Elastic cluster, you should uncomment the logging section and restart Auditbeat.
Packetbeat on macOS
While it is ideal to deploy Packetbeat to either existing application servers or dedicated servers (to get traffic from mirror ports or tap devices), some data can be collected on workstations. Since we’ve downloaded Packetbeat, it’s a good idea to read through the Packetbeat installation guide before we get started.
Installing Packetbeat
Since we have our Elastic Homebrew Tap added, we will download and install Packetbeat by issuing the brew install elastic/tap/packetbeat-full command (or the brew install elastic/tap/packetbeat-oss command if you want to use the OSS distribution). Once we have our configuration file ready, we‘ll also configure Packetbeat to start automatically during boot. After installing Packetbeat, do not start the service until we finish initial configuration.
mymac:~ rob$ brew install elastic/tap/packetbeat-full ==> Installing packetbeat-full from elastic/tap ==> Downloading https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-7.4.0-darwin-x86_64.tar.gz ######################################################################## 100.0% ==> Caveats To have launchd start packetbeat-full now and restart at login: brew services start packetbeat-full Or, if you don't want/need a background service you can just run: packetbeat ==> Summary /usr/local/Cellar/packetbeat-full/7.4.0: 22 files, 67.6MB, built in 11 seconds mymac:~ rob$
Remember, do not start the Packetbeat service until we finish initial configuration.
Configuring Packetbeat
Now that Packetbeat is installed, we need to update our packetbeat.yml configuration file. Since we have already determined our common Beats configurations, we only need to finalize our Packetbeat-specific options, which will be stored in the first section of the file. Make sure to read through the packetbeat.reference.yml page or the packetbeat.reference.yml file in the /usr/local/etc/packetbeat directory. For clarity, the Beats common configuration sections will be stored below the Packetbeat specific options.
When running Packetbeat on Mac, we are not able to capture traffic from all interfaces. Instead, we have to set a specific interface from which to capture. In order to determine the interface we need to capture, we will run the packetbeat devices command as mentioned in the device section of the set traffic capturing options guide.
mymac:~ rob$ packetbeat devices 0: en0 (No description available) (Not assigned ip address) 2: en1 (No description available) (Not assigned ip address) 3: en2 (No description available) (Not assigned ip address) 4: en3 (No description available) (Not assigned ip address) 5: en4 (No description available) (Not assigned ip address) 6: en7 (No description available) (10.101.10.45) 7: lo0 (No description available) (127.0.0.1 ::1 fe80::1) mymac:~ rob$
Based on the output from the packetbeat devices command, we will need to configure Packetbeat to capture data from device 6 since it has an IP address on our network — meaning we will set packetbeat.interfaces.device: 6 in our packetbeat.yml configuration file.
If you plan on using the GeoIP ingest processor, you should configure logging for Packetbeat. Since we will store our log files in the /var/log/elastic directory, our logging configuration will look like this:
#=== Packetbeat logging === # Initially use `info` for the logging.level, set logging.level to `debug` if you see # an `Failed to publish events: temporary bulk send failure` error message in the logs logging.level: info logging.to_files: true logging.files: path: /var/log/elastic name: packetbeat keepfiles: 7 permissions: 0644
We will add the Packetbeat logging section to our configuration, but leave it commented out for now. With that said, after issuing cat /usr/local/etc/packetbeat/packetbeat.yml, here is what I have set in packetbeat.yml:
#=== Packetbeat specific options ===
#=== Network device ===
# Issue the `packetbeat devices` command to determine the interfaces on your system
# Select the network interface to sniff the data. On Linux, you can use the
# "any" keyword to sniff on all connected interfaces.
packetbeat.interfaces.device: 6
#=== Flows ===
packetbeat.flows:
timeout: 30s
period: 10s
#=== Transaction protocols ===
# For more information on the transaction protocols, see
# https://www.elastic.co/guide/en/beats/packetbeat/7.4/configuration-protocols.html
packetbeat.protocols:
- type: icmp
# Enable ICMPv4 and ICMPv6 monitoring. Default: false
enabled: true
- type: dhcpv4
# Configure the DHCP for IPv4 ports.
ports: [67, 68]
send_request: true
send_response: true
- type: dns
# Configure the ports where to listen for DNS traffic. You can disable
# the DNS protocol by commenting out the list of ports.
ports: [53]
include_authorities: true
include_additionals: true
send_request: true
send_response: true
- type: http
# Configure the ports where to listen for HTTP traffic. You can disable
# the HTTP protocol by commenting out the list of ports.
ports: [80, 8080, 8000, 5000, 8002]
- type: tls
# Configure the ports where to listen for TLS traffic. You can disable
# the TLS protocol by commenting out the list of ports.
ports:
- 443 # HTTPS
- 993 # IMAPS
- 995 # POP3S
- 5223 # XMPP over SSL
- 8443
- 8883 # Secure MQTT
- 9243 # Elasticsearch
#=== Packetbeat logging ===
# Configure logging for Packetbeat if you plan on using the GeoIP ingest processor
# Initially use `info` for the logging.level, set logging.level to `debug` if you see
# an `Failed to publish events: temporary bulk send failure` error message in the logs
#logging.level: info
#logging.to_files: true
#logging.files:
# path: /var/log/elastic
# name: packetbeat
# keepfiles: 7
# permissions: 0644
#=== Beats Common Configs Here ===
You may have noticed the include_authorities: true and include_additionals: true entries in the DNS transaction protocol section, which is an example using DNS-specific configuration options. You may have also noticed the send_request: true and send_response: true entries in both the DHCP and DNS transaction protocol sections, which are examples using common protocol options. For reference, this example configuration can be found in the examples GitHub repo under Security Analytics/SIEM-at-Home.
Next, we will add the settings from our Beats common configuration settings to our packetbeat.yml configuration file. We will use the settings from the General, Top Level Processor, Elastic Cloud, Xpack Monitoring, and Queue sections and add them to the end of our configuration file. Our completed packetbeat.yml file now looks like this:
#=== Packetbeat specific options ===
#=== Network device ===
# Issue the `packetbeat devices` command to determine the interfaces on your system
# Select the network interface to sniff the data. On Linux, you can use the
# "any" keyword to sniff on all connected interfaces.
packetbeat.interfaces.device: 6
#=== Flows ===
packetbeat.flows:
timeout: 30s
period: 10s
#=== Transaction protocols ===
# For more information on the transaction protocols, see
# https://www.elastic.co/guide/en/beats/packetbeat/7.4/configuration-protocols.html
packetbeat.protocols:
- type: icmp
# Enable ICMPv4 and ICMPv6 monitoring. Default: false
enabled: true
- type: dhcpv4
# Configure the DHCP for IPv4 ports.
ports: [67, 68]
send_request: true
send_response: true
- type: dns
# Configure the ports where to listen for DNS traffic. You can disable
# the DNS protocol by commenting out the list of ports.
ports: [53]
include_authorities: true
include_additionals: true
send_request: true
send_response: true
- type: http
# Configure the ports where to listen for HTTP traffic. You can disable
# the HTTP protocol by commenting out the list of ports.
ports: [80, 8080, 8000, 5000, 8002]
- type: tls
# Configure the ports where to listen for TLS traffic. You can disable
# the TLS protocol by commenting out the list of ports.
ports:
- 443 # HTTPS
- 993 # IMAPS
- 995 # POP3S
- 5223 # XMPP over SSL
- 8443
- 8883 # Secure MQTT
- 9243 # Elasticsearch
#=== Packetbeat logging ===
# Configure logging for Packetbeat if you plan on using the GeoIP ingest processor
# Initially use `info` for the logging.level, set logging.level to `debug` if you see
# an `Failed to publish events: temporary bulk send failure` error message in the logs
#logging.level: info
#logging.to_files: true
#logging.files:
# path: /var/log/elastic
# name: packetbeat
# keepfiles: 7
# permissions: 0644
# Configuration applicable for all beats on a specific device
#=== General ===
name: myMac
tags: ["myTag", "myMac"]
fields:
env: myEnv
version: 12-18-2019
#=== Top Level Processor ===
processors:
- add_host_metadata:
# netinfo.enabled should be set to `false` until GitHub issue
# https://github.com/elastic/elasticsearch/issues/46193 is resolved
netinfo.enabled: false
Geo: # These Geo configurations are optional
location: 40.7128, -74.0060
continent_name: North America
country_iso_code: US
region_name: New York
region_iso_code: US-NY
city_name: New York City
name: myHomeLocation
- add_locale: ~
# - add_cloud_metadata: ~
- add_fields:
#when.network.source.ip: 10.101.101.0/24
when.network.source.ip: private
fields:
source.geo.location:
lat: 40.7128
lon: -74.0060
source.geo.continent_name: North America
source.geo.country_iso_code: US
source.geo.region_name: New York
source.geo.region_iso_code: US-NY
source.geo.city_name: New York City
source.geo.name: myHomeLocation
target: ''
- add_fields:
#when.network.destination.ip: 10.101.101.0/24
when.network.destination.ip: private
fields:
destination.geo.location:
lat: 40.7128
lon: -74.0060
destination.geo.continent_name: North America
destination.geo.country_iso_code: US
destination.geo.region_name: New York
destination.geo.region_iso_code: US-NY
destination.geo.city_name: New York City
destination.geo.name: myHomeLocation
target: ''
#=== Elastic Cloud ===
# These settings simplify using beats with the Elastic Cloud (https://cloud.elastic.co/).
cloud.id: "My_Elastic_Cloud_Deployment:abcdefghijklmnopqrstuvwxyz1234567890ABCDEFGHIJKLMNOPQRSTUVWXYZ"
cloud.auth: "home_beats:0987654321abcDEF"
# The geoip-info pipeline is used to enrich GeoIP information in Elasticsearch
# You must configure the pipeline in Elasticsearch before enabling the pipeline in Beats.
# The `output.elasticsearch.pipeline: geoip-info` setting should be commented out until
# until GitHub issue https://github.com/elastic/elasticsearch/issues/46193 is resolved
#output.elasticsearch.pipeline: geoip-info
# The `max_retries` setting is the number of times to retry publishing an event after
# a publishing failure. After the specified number of retries, the events are typically dropped.
output.elasticsearch.max_retries: 5
# When deploying beats to multiple systems or locations, uncomment the following
# setup.template.* and setup.ilm.* configurations after running the beats setup command
# Otherwise, uncomment the following configurations for beats that will only publish data
#setup.template.enabled: false
#setup.ilm.check_exists: false
#setup.ilm.overwrite: false
#=== Xpack Monitoring ===
# When monitoring is enabled, the beat will ship monitoring data to the cluster
monitoring.enabled: true
#=== Queue ===
# See the 'Configure the internal queue' documentation for each Beat before
# configuring the queue. Note that only one queue type can be configured.
# You need to uncomment the specific queue type you decide to use.
# The `queue.mem` settings will cache events to memory in case access to the
# Elasticsearch cluster, via the internet, is unavailable (internet outage)
#queue.mem:
# events: 4096
# flush.min_events: 512
# flush.timeout: 1s
# The `queue.spool` settings will cache events to disk in case the system is offline
# NOTE: The file spool queue is a beta functionality in 7.4
queue.spool:
file:
size: 512MiB
page_size: 16KiB
prealloc: ~
write:
buffer_size: 10MiB
flush.timeout: 1s
flush.min.events: 1024
Once the file is updated, don’t forget to save the changes made to packetbeat.yml before proceeding.
Root ownership of Packetbeat files
Before we continue, we need to set all of the files to root ownership. If you are running Beats as a non-root user, you can skip this section. We need to change the owner of all files in both the /usr/local/Cellar/packetbeat-full and the /usr/local/etc/packetbeat directories. For each directory, we will issue the sudo chown -Rv root:wheel <directory> command. Here is the output from my MacBook when updating ownership of the /usr/local/Cellar/packetbeat-full directory:
mymac:~ rob$ sudo chown -Rv root:wheel /usr/local/Cellar/packetbeat-full Password: /usr/local/Cellar/packetbeat-full/7.4.0/INSTALL_RECEIPT.json /usr/local/Cellar/packetbeat-full/7.4.0/bin/packetbeat /usr/local/Cellar/packetbeat-full/7.4.0/bin /usr/local/Cellar/packetbeat-full/7.4.0/.brew/packetbeat-full.rb /usr/local/Cellar/packetbeat-full/7.4.0/.brew /usr/local/Cellar/packetbeat-full/7.4.0/libexec/kibana/7/dashboard/Packetbeat-thrift.json /usr/local/Cellar/packetbeat-full/7.4.0/libexec/kibana/7/dashboard/Packetbeat-mongodb.json /usr/local/Cellar/packetbeat-full/7.4.0/libexec/kibana/7/dashboard/Packetbeat-mysql.json /usr/local/Cellar/packetbeat-full/7.4.0/libexec/kibana/7/dashboard/Packetbeat-flows.json /usr/local/Cellar/packetbeat-full/7.4.0/libexec/kibana/7/dashboard/Packetbeat-nfs.json /usr/local/Cellar/packetbeat-full/7.4.0/libexec/kibana/7/dashboard/Packetbeat-overview.json /usr/local/Cellar/packetbeat-full/7.4.0/libexec/kibana/7/dashboard/Packetbeat-cassandra.json /usr/local/Cellar/packetbeat-full/7.4.0/libexec/kibana/7/dashboard/Packetbeat-tls.json /usr/local/Cellar/packetbeat-full/7.4.0/libexec/kibana/7/dashboard/Packetbeat-http.json /usr/local/Cellar/packetbeat-full/7.4.0/libexec/kibana/7/dashboard/Packetbeat-dns-tunneling.json /usr/local/Cellar/packetbeat-full/7.4.0/libexec/kibana/7/dashboard/Packetbeat-pgsql.json /usr/local/Cellar/packetbeat-full/7.4.0/libexec/kibana/7/dashboard/Packetbeat-dhcpv4.json /usr/local/Cellar/packetbeat-full/7.4.0/libexec/kibana/7/dashboard/Packetbeat-dns-overview.json /usr/local/Cellar/packetbeat-full/7.4.0/libexec/kibana/7/dashboard /usr/local/Cellar/packetbeat-full/7.4.0/libexec/kibana/7 /usr/local/Cellar/packetbeat-full/7.4.0/libexec/kibana /usr/local/Cellar/packetbeat-full/7.4.0/libexec/bin/packetbeat /usr/local/Cellar/packetbeat-full/7.4.0/libexec/bin /usr/local/Cellar/packetbeat-full/7.4.0/libexec/fields.yml /usr/local/Cellar/packetbeat-full/7.4.0/libexec /usr/local/Cellar/packetbeat-full/7.4.0/homebrew.mxcl.packetbeat-full.plist /usr/local/Cellar/packetbeat-full/7.4.0/README.md /usr/local/Cellar/packetbeat-full/7.4.0/NOTICE.txt /usr/local/Cellar/packetbeat-full/7.4.0/LICENSE.txt /usr/local/Cellar/packetbeat-full/7.4.0 /usr/local/Cellar/packetbeat-full mymac:~ rob$
Here is the output from my MacBook when updating ownership of the /usr/local/etc/packetbeat directory:
mymac:~ rob$ sudo chown -Rv root:wheel /usr/local/etc/packetbeat Password: /usr/local/etc/packetbeat/packetbeat.yml /usr/local/etc/packetbeat/packetbeat.yml.default /usr/local/etc/packetbeat/packetbeat.reference.yml /usr/local/etc/packetbeat/fields.yml /usr/local/etc/packetbeat mymac:~ rob$
Now that root owns the Auditbeat files, we can proceed.
Starting Packetbeat
Now that the configuration is updated, it is time to set up Packetbeat with our Elasticsearch Service deployment. Running the setup command only needs to be performed once for each beat (for instance only once for Packetbeat 7.4.0). In our console, we will issue the sudo packetbeat setup command. If you are running Packetbeat as a non-root user, you will need to exclude sudo from the commands below. Then we will confirm we receive “Index setup finished” and “Loaded dashboards” messages (shown below).
mymac:~ rob$ sudo packetbeat setup Index setup finished. Loading dashboards (Kibana must be running and reachable) Loaded dashboards mymac:~ rob$
Now that we’ve set up the index and loaded the dashboards, it is time to enable and start the Packetbeat service.
mymac:~ rob$ sudo brew services start elastic/tap/packetbeat-full
We will verify Packetbeat is running by issuing the brew services list command:
mymac:~ rob$ brew services list Name Status User Plist auditbeat-full started root /Library/LaunchDaemons/homebrew.mxcl.auditbeat-full.plist packetbeat-full started root /Library/LaunchDaemons/homebrew.mxcl.packetbeat-full.plist mymac:~ rob$
If you are using the GeoIP ingest pipeline and do not see data coming from Packetbeat for this device in your Elastic cluster, you should uncomment the logging section and restart Packetbeat.
Wrapping up with Beats on macOS
Hurray! Now we have finished installing and configuring Auditbeat and Packetbeat on this macOS system.
Next steps: Elastic SIEM walkthrough
In the next blog, we will review the data collected by Beats as we walk through the Elastic SIEM app. For better visibility of your network, you will want to collect data from other devices on your network — especially network devices. Since your data collection needs are determined by your infrastructure, I recommend reading through the Elastic SIEM Get up and running guide to see what options are available for collecting data from the remaining devices in your network. After the SIEM app walkthrough, we will review our data in Elastic Maps and finish up the series by reviewing ways to maintain our Elasticsearch Service deployment.
Follow along with this Elastic SIEM for home and small business blog series as we develop a powerful, yet simple, security solution at home (or for your small business). Remember that once we install and configure Beats on our systems, we can go to the SIEM app to see what data is available.
- Part 1: Getting started
- Part 2: Securing cluster access
- Part 3: GeoIP data and Beats config
- Part 4: Beats on Windows
- Part 5: Beats on CentOS
- Part 6: Beats on macOS
- Part 7: SIEM overview
A few last things...
If you run into any issues, the first place we’d recommend turning is to our documentation. It can help with many common issues. If you still have outstanding questions, check out our Elastic forums for additional help. Or, if you want to talk to the Elastic Support team directly, you have direct access to a team of experts if you’ve deployed on Elasticsearch Service. If you are self-hosting, you can start an Elastic subscription today and have direct access to a team of experts. Be safe out there!