Auditing Security Events
editAuditing Security Events
editAudit logs are disabled by default. To enable audit logging, set shield.audit.enabled
in
elasticsearch.yml
:
shield.audit.enabled: true
You can enable auditing to keep track of security-related events such as authentication failures and refused connections. Logging these events enables you to monitor your cluster for suspicious activity and provides evidence in the event of an attack.
Shield provides two ways to persist audit logs:
-
A dedicated
access.log
file on the host’s file system. - An Elasticsearch index. The audit index can reside on the same cluster, or a separate cluster.
By default, audit events are written to the access.log
. To facilitate browsing
and analyzing the events, you can also enable indexing by setting shield.audit.outputs
in elasticsearch.yml
:
shield.audit.outputs: [index, logfile]
The index
output type should be used in conjunction with the logfile
output type. Because it is
possible for the index
output type to lose messages if the target index is unavailable, the
logfile
output type should be used as an official record of events.
Audit events are batched for indexing so there is a lag before events appear in the index. You can
control how frequently batches of events are pushed to the index by setting shield.audit.index.flush_interval
in elasticsearch.yml
.
Log Entry Types
editEach audit event is represented by a single log entry. The type specified in a log entry represents the type of event that occurred:
-
anonymous_access_denied
is logged when the request is denied due to missing authentication token. -
authentication_failed
is logged when the authentication token cannot be matched to a known user. -
authentication_failed [<realm>]
is logged for every realm that fails to present a valid authentication token. The value of <realm> is the realm type. -
access_denied
is logged when an authenticated user attempts an action the user does not have the privilege to perform. -
access_granted
is logged when an authenticated user attempts an action the user has the correct privilege to perform. In TRACE level all system (internal) actions are logged as well (in all other level they’re not logged to avoid cluttering of the logs. -
tampered_request
is logged when the request was detected to be tampered (typically relates tosearch/scroll
requests when the scroll id is believed to be tampered) -
connection_granted
is logged when an incoming tcp connection has passed the ip filtering for a specific profile -
connection_denied
is logged when an incoming tcp connection did not pass the ip filtering for a specific profile
You can set the log level in logging.yml
to control what types of events are logged.
The following table lists the log entry types that are logged for each of the
possible log levels.
Table 7. Log Entry Types and Levels
Log Level | Entry Type |
---|---|
|
|
|
|
|
|
|
(doesn’t output additional entry types beyond |
|
|
For more information about configuring the audit logs, see Audit Log Settings.
Log Entry Format
editEvery log entry contains the following information:
- A timestamp that indicates when the event occurred.
- The component/layer the event is associated with
- The type of event that occurred.
- Information about the event (varies depending to the type of event)
The format of a log entry is:
[<timestamp>] [<local_node_info>] [<layer>] [<entry_type>] <attribute_list>
Where:
-
<timestamp>
- The time the event occurred. You can configure the timestamp format inlogging.yml
. -
<local_node_info>
- Information about the local node that this log entry is generated. You can control what node information is included by configuring the local node info settings. -
<layer>
- The layer from which this event originated:rest
,transport
orip_filter
. -
<entry_type>
- The type of event that occurred:anonymous_access_denied
,authentication_failed
,access_denied
,access_granted
,connection_granted
,connection_denied
. -
<attribute_list>
- A comma-separated list of key-value pairs that contain data pertaining to the event. Formatted asattr1=[val1], attr2=[val2],...
). See Log Entry Attributes for the attributes that can be included for each type of event.
Table 8. Local Node Info Settings
Name | Default | Description |
---|---|---|
|
true |
Include or exclude the node’s name from the local node info. |
|
false |
Include or exclude the node’s IP address from the local node info. |
|
false |
Include or exclude the node’s host name from the local node info. |
Log Entry Attributes
editThe following tables show the attributes that can be associated with each type of event. The log level determines which attributes are included in a log entry.
Table 9. [rest] [anonymous_access_denied]
attributes
Attribute | Minimum Log Level | Description |
---|---|---|
|
WARN |
The IP address from which the request originated |
|
WARN |
The REST endpoint URI |
|
DEBUG |
The body of the request |
Table 10. [rest] [authentication_failed]
attributes
Attribute | Minimum Log Level | Description |
---|---|---|
|
ERROR |
The IP address from which the request originated |
|
ERROR |
The principal (username) that failed authentication |
|
ERROR |
The REST endpoint URI |
|
DEBUG |
The body of the request |
|
TRACE |
The realm that failed to authenticate the user NOTE: A separate log entry is generated for each consulted realm. |
Table 11. [transport] [anonymous_access_denied]
attributes
Attribute | Minimum Log Level | Description |
---|---|---|
|
WARN |
Where the request originated: |
|
WARN |
The IP address from which the request originated |
|
WARN |
The name of the action that was executed |
|
DEBUG |
The type of request that was executed |
|
WARN |
A comma-separated list of indices the request pertained to (when applicable) |
Table 12. [transport] [authentication_failed]
attributes
Attribute | Minimum Log Level | Description |
---|---|---|
|
ERROR |
Where the request originated: |
|
ERROR |
The IP address from which the request originated |
|
INFO |
The principal (username) that failed authentication |
|
ERROR |
The name of the action that was executed |
|
DEBUG |
The type of request that was executed |
|
ERROR |
A comma-separated list of indices the request pertained to (when applicable) |
|
TRACE |
The realm that failed to authenticate the user NOTE: A separate log entry is generated for each consulted realm |
Table 13. [transport] [access_granted]
attributes
Attribute | Minimum Log Level | Description |
---|---|---|
|
INFO |
Where the request originated: |
|
INFO |
The IP address from which the request originated |
|
INFO |
The principal (username) that failed authentication |
|
INFO |
The name of the action that was executed |
|
DEBUG |
The type of request that was executed |
|
INFO |
A comma-separated list of indices the request pertained to (when applicable) |
Table 14. [transport] [access_denied]
attributes
Attribute | Minimum Log Level | Description |
---|---|---|
|
ERROR |
The type of the origin the request originated from. Can be either |
|
ERROR |
The IP address from which the request originated |
|
ERROR |
The principal (username) that failed authentication |
|
ERROR |
The name of the action that was executed |
|
DEBUG |
The type of request that was executed |
|
ERROR |
A comma-separated list of indices the request pertained to (when applicable) |
Table 15. [transport] [tampered_request]
attributes
Attribute | Minimum Log Level | Description |
---|---|---|
|
ERROR |
The type of the origin the request originated from. Can be either |
|
ERROR |
The IP address from which the request originated |
|
ERROR |
TThe principal (username) that failed authentication |
|
ERROR |
The name of the action that was executed |
|
DEBUG |
The type of request that was executed |
|
ERROR |
A comma-separated list of indices the request pertained to (when applicable) |
Table 16. [ip_filter] [connection_granted]
attributes
Attribute | Minimum Log Level | Description |
---|---|---|
|
TRACE |
The IP address from which the request originated |
|
TRACE |
The principal (username) that failed authentication |
|
TRACE |
The IP filtering rule that granted the request |
Table 17. [ip_filter] [connection_denied]
attributes
Attribute | Minimum Log Level | Description |
---|---|---|
|
ERROR |
The IP address from which the request originated |
|
ERROR |
The principal (username) that failed authentication |
|
ERROR |
The IP filtering rule that denied the request |
Audit Log Settings
editYou configure audit log options in the logging.yml
file located in CONFIG_DIR/shield
. The following snippet shows the default log configuration:
logger: shield.audit.logfile: INFO, access_log additivity: shield.audit.logfile: false appender: access_log: type: dailyRollingFile file: ${path.logs}/${cluster.name}-access.log datePattern: "'.'yyyy-MM-dd" layout: type: pattern conversionPattern: "[%d{ISO8601}] %m%n"
By default, audit information is appended to the access.log
file located in the
standard Elasticsearch logs
directory (typically located at $ES_HOME/logs
).
Storing Audit Logs in an Elasticsearch Index
editIn addition to logging to a file, you can store audit logs in Elasticsearch indices.
You can configure the following settings in elasticsearch.yml
to control how audit log entries
are indexed.
Table 18. Audit Log Index Settings
Attribute | Default Setting | Description |
---|---|---|
|
|
Must be set to |
|
|
Controls how many audit events are batched into a single write |
|
|
Controls how often to flush buffered events into the index |
|
|
Controls how often to roll over to a new index:
|
|
|
The audit events to be indexed. Valid values: |
|
|
The audit events to exclude from indexing. By default, |
You can also configure standard index settings for the audit log indices in shield.audit.index.settings
.
For example, the following configuration sets the number of shards and replicas to 1 for
the audit indices:
shield.audit.index.settings: index: number_of_shards: 1 number_of_replicas: 1
Forwarding Audit Logs to a Remote Cluster
editTo index audit events to a remote Elasticsearch cluster, you configure the following shield.audit.index.client
settings.
Table 19. Remote Audit Log Indexing Settings
Attribute | Default | Description |
---|---|---|
|
None |
Comma separated list of host:port pairs. These hosts should be nodes in the cluster to which you want to index. |
|
None |
The name of the remote cluster. |
|
None |
The username:password pair used to authenticate with the remote cluster. |
You can pass additional settings to the remote client using the shield.audit.index.client
namespace.
For example, to allow the remote client to discover all of the nodes in the remote cluster you could set
client.transport.sniff
:
shield.audit.index.client.transport.sniff: true