WARNING: Version 6.2 of the Elastic Stack has passed its EOL date.
This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.
Auditing Security Events
editAuditing Security Events
editYou can enable auditing to keep track of security-related events such as authentication failures and refused connections. Logging these events enables you to monitor your cluster for suspicious activity and provides evidence in the event of an attack.
Audit logs are disabled by default. To enable this functionality, you
must set xpack.security.audit.enabled
to true
in elasticsearch.yml
.
X-Pack security provides two ways to persist audit logs:
By default, only the logfile
output is used when enabling auditing.
To facilitate browsing and analyzing the events, you can also enable
indexing by setting xpack.security.audit.outputs
in elasticsearch.yml
:
xpack.security.audit.outputs: [ index, logfile ]
The index
output type should be used in conjunction with the logfile
output type Because it is possible for the index
output type to lose
messages if the target index is unavailable, the access.log
should be
used as the official record of events.
Audit events are batched for indexing so there is a lag before
events appear in the index. You can control how frequently batches of
events are pushed to the index by setting
xpack.security.audit.index.flush_interval
in elasticsearch.yml
.
Audit Event Types
editEach request may generate multiple audit events. The following is a list of the events that can be generated:
|
Logged when a request is denied due to a missing authentication token. |
||
|
Logged when a user successfully authenticates. |
||
|
Logged when the authentication token cannot be matched to a known user. |
||
|
Logged for every realm that fails to present a valid
authentication token. |
||
|
Logged when an authenticated user attempts to execute an action they do not have the necessary privilege to perform. |
||
|
Logged when an authenticated user attempts to execute
an action they have the necessary privilege to perform.
When the |
||
|
Logged when an authenticated user attempts to run as another user that they have the necessary privileges to do. |
||
|
Logged when an authenticated user attempts to run as another user action they do not have the necessary privilege to do so. |
||
|
Logged when X-Pack security detects that the request has
been tampered with. Typically relates to |
||
|
Logged when an incoming TCP connection passes the IP Filter for a specific profile. |
||
|
Logged when an incoming TCP connection does not pass the IP Filter for a specific profile. |
Audit Event Attributes
editThe following table shows the common attributes that can be associated with every event.
Table 8. Common Attributes
Attribute | Description |
---|---|
|
When the event occurred. |
|
The name of the node. |
|
The hostname of the node. |
|
The IP address of the node. |
|
The layer from which this event originated: |
|
The type of event that occurred: |
The following tables show the attributes that can be associated with each type of event. The log level determines which attributes are included in a log entry.
Table 9. REST anonymous_access_denied Attributes
Attribute | Description |
---|---|
|
The IP address from which the request originated. |
|
The REST endpoint URI. |
|
The body of the request, if enabled. |
Table 10. REST authentication_success Attributes
Attribute | Description |
---|---|
|
The authenticated user. |
|
The realm that authenticated the user. |
|
The REST endpoint URI. |
|
The REST URI query parameters. |
|
The body of the request, if enabled. |
Table 11. REST authentication_failed Attributes
Attribute | Description |
---|---|
|
The IP address from which the request originated. |
|
The principal (username) that failed authentication. |
|
The REST endpoint URI. |
|
The body of the request, if enabled. |
Table 12. REST realm_authentication_failed Attributes
Attribute | Description |
---|---|
|
The IP address from which the request originated. |
|
The principal (username) that failed authentication. |
|
The REST endpoint URI. |
|
The body of the request, if enabled. |
|
The realm that failed to authenticate the user. NOTE: A separate entry is logged for each consulted realm. |
Table 13. Transport anonymous_access_denied Attributes
Attribute | Description |
---|---|
|
Where the request originated: |
|
The IP address from which the request originated. |
|
The name of the action that was executed. |
|
The type of request that was executed. |
|
A comma-separated list of indices this request pertains to (when applicable). |
Table 14. Transport authentication_success Attributes
Attribute | Description |
---|---|
|
Where the request originated: |
|
The IP address from which the request originated. |
|
The authenticated user. |
|
The realm that authenticated the user. |
|
The name of the action that was executed. |
|
The type of request that was executed. |
Table 15. Transport authentication_failed Attributes
Attribute | Description |
---|---|
|
Where the request originated: |
|
The IP address from which the request originated. |
|
The principal (username) that failed authentication. |
|
The name of the action that was executed. |
|
The type of request that was executed. |
|
A comma-separated list of indices this request pertains to (when applicable). |
Table 16. Transport realm_authentication_failed Attributes
Attribute | Description |
---|---|
|
Where the request originated: |
|
The IP address from which the request originated. |
|
The principal (username) that failed authentication. |
|
The name of the action that was executed. |
|
The type of request that was executed. |
|
A comma-separated list of indices this request pertains to (when applicable). |
|
The realm that failed to authenticate the user. NOTE: A separate entry is logged for each consulted realm. |
Table 17. Transport access_granted Attributes
Attribute | Description |
---|---|
|
Where the request originated: |
|
The IP address from which the request originated. |
|
The principal (username) that passed authentication. |
|
The set of roles granting permissions. |
|
The name of the action that was executed. |
|
The type of request that was executed. |
|
A comma-separated list of indices this request pertains to (when applicable). |
Table 18. Transport access_denied Attributes
Attribute | Description |
---|---|
|
Where the request originated: |
|
The IP address from which the request originated. |
|
The principal (username) that failed authentication. |
|
The set of roles granting permissions. |
|
The name of the action that was executed. |
|
The type of request that was executed. |
|
A comma-separated list of indices this request relates to (when applicable). |
Table 19. Transport tampered_request Attributes
Attribute | Description |
---|---|
|
Where the request originated: |
|
The IP address from which the request originated. |
|
The principal (username) that failed to authenticate. |
|
The name of the action that was executed. |
|
The type of request that was executed. |
|
A comma-separated list of indices this request pertains to (when applicable). |
Table 20. IP Filter connection_granted Attributes
Attribute | Description |
---|---|
|
The IP address from which the request originated. |
|
The transport profile the request targeted. |
|
The IP filtering rule that granted the request. |
Table 21. IP Filter connection_denied Attributes
Attribute | Description |
---|---|
|
The IP address from which the request originated. |
|
The transport profile the request targeted. |
|
The IP filtering rule that denied the request. |
Logfile Audit Output
editThe logfile
audit output is the default output for auditing. It writes data to
the <clustername>_access.log
file in the logs directory.
Log Entry Format
editThe format of a log entry is:
[<timestamp>] [<local_node_info>] [<layer>] [<entry_type>] <attribute_list>
-
<timestamp>
-
When the event occurred. You can configure the
timestamp format in
log4j2.properties
. -
<local_node_info>
- Information about the local node that generated the log entry. You can control what node information is included by configuring the local node info settings.
-
<layer>
-
The layer from which this event originated:
rest
,transport
orip_filter
. -
<entry_type>
-
The type of event that occurred:
anonymous_access_denied
,authentication_failed
,access_denied
,access_granted
,connection_granted
,connection_denied
. -
<attribute_list>
-
A comma-separated list of key-value pairs that contain
data pertaining to the event. Formatted as
attr1=[val1], attr2=[val2]
. See Audit Entry Attributes for the attributes that can be included for each type of event.
Logfile Output Settings
editThe events and some other information about what gets logged can be
controlled using settings in the elasticsearch.yml
file. See
Audited Event Settings and
Local Node Info Settings.
No filtering is performed when auditing, so sensitive data may be audited in plain text when including the request body in audit events.
You can also configure how the logfile is written in the log4j2.properties
file located in ES_PATH_CONF/x-pack
. By default, audit information is appended to the
<clustername>_access.log
file located in the standard Elasticsearch logs
directory
(typically located at $ES_HOME/logs
). The file rolls over on a daily basis.
Logfile Audit Events Ignore Policies
editThe comprehensive audit trail is necessary to ensure accountability. It offers tremendous value during incident response and can even be required for demonstrating compliance.
The drawback of an audited system is represented by the inevitable performance penalty incurred.
In all truth, the audit trail spends I/O ops that are not available anymore for the user’s queries.
Sometimes the verbosity of the audit trail may become a problem that the event type restrictions,
defined by include
and exclude
, will not alleviate.
Audit events ignore policies are a finer way to tune the verbosity of the audit trail. These policies define rules that match audit events which will be ignored (read as: not printed). Rules match on the values of attributes of audit events and complement the include/exclude method. Imagine the corpus of audit events and the policies chopping off unwanted events.
When utilizing audit events ignore policies you are acknowledging potential accountability gaps that could render illegitimate actions undetectable. Please take time to review these policies whenever your system architecture changes.
A policy is a named set of filter rules. Each filter rule applies to a single event attribute,
one of the users
, realms
, roles
or indices
attributes. The filter rule defines
a list of Lucene regexp, any of which has to match the value of the audit
event attribute for the rule to match.
A policy matches an event if all the rules comprising it match the event.
An audit event is ignored, therefore not printed, if it matches any policy. All other
non-matching events are printed as usual.
All policies are defined under the xpack.security.audit.logfile.events.ignore_filters
settings namespace. For example, the following policy named example1 matches
events from the kibana or admin_user principals and operating over indices of the
wildcard form app-logs*:
xpack.security.audit.logfile.events.ignore_filters: example1: users: ["kibana", "admin_user"] indices: ["app-logs*"]
An audit event generated by the kibana user and operating over multiple indices , some of which do not match the indices wildcard, will not match. As expected, operations generated by all other users (even operating only on indices that match the indices filter) will not match this policy either.
Audit events of different types may have different attributes.
If an event does not contain an attribute for which some policy defines filters, the
event will not match the policy.
For example, the following policy named example2, will never match authentication_success
or
authentication_failed
events, irrespective of the user’s roles, because these
event schemas do not contain the role
attribute:
xpack.security.audit.logfile.events.ignore_filters: example2: roles: ["admin", "ops_admin_*"]
Likewise, any events of users with multiple roles, some of which do not match the regexps will not match this policy.
For completeness, although practical use cases should be sparse, a filter can match
a missing attribute of an event, using the empty string ("") or the empty list ([]).
For example, the following policy will match events that do not have the indices
attribute (anonymous_access_denied
, authentication_success
and other types) as well
as events over the next index.
xpack.security.audit.logfile.events.ignore_filters: example3: indices: ["next", ""]
Index Audit Output
editIn addition to logging to a file, you can store audit logs in Elasticsearch
rolling indices. These indices can be either on the same cluster, or on a
remote cluster. You configure the following settings in
elasticsearch.yml
to control how audit entries are indexed. To enable
this output, you need to configure the setting xpack.security.audit.outputs
in the elasticsearch.yml
file:
xpack.security.audit.outputs: [ index, logfile ]
For more configuration options, see Audit Log Indexing Configuration Settings.
No filtering is performed when auditing, so sensitive data may be audited in plain text when including the request body in audit events.
Audit Index Settings
editYou can also configure settings for the indices that the events are stored in.
These settings are configured in the xpack.security.audit.index.settings
namespace
in elasticsearch.yml
. For example, the following configuration sets the
number of shards and replicas to 1 for the audit indices:
xpack.security.audit.index.settings: index: number_of_shards: 1 number_of_replicas: 1
These settings apply to the local audit indices, as well as to the remote audit indices, but only if the remote cluster does not have X-Pack security installed, or the Elasticsearch versions are different. If the remote cluster has X-Pack security installed, and the versions coincide, the settings for the audit indices there will take precedence, even if they are unspecified (i.e. left to defaults).
Forwarding Audit Logs to a Remote Cluster
editTo index audit events to a remote Elasticsearch cluster, you configure
the following xpack.security.audit.index.client
settings:
-
xpack.security.audit.index.client.hosts
-
xpack.security.audit.index.client.cluster.name
-
xpack.security.audit.index.client.xpack.security.user
For more information about these settings, see Remote Audit Log Indexing Configuration Settings.
You can pass additional settings to the remote client by specifying them in the
xpack.security.audit.index.client
namespace. For example, to allow the remote
client to discover all of the nodes in the remote cluster you can specify the
client.transport.sniff
setting:
xpack.security.audit.index.client.transport.sniff: true