Logfile audit output
editLogfile audit output
editThe logfile
audit output is the default output for auditing. It writes data to
the <clustername>_audit.json
file in the logs directory. To maintain
compatibility with releases prior to 6.5.0, a <clustername>_access.log
file
is also generated. They differ in the output format but the contents
are similar. For systems that are not ingesting the audit file for search or
analytics it is strongly recommended to keep only the newer format.
To turn off the deprecated output format, you can disable the logger in the
log4j2.properties
file:
# change info to off # logger.xpack_security_audit_deprecated_logfile.level = info logger.xpack_security_audit_deprecated_logfile.level = off
Alternatively, use the cluster update settings API to dynamically configure the logger:
PUT /_cluster/settings { "persistent": { "logger.org.elasticsearch.xpack.security.audit.logfile.DeprecatedLoggingAuditTrail": "off" } }
If you overwrite the log4j2.properties
and do not specify appenders for
any of the audit trails, audit events are forwarded to the root appender, which
by default points to the elasticsearch.log
file.
Log entry format
editThe log entries in the <clustername>_audit.json
file have the following format:
- Each log entry is a one line JSON document and each one is printed on a separate line.
-
The fields of a log entry are ordered. However, if a field does not have a value it
will not be printed. The precise line pattern, together with the complete field
order, are specified in the
log4j2.properties
config file. - The log entry does not contain nested inner JSON objects, i.e. the doc is flat.
- The field names follow a dotted notation to flatten inner objects.
- A field’s value can be a string, a number or an array of strings.
- A field’s value, a request body as well, will be escaped as per the JSON RFC 4627.
There is a list of audit event types specifying the set of fields for each sog entry type.
Deprecated log entry format
editThe log entries in the <clustername>_access.log
file have the following format:
[<timestamp>] [<local_node_info>] [<layer>] [<entry_type>] <attribute_list>
-
<timestamp>
-
When the event occurred. You can configure the
timestamp format in
log4j2.properties
. -
<local_node_info>
- Information about the local node that generated the log entry. You can control what node information is included by configuring the local node info settings.
-
<layer>
-
The layer from which this event originated:
rest
,transport
orip_filter
. -
<entry_type>
-
The type of event that occurred:
anonymous_access_denied
,authentication_failed
,access_denied
,access_granted
,connection_granted
,connection_denied
. -
<attribute_list>
-
A comma-separated list of key-value pairs that contain
data pertaining to the event. Formatted as
attr1=[val1], attr2=[val2]
. See Audit Entry Attributes for the attributes that can be included for each type of event.
Logfile output settings
editThe events and some other information about what gets logged can be
controlled using settings in the elasticsearch.yml
file. See
Audited Event Settings and
Local Node Info Settings.
No filtering is performed when auditing, so sensitive data may be audited in plain text when including the request body in audit events.
You can also configure how the logfile is written in the log4j2.properties
file located in ES_PATH_CONF
. By default, audit information is appended to the
<clustername>_audit.json
file located in the standard Elasticsearch logs
directory
(typically located at $ES_HOME/logs
). The file rolls over on a daily basis.
The deprecated logfile audit format (<clustername>_access.log
) can be disabled
from the same log4j2.properties
file (hint: look for the comment
instructing to set the log level to off
). The deprecated format is a duplication
of information that is in place to assure backwards compatibility. If you are
not strict about the audit format it is strongly recommended to only use the
<clustername>_audit.json
log appender.
Logfile audit events ignore policies
editThe comprehensive audit trail is necessary to ensure accountability. It offers tremendous value during incident response and can even be required for demonstrating compliance.
The drawback of an audited system is represented by the inevitable performance penalty incurred.
In all truth, the audit trail spends I/O ops that are not available anymore for the user’s queries.
Sometimes the verbosity of the audit trail may become a problem that the event type restrictions,
defined by include
and exclude
, will not alleviate.
Audit events ignore policies are a finer way to tune the verbosity of the audit trail. These policies define rules that match audit events which will be ignored (read as: not printed). Rules match on the values of attributes of audit events and complement the include/exclude method. Imagine the corpus of audit events and the policies chopping off unwanted events.
When utilizing audit events ignore policies you are acknowledging potential accountability gaps that could render illegitimate actions undetectable. Please take time to review these policies whenever your system architecture changes.
A policy is a named set of filter rules. Each filter rule applies to a single event attribute,
one of the users
, realms
, roles
or indices
attributes. The filter rule defines
a list of Lucene regexp, any of which has to match the value of the audit
event attribute for the rule to match.
A policy matches an event if all the rules comprising it match the event.
An audit event is ignored, therefore not printed, if it matches any policy. All other
non-matching events are printed as usual.
All policies are defined under the xpack.security.audit.logfile.events.ignore_filters
settings namespace. For example, the following policy named example1 matches
events from the kibana or admin_user principals and operating over indices of the
wildcard form app-logs*:
xpack.security.audit.logfile.events.ignore_filters: example1: users: ["kibana", "admin_user"] indices: ["app-logs*"]
An audit event generated by the kibana user and operating over multiple indices , some of which do not match the indices wildcard, will not match. As expected, operations generated by all other users (even operating only on indices that match the indices filter) will not match this policy either.
Audit events of different types may have different attributes.
If an event does not contain an attribute for which some policy defines filters, the
event will not match the policy.
For example, the following policy named example2, will never match authentication_success
or
authentication_failed
events, irrespective of the user’s roles, because these
event schemas do not contain the role
attribute:
xpack.security.audit.logfile.events.ignore_filters: example2: roles: ["admin", "ops_admin_*"]
Likewise, any events of users with multiple roles, some of which do not match the regexps will not match this policy.
For completeness, although practical use cases should be sparse, a filter can match
a missing attribute of an event, using the empty string ("") or the empty list ([]).
For example, the following policy will match events that do not have the indices
attribute (anonymous_access_denied
, authentication_success
and other types) as well
as events over the next index.
xpack.security.audit.logfile.events.ignore_filters: example3: indices: ["next", ""]