Logfile audit events ignore policies

The comprehensive audit trail is necessary to ensure accountability. It offers tremendous value during incident response and can even be required for demonstrating compliance.

The drawback of an audited system is represented by the inevitable performance penalty incurred. In all truth, the audit trail spends I/O ops that are not available anymore for the user’s queries. Sometimes the verbosity of the audit trail may become a problem that the event type restrictions, defined by include and exclude, will not alleviate.

Audit events ignore policies are a finer way to tune the verbosity of the audit trail. These policies define rules that match audit events which will be ignored (read as: not printed). Rules match on the values of attributes of audit events and complement the include or exclude method. Imagine the corpus of audit events and the policies chopping off unwanted events. With a sole exception, all audit events are subject to the ignore policies. The exception are events of type security_config_change, which cannot be filtered out, unless excluded altogether.

A policy is a named set of filter rules. Each filter rule applies to a single event attribute, one of the users, realms, actions, roles or indices attributes. The filter rule defines a list of Lucene regexp, any of which has to match the value of the audit event attribute for the rule to match. A policy matches an event if all the rules comprising it match the event. An audit event is ignored, therefore not printed, if it matches any policy. All other non-matching events are printed as usual.

All policies are defined under the xpack.security.audit.logfile.events.ignore_filters settings namespace. For example, the following policy named example1 matches events from the kibana_system or admin_user principals that operate over indices of the wildcard form app-logs*:

xpack.security.audit.logfile.events.ignore_filters:
  example1:
    users: ["kibana_system", "admin_user"]
    indices: ["app-logs*"]

An audit event generated by the kibana_system user and operating over multiple indices , some of which do not match the indices wildcard, will not match. As expected, operations generated by all other users (even operating only on indices that match the indices filter) will not match this policy either.

Audit events of different types may have different attributes. If an event does not contain an attribute for which some policy defines filters, the event will not match the policy. For example, the following policy will never match authentication_success or authentication_failed events, irrespective of the user’s roles, because these event schemas do not contain the role attribute:

xpack.security.audit.logfile.events.ignore_filters:
  example2:
    roles: ["admin", "ops_admin_*"]

Likewise, any events of users with multiple roles, some of which do not match the regexps will not match this policy.

For completeness, although practical use cases should be sparse, a filter can match a missing attribute of an event, using the empty string ("") or the empty list ([]). For example, the following policy will match events that do not have the indices attribute (anonymous_access_denied, authentication_success and other types) as well as events over the next index.

xpack.security.audit.logfile.events.ignore_filters:
  example3:
    indices: ["next", ""]