Migrating from Shield

Most Shield settings have simply had their prefix changes from shield. to xpack.security.. Hare a few examples of Shield 2.x settings and their new X-Pack counterparts:

Persistent cluster state settings used for IP Filtering are not automatically migrated to the new names. You will need to manually migrate these settings.

From 5.0 onward, you should use the native realm to manage roles and local users. To migrate existing file-based users and roles to the native realm, use the migrate tool that’s included with the X-Pack plugin.

The migrate tool loads the existing file-based users and roles and calls the user and roles APIs to add them to the native realm. You can migrate all users and roles, or specify the ones you want to migrate. Users and roles that already exist in the native realm are not replaced or overridden. If the names you specify with the --users and --roles options don’t exist in the file realm, they are skipped.

Run the migrate tool after you install the X-Pack plugin. For example:

$ bin/x-pack/migrate native -U http://localhost:9200 -u elastic -p changeme
-n lee,foo -r role1,role2,role3,role4,foo
starting migration of users and roles...
importing users from [/home/es/config/shield/users]...
found existing users: [test_user, joe3, joe2]
migrating user [lee]
{"user":{"created":true}}
no user [foo] found, skipping
importing roles from [/home/es/config/shield/roles.yml]...
found existing roles: [marvel_user, role_query_fields, admin_role, role3, admin,
remote_marvel_agent, power_user, role_new_format_name_array, role_run_as,
logstash, role_fields, role_run_as1, role_new_format, kibana4_server, user,
transport_client, role1.ab, role_query]
migrating role [role1]
{"role":{"created":true}}
migrating role [role2]
{"role":{"created":true}}
role [role3] already exists, skipping
no role [foo] found, skipping
users and roles imported.

The native subcommand supports the following options:

Additionally, the -E flag can be used to specify additional settings. For example to specify a different configuration directory, the command would look like:

$ bin/x-pack/migrate native -U http://localhost:9200 -u elastic -p changeme -Epath.conf=/etc/elasticsearch

In Shield, the location of the users, users_roles, roles.yml, and system_key files could be changed using settings. In X-Pack, these files must always reside in the CONFIG_DIR/x-pack directory.

In Shield, esusers was the type for the file based realm. This realm type has been renamed to file in X-Pack.

Additionally, the names of users in the file based realm are not allowed to conflict with the built-in users.

In X-Pack, the format for specifying roles in the roles.yml file is much stricter:

The easiest way to migrate your roles, will be by converting the roles to native roles using the migrate tool prior to upgrading to X-Pack.

The following role shows how to specify multiple cluster and index permissions using collections.

Using collection notation to define roles.

get_user: 
  cluster: 
    - monitor
    - manage_security
  indices:
    - names: 
        - events_index
        - results
      privileges:
        - read
        - monitor
    - names: 
        - user_index
      privileges:
        - monitor
        - read
      query:
        term:
          department_id: 12
      field_security:
        grant:
          - name
          - email
          - contact.*
        except:
          - contact.mobile

You can also use arrays to specify multiple cluster and index permissions. The following role uses the array notation to define the same role as in the previous example.

Using array notation to define roles.

get_user:
  cluster: [ monitor, manage_security ]
  indices:
    - names: [ events_index, results ]
      privileges: [ read, monitor ]
    - names: user_index
      privileges: [ monitor, read ]
      query:
        term:
          department_id: 12
      field_security:
        grant: [ name, email, contact.* ]
        except: [ contact.mobile ]

Values for cluster, privileges, and the entries in fields_security must be specified as a collection or array even if there’s only one value. You can specify a single value for the names field.

If you have roles that use an index name or expression as a key, you’ll need to update them. For example, the following role definitions are no longer valid.

# Unsupported role format: index name as key and privilege as value
get_user:
  indices:
    'events_index': read

# Unsupported role format: index name as the key with privileges as a value within the object
get_user:
  indices:
    'events_index':
      privileges: read

You can rewrite this role to work with X-Pack using collection notation:

# Correct role format for X-Pack
get_user:
  indices:
    - names: 
        - events_index
      privileges:
        - read

Similarly, if you have roles that use comma-separated values to specify multiple values, you need to rewrite them to use the collection or array notation. For example, the following role definition is no longer valid.

# Unsupported role format: comma-separated values
get_user:
  cluster: monitor, manage
  indices:
    'events_index,results':
       - privileges: read, write, create_index
       - field_security:
         - grant: fieldA, fieldB

You can rewrite this role using either collections or arrays.

Specifying multiple values with a collection.

get_user:
  cluster:
    - monitor
    - manage
  indices:
    - names :
        - 'events_index'
        - 'results'
      privileges:
        - read
        - write
        - create_index
      field_security:
        grant:
          - fieldA
          - fieldB

Specifying multiple values with an array.

get_user:
  cluster: [ 'monitor', 'manage' ]
  indices:
    - names: [ 'events_index', 'results' ]
      privileges: [ 'read', 'write', 'create_index' ]
      field_security:
        grant: [ 'fieldA', 'fieldB' ]

If you have roles that specify single values for cluster, privileges, and fields, you need to rewrite them to use the collection or array notation. For example, the following role definition is no longer valid.

# Unsupported role format: single values
get_user:
  cluster: manage
  indices:
    'events_index':
       - privileges: read
         field_security:
           grant: fieldA

You can rewrite this role using collections or arrays (or a combination of the two). Note that the names field still accepts a single value.

Specifying multiple values with collections and arrays.

get_user:
  cluster:
    - manage
  indices:
    - names: 'events_index'
      privileges:
        - read
      field_security:
        grant: [ 'fieldA' ]

A few of the privileges that were available in Shield 2.x are no longer available in X-Pack. The following tables list the privileges that have been removed and their suggested replacements.

The percolator has been refactored and now requires a percolator field in the mapping. As a result, when using field level security with the percolator, access to the percolator field needs to be granted.

A new event type, realm_authentication_failed, has been added. These events were previously audited as authentication_failed events but in some cases produced confusing output. The realm_authentication_failed event indicates that the credentials provided by the user failed authentication for a specific realm; it does not indicate a failed login attempt as there could be other realms to try to authenticate against.

The logfile output no longer relies on the logger level to determine what events to output. The xpack.security.audit.logfile.events.include setting can be used to specify the events that should be output; you may also use the xpack.security.audit.logfile.events.exclude setting to simply exclude values from the defaults.

Some audit events would output the content of a REST request if the log level was configured at the TRACE level; control over this output has been replaced with the settings xpack.security.audit.logfile.events.emit_request_body and xpack.security.audit.index.events.emit_request_body for the logfile and index outputs. The default is false.

Finally, as part of the move to Log4j2 the logging.yml file has been replaced with the log4j2.properties file.

In X-Pack security, the default header name for submitting requests on behalf of other users has changed to es-security-runas-user. Applications making use of the run as functionality will need to use the new header name.