IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Migrating from App Search on Swiftype.com Logs and logging »
Elastic Docs › ›Elastic Enterprise Search server

Migrating from self-managed App Search

edit

Migrating from self-managed App Search

edit

From 7.7.0 onwards, App Search is distributed as Elastic Enterprise Search, a solution which combines Elastic App Search and Elastic Workplace Search into a single package (read the Elastic Cloud announcement). That transition means there is no direct upgrade path from App Search 7.6 to 7.7. Rather, App Search 7.6 users must migrate to Enterprise Search. The following document outlines the migration process.

Before attempting a migration, please ensure you are using the latest 7.x version of Enterprise Search. There are some known issues with migrations to versions before 7.9.1. See the the dedicated section below for more details.

Although this document describes a migration, the process resembles a standard Enterprise Search upgrade. We recommend first reviewing the Enterprise Search upgrade documentation.

Migration process overview

edit

When an App Search deployment is migrated to the new Enterprise Search solution, the App Search data stored in Elasticsearch needs to be converted into a new set of indexes compatible with Enterprise Search. This process is non-destructive (meaning App Search data is never removed or modified during an upgrade), allowing operators a reliable way to roll back in case of a failed upgrade attempt. Combined with the usage of the read-only mode flag, this means that an upgrade can be performed in-place without downtime.

After you begin the migration, there is no "downgrade" support. To restore App Search after migrating, you must start with a clean environment: remove all .ent-search-* and .app-search-* indices. Then restore a snapshot from before the migration and re-install App Search.

Limitations of the migration process

edit

While the migration process is designed to help you with your move to the Enterprise Search solution without any data loss, it has some limitations. Review these limitations before you proceed:

  • Disk Space – your Elasticsearch cluster needs to have enough disk space to accommodate two copies of your data for the period of the migration (since your data is re-indexed in a new format).
  • Analytics and Application Logs are not migrated – to reduce disk space requirements and the time it takes to perform a migration, we have decided not to support old Analytics and Application request logs in Enterprise Search. Your old analytics data will remain stored in Elasticsearch and could be accessed with Kibana, but the Enterprise Search analytics dashboard will not be able to see that data.
  • You cannot skip any migration steps – when deploying the new Enterprise Search solution on your Elasticsearch cluster that used to run App Search, you have to go through the migration process completely. If Enterprise Search detects any App Search indexes on the cluster, a migration will be attempted for each of them automatically and the service will be unavailable until the migration is complete.

If you do not want to migrate your App Search data, you can use a new Elasticsearch cluster or, as a last resort, you can take an Elasticsearch snapshot and then delete all App Search indexes (.app-search*) from the cluster before attempting an Enterprise Search deployment.

Before you begin

edit

Before attempting a migration of an App Search cluster to a new Enterprise Search solution, you need to take a few steps to guarantee your data safety and increase the probability of a successful upgrade:

  • Before you migrate production servers, test the migration process in a development environment to familiarize yourself with the process. Using an Elasticsearch snapshot created from a production deployment may be the best option to completely test the upgrade process without risking your service availability or data consistency.
  • Stop writing to your Elasticsearch cluster. If running App Search version 7.6 or later, enable the read-only mode to guarantee a consistent snapshot of your data. For versions before 7.6, manually stop all write/indexing operations to your cluster at the source.
  • Back up your data with Elasticsearch snapshots. Having a backup of your data will ensure your ability to safely roll back in case of a failed Enterprise Search upgrade.

Migrating from self-managed App Search deployments

edit

The following sections describe different options for migrating on-prem/self-managed App Search deployments.

When dealing with self-managed deployments, you have a lot of flexibility on how you want to approach the migration process. The options below are ordered by their complexity, which often is in reverse proportion to the option’s risk and associated downtime. Which option you choose will depend on your tolerance for downtime, the resources available to you and so on.

In-place migration to Enterprise Search

edit

For cases when you can tolerate some amount of downtime during the upgrade process, an in-place upgrade option may be the easiest one to attempt. Here are the steps you will need to take:

  1. Stop writing to your cluster (enable read-only mode if on the latest App Search version).
  2. Back up your data using an Elasticsearch snapshot.
  3. Stop your App Search instance.
  4. Deploy an Enterprise Search instance using the same Elasticsearch credentials and other settings from app-search.yml in your new enterprise-search.yml configuration file.
  5. Start Enterprise Search and allow it to perform the migration as needed.
  6. After Enterprise Search starts, you should have access to the App Search UI and APIs using your old credentials as usual.
  7. Remove read-only mode from Enterprise Search.
Live in-place migrations
edit

If you can only use one Elasticsearch cluster, but really need to keep your App Search downtime to a minimum, you can slightly change the migration process to perform an upgrade on a live App Search deployment:

  1. Stop writing to your cluster (enable read-only mode if on the latest App Search version).
  2. Back up your data using an Elasticsearch snapshot.
  3. Deploy an Enterprise Search instance using the same Elasticsearch credentials and other settings from your old app-search.yml in your new enterprise-search.yml configuration file. If the instance is going to be running on the same server(s) as App Search, make sure to change the port used by Enterprise Search (ent_search.listen_port) so that your instances could run in parallel. Also set the port in ent_search.external_url to match the new ent_search.listen_port port number.
  4. Start Enterprise Search and allow it to perform the migration as needed.
  5. Use your newly deployed Enterprise Search instance to verify that everything is working.
  6. Switch your production traffic to the new instance (by shifting traffic to the new port via a load-balancer or by changing client configuration).
  7. Shut down App Search when you are sure your deployment works as expected.
  8. Remove read-only mode from Enterprise Search.

Migrating to a new infrastructure

edit

Finally, the safest way to migrate from App Search to Enterprise Search is by migrating to a completely new infrastructure using the backups taken from the existing one:

  1. Stop writing to your cluster (enable read-only mode if on the latest App Search version).
  2. Back up your data using an Elasticsearch snapshot.
  3. Provision a new Elasticsearch cluster using the snapshot taken from the old cluster.
  4. Deploy an Enterprise Search instance using the new Elasticsearch cluster.
  5. Start Enterprise Search and allow it to perform the migration as needed.
  6. After Enterprise Search starts, you should have access to the App Search UI and APIs using your old credentials as usual.
  7. Switch your production traffic to the new product by shifting traffic to the instances via a load-balancer or by changing client configuration.
  8. Shut down App Search when you are sure your deployment works as expected.
  9. Remove read-only mode from Enterprise Search.

Known migration issues

edit

The known issues in Enterprise Versions before 7.9.1 related to migrations from App Search deployments are detailed below.

If you have migrated to an Enterprise Search before the the 7.9.1 version was released, we recommend you upgrade to 7.9.1 using the standard upgrade procedure.

For those who have not migrated yet, it should be safe to perform a direct migration from App Search 7.6 to Enterprise Search 7.9.1 or later as described above.

Incomplete Index Pointers Migration

edit

Enterprise Search has a notion of an index pointer – a database record, that controls which Elasticsearch index would be used to index documents for a specific App Search engine.

All Enterprise Search versions before 7.9.1 had a bug in their migration code that caused a newly migrated Enterprise Search deployment to continue attempting to index documents into an old (read-only) App Search index. As a result, successful updates for App Search documents are not visible in search results.

We recommend all Enterprise Search users to upgrade to the latest release to address this issue automatically.

Unique Constraints Migration

edit

Enterprise Search uses Elasticsearch as a data store for all configuration data like users, API keys, engines, etc. A part of the data store functionality is ensuring uniqueness constraints on things like engine names, user names, etc.

All Enterprise Search versions before 7.9.1 had a bug in their migration code that caused a subset of unique constraints to be migrated incorrectly from App Search deployments. As a result, operators were unable to generate new engine-scoped API tokens or create new role mappings for users with access limited to a single engine.

We recommend all Enterprise Search users to upgrade to the latest release to address this issue automatically.

« Migrating from App Search on Swiftype.com Logs and logging »