Common source synchronization issues

Symptoms:

Source synchronization fails and the following error is shown in the administrative area or observed in the Workplace Search logs:

    OAuth client ID and secret mismatch. Please review OAuth configuration.

Resolution:

In order for Workplace Search to synchronize data from the content source, a connection via OAuth is established between Workplace Search and the source. Authentication of the connection happens via OAuth Client ID and OAuth Client Secret configured on Workplace Search under content source configuration. OAuth Client ID and Secret for the source must match the one configured on Workplace Search settings for the respective source.

If you observe an error like the one above, it possibly means that the Client ID or Client Secret provided in your content source configuration is incorrect. Refer to source configuration documentation and update the OAuth credentials in your Workplace Search Sources > Details > Source settings for the content source in question.

Symptoms:

Source synchronization fails and the following error is shown in the administrative area or observed in the Workplace Search logs:

    OAuth access token could not be refreshed.

Resolution:

In order to keep the connection between Workplace Search and content source secure, the API access token is automatically refreshed on a frequent basis. If the you observe an error like the one above, it possibly means one of the following:

User access for the connected user has been revoked at the source. The source user profile used to establish connection with Workplace Search doesn’t have access to the source anymore. Contact your source administrator to ensure that the user still has access to the source. If the user access has been revoked then either provide the user access to the source again or connect to a new user on your Workplace Search Sources > Add an organization content source > Connect for the content source in question.

Source OAuth credentials have been updated since the last synchronization and are incorrect. Ensure that the Client ID and Client Secret provided in your content source configuration are correct. Refer to source configuration documentation and update OAuth credentials in your Workplace Search Sources > Details > Source settings for the content source in question.

The OAuth token granted by the source has expired and requires reauthentication. In such a case, a “Re-authenticate” button will show up on the Sources page for the source with the issue. Click the “Re-authenticate” button and re-authenticate with the source.

Symptoms:

Source synchronization fails and one of the following errors is shown in the administrative area or observed in Workplace Search logs:

    * Source sync could not be completed due to third-party throttling.

    * Source sync could not be completed due to third-party rate limiting.

Resolution:

To keep your content up to date, Workplace Search makes API calls to the content sources as often as necessary when synchronizing data.

If you are getting an error like the one above, it possibly means that Workplace Search API requests are being throttled or limited, causing the synchronization to fail. It is recommended to reach out to the third-party source account representative to increase or do away with throttling or rate limit ceiling for your Workplace Search deployment.

Symptoms:

Source synchronization fails and the following error is shown in the administrative area or observed in the Workplace Search logs:

    * Source could not be reached.

    * Connection closed by remote host.

Resolution:

If you are getting an error like the one above, it possibly means one of the following:

The source may be experiencing high network latency causing the API requests to fail. This can happen when the source is getting more requests than it can handle. Ensure that the source can be reached from your Workplace Search deployment.

The source may be down. This can happen due to unhandled exceptions, system overload or Workplace Search hitting source API rate limits. Ensure that the source is up and reachable from your Workplace Search deployment, and reach out to your source representative to get guidance on source API rate limits.

The source may be slow to respond causing the API calls to timeout. This can happen is the source is overloaded and is unable to process all requests in time causing some of the requests to fail. Ensure that the source is up and reachable from your Workplace Search deployment.

If all of the above resolution methods fail - reach out to Elastic Support for assistance.

Symptoms:

Source synchronization fails and one of the following errors is shown in the administrative area or observed in the Workplace Search logs:

    * Source sync halted after reaching a limit of X documents or Y seconds.

    * Sync job reached the maximum number of documents extracted X with Y documents.

    * Document limit reached for the source sync.

    * Source sync halted after reaching the execution limit.

    * Source encountered a problem while syncing.

Resolution:

In order to ensure optimal usage of resources for your deployment, Workplace Search is designed to ensure stability of deployment. Synchronization is halted if Workplace Search observes consumption of all available resources during extraction of content from a source.

If the error like the one above happens, Workplace Search will retry to synchronize content from the source at a later time. If the error comes up repeatedly, reach out to Elastic Support for assistance.

Symptoms:

Source synchronization fails and the following error is shown in the administrative area or observed in the Workplace Search logs:

    Exceeded maximum consecutive errors - saw X errors in a row.

Resolution:

Workplace Search ensures stability of deployment and has mechanisms in place that stop the system if unknown issues occur repeatedly. Source synchronization is one such process that has these fail-safes in place that stop the synchronization process if a number of errors are observed consecutively.

This number, workplace_search.content_source.max_consecutive_errors, defaults to 10 and can be configured in enterprise-search.yml. Setting a higher number can make the system prone to instability if there is a spike in issues while a lower number may cause the synchronizations to fail with fewer issues. When configuring, consider taking into account the total number of documents that need to be synchronized from the source.

If the error like the one above happens, refer to the diagnostics data for issue details. Reach out to Elastic Support for assistance if the issue remains unresolved.

Symptoms:

Source synchronization fails and the following error is shown in the administrative area or observed in the Workplace Search logs:

    Exceeded maximum error ratio of X. Of the last N documents, M had errors.

Resolution:

Workplace Search ensures stability of deployment and has mechanisms in place that stop the system if unknown issues occur repeatedly. Source synchronization is one such process that has these fail-safes in place that stop the synchronization process if a large number of errors are observed in a short span.

These numbers can be configured in enterprise-search.yml.

Setting a higher ratio can mask widespread issues, while a lower number may cause the synchronizations to fail due to just a few transient issues. When configuring, consider taking into account the total number of documents that need to be synchronized from the source.

If the error like the one above happens, refer to the diagnostics data for issue details. Reach out to Elastic Support for assistance if the issue remains unresolved.

Symptoms:

Source synchronization fails and the following error is shown in the administrative area or observed in the Workplace Search logs:

    Exceeded maximum number of errors - saw X errors in total.

Resolution:

Workplace Search ensures stability of deployment and has mechanisms in place that stop the system if unknown issues occur repeatedly. Source synchronization is one such process that has these fail-safes in place that stop the synchronization process if a number of errors are observed.

This number, workplace_search.content_source.max_errors, defaults to 1000 and can be configured in enterprise-search.yml. Setting a higher number can mask widespread issues, while a lower number may cause the synchronizations to fail due to just a few transient issues. When configuring, consider taking into account the total number of documents that need to be synchronized from the source.

If the error like the one above happens, refer to the diagnostics data for issue details. Reach out to Elastic Support for assistance if the issue remains unresolved.

Symptoms:

Source synchronization fails and the following error is shown in the administrative area or observed in the Workplace Search logs:

    Failed to save data to search index.

Resolution:

Workplace Search uses Elasticsearch under the hood to synchronize content from source. If the error like the one above happens, it possibly means that Workplace Search failed to save data to the search index on Elasticsearch. This could happen if the Elasticsearch is unreachable from your Workplace Search deployment or Elasticsearch server failed. Ensure that Elasticsearch is healthy and reachable from your Workplace Search deployment.

Reach out to Elastic Support for assistance if the issue remains unresolved.

Symptoms:

Source synchronization fails and one of the following errors is shown in the administrative area or observed in the Workplace Search logs:

    * SSL handshake failed.

    * SSL connection unsuccessful.

Resolution:

SSL/TLS for Workplace Search is configured at Enterprise Search level. If you get an error like the one above, it may possibly mean that there is a problem with your SSL configuration. Refer to the documentation for configuring SSL/TLS for your Enterprise Search instance.

Symptoms:

Source synchronization fails and one of the following errors is shown in the administrative area or observed in the Workplace Search logs:

    * Source returned a malformed response.

    * Source returned an unexpected error during synchronization.

Resolution:

If you get an error like the one above, it may possibly mean that Workplace Search was unable to parse a response from the source. This can happen due to multiple reasons like incomplete response, broken files etc. Refer to the diagnostics data for issue details and reach out to Elastic Support to report the issue.

Symptoms:

Source synchronization fails and one of the following errors is shown in the administrative area or observed in the Workplace Search logs:

    * Internal server error.

    * Unknown problem occurred when syncing.

Resolution:

If you get an error like the one above, it may possibly mean Workplace Search has come across an unknown exception during source synchronization. Refer to the diagnostics data to get more details on the exception. Please reach out to Elastic Support to report the issue.

Symptoms:

Source synchronization fails and the following error is shown in the administrative area or observed in the Workplace Search logs:

    Worker failed to claim invalid job. This is likely because another instance of Enterprise Search scheduled a duplicate job. If the error keeps happening, please contact Elastic Support.

Resolution:

Workplace Search uses an internal queue to store and execute asyncronous jobs. Each time a job is added or updated in the queue, internal validation prevents the job from entering an undesired state that can block execution of the queue. However, if a job gets into an invalid state, the system will mark the job as failed when running it.

Most likely this error is transient and should not affect synchronization, however if it keeps repeating, please reach out to Elastic Support for assistance.

Symptoms:

Source synchronization fails and the following error is shown in the administrative area or observed in the Workplace Search logs:

    Indexing configuration for Content Source is invalid. Please review indexing configuration for this content source.

Resolution:

Workplace Search allows you to customize indexing rules which can change what data is synchronized from the remote source. It’s possible to specify an incorrect indexing configuration which will cause the error to happen when extraction of the data is happening from the remote system.

The error message in the Workplace Search interface will contain the error message that was received from the remote system and will most likely contain the name of the field that is inaccessible or invalid in the indexing configuration. Removing this field or changing it to a valid field in the content source indexing configuration will resolve the problem.

For further assistance reach out to Elastic support at https://support.elastic.co/.

Review the documentation for content source sync diagnostics and recently completed synchronization jobs.