IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« PostgreSQL connector client tutorial Content syncs »
Elastic Docs › ›Elastic connectors

Using connectors

edit

Using connectors

edit

This document describes operations available to native connectors and connector clients.

Create and configure connector index

edit

Create a new index and configure a connector to manage the index’s documents.

The procedures for these operations are different for native connectors and connector clients:

Manage connector indices

edit

View and manage all Elasticsearch indices managed by connectors.

Within Kibana, navigate to:

Search > Content > Elasticsearch indices

View a list of indices and their attributes, including index health and ingestion status. Use the Ingestion method field to identify which indices are managed by connectors.

Within this interface, you can choose to view the details for each existing index or delete an index. Or, you can create a new connector index.

These operations require access to Kibana and additional index privileges.

Manage recurring syncs

edit

After creating an index to be managed by a connector, you can configure automatic, recurring syncs.

Within Kibana, navigate to:

Search > Content > Elasticsearch indices

Choose the index to configure, and then choose the Scheduling tab.

Within this interface, you can enable or disable scheduled:

  1. Full content syncs
  2. Incremental content syncs (if supported)
  3. Access control syncs (if supported)

When enabled, you can additionally manage the sync schedule.

This operation requires access to Kibana and the write indices privilege for the .elastic-connectors index.

Alternatively, you can sync once.

After you enable recurring syncs or sync once, the first sync will begin. (There may be a short delay before the connector service begins the first sync.) You may want to view the index details to see the status or errors, or view the synced documents.

Sync once

edit

After creating the index to be managed by a connector, you can request a single sync at any time.

Within Kibana, navigate to:

Search > Content > Elasticsearch indices

Then choose the index to sync.

Regardless of which tab is active, the Sync button is always visible in the top right. Choose this button to reveal sync options:

  1. Full content
  2. Incremental content (if supported)
  3. Access control (if supported)

Choose one of the options to request a sync. (There may be a short delay before the connector service begins the sync.)

This operation requires access to Kibana and the write indices privilege for the .elastic-connectors index.

Cancel sync

edit

After a sync has started, you can cancel the sync before it completes.

Within Kibana, navigate to:

Search > Content > Elasticsearch indices

Then choose the index with the running sync.

Regardless of which tab is active, the Sync button is always visible in the top right. Choose this button to reveal sync options, and choose Cancel Syncs to cancel active syncs. This will cancel the running job, and marks all pending and suspended jobs as canceled as well. (There may be a short delay before the connector service cancels the syncs.)

This operation requires access to Kibana and the write indices privilege for the .elastic-connectors and .elastic-connectors-sync-jobs index.

View status

edit

View the index details to see a variety of information that communicate the status of the index and connector.

Within Kibana, navigate to:

Search > Content > Elasticsearch indices

Then choose the index to view.

The Overview tab presents a variety of information, including:

  • General information about the connector index, for example: name, description, ingestion type, connector type, and language analyzer.
  • Any errors affecting the connector or sync process.
  • The current ingestion status (see below for possible values).
  • The current document count.

Possible values of ingestion status:

  • Incomplete - A connector that is not configured yet.
  • Configured - A connector that is configured.
  • Connected - A connector that can successfully connect to a data source.
  • Error - A connector that failed to connect to the data source.
  • Connector failure - A connector that has not seen any update for more than 30 minutes.
  • Sync failure - A connector that failed in the last sync job.

This tab also displays the recent sync history, including sync status (see below for possible values).

Possible values of sync status:

  • Sync pending - The initial job status, the job is pending to be picked up.
  • Sync in progress - The job is running.
  • Canceling sync - Cancelation of the job has been requested.
  • Sync canceled - The job was canceled
  • Sync suspended - The job was suspended due to service shutdown, and it can be resumed when the service restarts.
  • Sync complete - The job completed successfully.
  • Sync failure - The job failed.

For each sync, choose the view button to display the job details, including:

  • The job ID
  • Document stats, including: number of documents added/deleted, total number of documents, and volume of documented added
  • Event logs
  • Sync rules that were active when the sync was requested
  • Pipelines that were active when the sync was requested

This operation requires access to Kibana and the read indices privilege for the .elastic-connectors index.

View documents

edit

View the documents the connector has synced from the data. Additionally view the index mappings to determine the current document schema.

Within Kibana, navigate to:

Search > Content > Elasticsearch indices

Then choose the index to view.

Choose the Documents tab to view the synced documents. Choose the Index Mappings tab to view the index mappings that were created by the connector.

When setting up a new connector, ensure you are getting the documents and fields you were expecting from the data source. If not, see Troubleshooting for help.

These operations require access to Kibana and the read and manage indices privileges for the index containing the documents.

See Security for security details.

Manage sync rules

edit

Use sync rules to limit which documents are fetched from the data source, or limit which fetched documents are stored in Elastic.

Within Kibana, navigate to:

Search > Content > Elasticsearch indices

Then choose the index to manage and choose the Sync rules tab.

Manage ingest pipelines

edit

Use ingest pipelines to transform fetched data before it is stored in Elastic.

Within Kibana, navigate to:

Search > Content > Elasticsearch indices

Then choose the index to manage and choose the Pipelines tab.

« PostgreSQL connector client tutorial Content syncs »