Connectors UI in Kibana

edit

This document describes operations available to managed connectors and self-managed connectors, using the UI.

In the Kibana UI, go to Search > Content > Connectors to view a summary of all your connectors and sync jobs, and to create new connectors.

In 8.12 we introduced a set of Connector APIs to create and manage Elastic connectors and sync jobs, along with a CLI tool. Use these tools if you’d like to work with connectors and sync jobs programmatically, without using the UI.

Create and configure connectors

edit

You connector writes data to an Elasticsearch index.

To create managed connectors or self-managed self-managed connectors, use the buttons under Search > Content > Connectors. Once you’ve chosen the data source type you’d like to sync, you’ll be prompted to create an Elasticsearch index.

Manage connector indices

edit

View and manage all Elasticsearch indices managed by connectors.

In the Kibana UI, navigate to Search > Content > Connectors to view a list of connector indices and their attributes, including connector type health and ingestion status.

Within this interface, you can choose to view the details for each existing index or delete an index. Or, you can create a new connector index.

These operations require access to Kibana and additional index privileges.

Customize connector index mappings and settings

edit

Elasticsearch stores your data as documents in an index. Each index is made up of a set of fields and each field has a type (such as keyword, boolean, or date).

Mapping is the process of defining how a document, and the fields it contains, are stored and indexed. Connectors use dynamic mapping to automatically create mappings based on the data fetched from the source.

Index settings are configurations that can be adjusted on a per-index basis. They control things like the index’s performance, the resources it uses, and how it should handle operations.

When you create an index with a connector, the index is created with default search-optimized field template mappings and index settings. Mappings for specific fields are then dynamically created based on the data fetched from the source.

You can inspect your index mappings in the following ways:

  • In the Kibana UI: Navigate to Search > Content > Indices > YOUR-INDEX > Index Mappings
  • By API: Use the Get mapping API

You can manually edit the mappings and settings via the Elasticsearch APIs:

It’s important to note that these updates are more complex when the index already contains data.

Refer to the following sections for more information.

Customize mappings and settings before syncing data

edit

Updating mappings and settings is simpler when your index has no data. If you create and attach a new index while setting up a connector, you can customize the mappings and settings before syncing data, using the APIs mentioned earlier.

Customize mappings and settings after syncing data

edit

Once data has been added to Elasticsearch using dynamic mappings, you can’t directly update existing field mappings. If you’ve already synced data into an index and want to change the mappings, you’ll need to reindex your data.

The workflow for these updates is as follows:

  1. Create a new index with the desired mappings and settings.
  2. Reindex your data from the old index into this new index.
  3. Delete the old index.
  4. (Optional) Use an alias, if you want to retain the old index name.
  5. Attach your connector to the new index or alias.

Manage recurring syncs

edit

After creating an index to be managed by a connector, you can configure automatic, recurring syncs.

In the Kibana UI, navigate to Search > Content > Connectors.

Choose the index to configure, and then choose the Scheduling tab.

Within this interface, you can enable or disable scheduled:

  1. Full content syncs
  2. Incremental content syncs (if supported)
  3. Access control syncs (if supported)

When enabled, you can additionally manage the sync schedule.

This operation requires access to Kibana and the write indices privilege for the .elastic-connectors index.

Alternatively, you can sync once.

After you enable recurring syncs or sync once, the first sync will begin. (There may be a short delay before the connector service begins the first sync.) You may want to view the index details to see the status or errors, or view the synced documents.

Sync once

edit

After creating the index to be managed by a connector, you can request a single sync at any time.

In the Kibana UI, navigate to Search > Content > Elasticsearch indices.

Then choose the index to sync.

Regardless of which tab is active, the Sync button is always visible in the top right. Choose this button to reveal sync options:

  1. Full content
  2. Incremental content (if supported)
  3. Access control (if supported)

Choose one of the options to request a sync. (There may be a short delay before the connector service begins the sync.)

This operation requires access to Kibana and the write indices privilege for the .elastic-connectors index.

Cancel sync

edit

After a sync has started, you can cancel the sync before it completes.

In the Kibana UI, navigate to Search > Content > Elasticsearch indices.

Then choose the index with the running sync.

Regardless of which tab is active, the Sync button is always visible in the top right. Choose this button to reveal sync options, and choose Cancel Syncs to cancel active syncs. This will cancel the running job, and marks all pending and suspended jobs as canceled as well. (There may be a short delay before the connector service cancels the syncs.)

This operation requires access to Kibana and the write indices privilege for the .elastic-connectors and .elastic-connectors-sync-jobs index.

View status

edit

View the index details to see a variety of information that communicate the status of the index and connector.

In the Kibana UI, navigate to Search > Content > Elasticsearch indices.

Then choose the index to view.

The Overview tab presents a variety of information, including:

  • General information about the connector index, for example: name, description, ingestion type, connector type, and language analyzer.
  • Any errors affecting the connector or sync process.
  • The current ingestion status (see below for possible values).
  • The current document count.

Possible values of ingestion status:

  • Incomplete - A connector that is not configured yet.
  • Configured - A connector that is configured.
  • Connected - A connector that can successfully connect to a data source.
  • Error - A connector that failed to connect to the data source.
  • Connector failure - A connector that has not seen any update for more than 30 minutes.
  • Sync failure - A connector that failed in the last sync job.

This tab also displays the recent sync history, including sync status (see below for possible values).

Possible values of sync status:

  • Sync pending - The initial job status, the job is pending to be picked up.
  • Sync in progress - The job is running.
  • Canceling sync - Cancelation of the job has been requested.
  • Sync canceled - The job was canceled
  • Sync suspended - The job was suspended due to service shutdown, and it can be resumed when the service restarts.
  • Sync complete - The job completed successfully.
  • Sync failure - The job failed.

For each sync, choose the view button to display the job details, including:

  • The job ID
  • Document stats, including: number of documents added/deleted, total number of documents, and volume of documented added
  • Event logs
  • Sync rules that were active when the sync was requested
  • Pipelines that were active when the sync was requested

This operation requires access to Kibana and the read indices privilege for the .elastic-connectors index.

View documents

edit

View the documents the connector has synced from the data. Additionally view the index mappings to determine the current document schema.

In the Kibana UI, navigate to Search > Content > Elasticsearch indices.

Then choose the index to view.

Choose the Documents tab to view the synced documents. Choose the Index Mappings tab to view the index mappings that were created by the connector.

When setting up a new connector, ensure you are getting the documents and fields you were expecting from the data source. If not, see Troubleshooting for help.

These operations require access to Kibana and the read and manage indices privileges for the index containing the documents.

See Security for security details.

Manage sync rules

edit

Use sync rules to limit which documents are fetched from the data source, or limit which fetched documents are stored in Elastic.

In the Kibana UI, navigate to Search > Content > Elasticsearch indices.

Then choose the index to manage and choose the Sync rules tab.

Manage ingest pipelines

edit

Use ingest pipelines to transform fetched data before it is stored in Elastic.

In the Kibana UI, navigate to Search > Content > Elasticsearch indices.

Then choose the index to manage and choose the Pipelines tab.