Native connectors

edit

Native connectors

edit

Native connectors are a collection of connectors that are available through Elastic Cloud. Use native connectors to integrate with data sources using only the Kibana UI—​no development required.

View Available connectors and Availability and prerequisites to evaluate native connectors for your use case.

See Usage for a complete reference to using these connectors, and review Troubleshooting if you encounter issues.

Available connectors

edit

The following native connectors are available:

Availability and prerequisites

edit

Native connectors were introduced in Elastic version 8.5.0.

Currently, all native connectors are in technical preview. Features in technical preview are subject to change and are not covered by the service level agreement (SLA) of features that have reached general availability (GA).

Native connectors are available to all Elastic Cloud deployments. Your deployment must include the Elasticsearch, Kibana, and Enterprise Search services. Your Enterprise Search service should have 4+ GB RAM per zone to avoid out of memory errors. See Infrastructure requirements to learn how to verify and change the RAM for your Enterprise Search service.

Native connectors are also available to self-managed deployments if the subscription requirements are satisfied. View the requirements for this feature under the Elastic Enterprise Search section of the Elastic Stack subscriptions page.

Self-managed deployments must additionally include a connector service, running in native mode. Connector services are available in multiple programming languages. Refer to the following repositories for documentation:

  • Elastic connector framework for Python (branch 8.5, compatible with Elastic 8.5)
  • Elastic connector framework for Ruby (branch 8.5, compatible with Elastic 8.5)

Usage

edit

Kibana provides an interface to manage all connector indices. From there, you can manage existing connectors and set up additional connectors.

For each connector you’d like to use, complete the following:

  1. Gather the information about the data source you will need to configure the index, such as host and access credentials. Refer to each connector’s reference documentation for details.
  2. Create a new connector index.
  3. Configure the index with information about the data source.
  4. Manage recurring syncs or sync once to begin syncing data to the index and manage ongoing updates.
  5. View the index status and details and synced documents.
  6. Optionally use ingest pipelines to post-process or transform document fields.

If you encounter issues, see Troubleshooting for help.

All native connectors share several prerequisites.

Each native connector has its own reference, which may include additional prerequisites or usage instructions.

Manage connector indices

edit

Within Kibana, navigate to:

Enterprise SearchContentElasticsearch indices

View a list of indices and their attributes, including index health and ingestion status. Use the Ingestion method field to identify which indices are associated with connectors.

Within this interface, you can choose to view the details for each existing index or delete an index. Or, you can create a new index.

These operations require access to Kibana and additional index privileges.

Create index

edit

Continue from above, or navigate to the following breadcrumb within Kibana:

Enterprise SearchContentElasticsearch indices

Create a new connector index:

  1. Choose Create new index.
  2. Provide the connector name and optionally change the language analyzer to match the human language of your data source. (The name you provide is automatically prefixed with search-.)
  3. Save your changes.
  4. When prompted, choose from the available connectors.
  5. Save your changes.

The index is created and ready to configure.

This operation requires access to Kibana and the write indices privilege for the .elastic-connectors index.

Configure index

edit

Continue from above, or navigate to the following breadcrumb within Kibana:

Enterprise SearchContentElasticsearch indices

Choose the index to configure, and then choose the Configuration tab.

Configure the index:

  1. Edit the name and description for the connector. Your team can use this information to differentiate this index from other connector indices. (These fields describe the connector and are independent of the Elasticsearch index name.)
  2. Save your changes.
  3. Edit the data source configuration. The fields here vary by connector. Refer to the documentation for each connector for details.
  4. Save your changes.
  5. Optionally choose Edit sync schedule to move to the Scheduling tab to manage recurring syncs.

This operation requires access to Kibana and the write indices privilege for the .elastic-connectors index.

See Security for security details.

Manage recurring syncs

edit

Continue from above, or navigate to the following breadcrumb within Kibana:

Enterprise SearchContentElasticsearch indices

Choose the index to configure, and then choose the Scheduling tab.

Within this interface, you can enable or disable recurring syncs. When enabled, you can additionally manage the sync schedule.

This operation requires access to Kibana and the write indices privilege for the .elastic-connectors index.

Alternatively, you can sync once.

After you enable recurring syncs or sync once, the first sync will begin. (There may be a short delay before the connector service begins the first sync.) You may want to view the index details to see the status or errors, or view the synced documents.

Sync once

edit

After a connector index is configured, you can request a sync at any time.

Within Kibana, navigate to:

Enterprise SearchContentElasticsearch indices

Then choose the index to sync.

Regardless of which tab is active, the Sync button is always visible in the top right. Choose this button to request a sync. (There may be a short delay before the connector service begins the sync.)

This operation requires access to Kibana and the write indices privilege for the .elastic-connectors index.

View status

edit

View the index details to see a variety of information that communicate the status of the index and connector.

Within Kibana, navigate to:

Enterprise SearchContentElasticsearch indices

Then choose the index to view.

The Overview tab presents a variety of information, including:

  • Any errors affecting the connector or sync process.
  • The current ingestion status.
  • The current document count.
  • Recent sync history, including sync status.

This operation requires access to Kibana and the read indices privilege for the .elastic-connectors index.

View documents

edit

View the documents the connector has synced from the data source. Additionally view the index mappings to determine the current document schema.

Within Kibana, navigate to:

Enterprise SearchContentElasticsearch indices

Then choose the index to view.

Choose the Documents tab to view the synced documents. Choose the Index Mappings tab to view the index mappings that were created by the connector.

When setting up a new connector, ensure you are getting the documents and fields you were expecting from the data source. If not, see Troubleshooting for help.

These operations require access to Kibana and the read and manage indices privileges for the index containing the documents.

See Security for security details.

Manage ingest pipelines

edit

Each connector determines how it extracts data from its data source and transforms that data into Elasticsearch documents. However, you can use Ingest pipelines to perform additional processing and transformation on each document, before it is written to Elasticsearch.

Each connector index has a default ingest pipeline, which you can customize or replace through Kibana.

Within Kibana, navigate to:

Enterprise SearchContentElasticsearch indices

Then choose the index to manage and choose the Pipelines tab.

This functionality applies to all search indices, including crawler and API indices. Refer to the following documentation for details: Ingest pipelines.

Troubleshooting

edit

Use the following actions to help diagnose and resolve connector issues:

Security

edit

Elastic Cloud deployments have strong security defaults. For example, data is encrypted by default, whether at rest or in transit.

Self-managed deployments require more upfront work to ensure strong security. Refer to Secure the Elastic Stack in the Elasticsearch documentation for more information.

Access to credentials

edit

Credentials for the data source — such as API keys or username/password pair— are stored in your deployment’s .elastic-connectors Elasticsearch index. Therefore, the credentials are visible to all Elastic users with the read indices privilege for that index. By default, the following Elastic users have this privilege: the elastic superuser and the kibana_system user. Enterprise Search service account tokens can also read the .elastic-connectors index.

Access to documents

edit

Data synced from your data source are stored as documents in the Elasticsearch index you created. This data is visible to all Elastic users with the read indices privilege for that index. Be careful to ensure that access to this index is at least as restrictive as access to the original data source.

Encryption

edit

Elastic Cloud automatically encrypts data at rest. Data in transit is automatically encrypted using https.

Self-managed deployments must implement encryption at rest. See Configure security for the Elastic Stack in the Elasticsearch documentation for more information.

Request help or support

edit

Elastic provides several resources for help, and your Elastic subscription may include Elastic support for this feature.

Use the following resources to request help from the Elastic community:

Use the following resources to request Elastic support: