IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« Internal knowledge search architecture MongoDB native connector tutorial »

› ›

Native connectors (managed service)

edit

Native connectors (managed service)

edit

Native connectors are Elastic connectors available directly within your Elastic Cloud deployment. No additional infrastructure is required.

Native connectors sync data sources directly to Elasticsearch indices. Create these indices using the Connector workflow within Kibana.

The following connectors are available as native connectors. See each connector reference for additional information specific to each connector.

Azure Blob Storage
Box
Confluence (including Confluence Data Center)
Dropbox
GitHub
Gmail
Google Cloud Storage
Google Drive
Jira (including Jira Data Center)
Microsoft SQL
MongoDB
MySQL
Network Drive
Notion
OneDrive
Oracle
Outlook
PostgreSQL
Amazon S3
Salesforce
ServiceNow
SharePoint Online
Slack
Teams
Zoom

Availability and prerequisites

edit

Native connectors were introduced in Elastic version 8.5.0.

Your Elastic cloud deployment must include the following Elastic services:

Elasticsearch
Kibana
Enterprise Search

Refer to Native Integrations on the Elastic subscriptions page, in the Elastic Search section for native connector licensing requirements.

Usage in Kibana UI

edit

Follow the Connector workflow in Kibana to select the Connector ingestion method. Choose a data source, create an Elasticsearch index, and configure a native connector to manage the index.

Select a connector

edit

Choose the data source to sync from the available options and select Continue.

Create index

edit

Create a new index to be managed by the connector:

Name your index and optionally change the language analyzer to match the human language of your data source. (The index name will be automatically prefixed with search-.)
Select Create index.

The index is created and ready to configure.

This operation requires:

Access to Kibana
Permission to create or manage the index
write indices privilege for the .elastic-connectors index
manage_api_key cluster privilege to allow the API key to be created
write_connector_secrets cluster privilege to allow the API key to be stored as a secret

Configure connector

edit

Create a new index to be managed by the connector.

Continue from above, or navigate to the following location within the Kibana UI:

Search > Content > Elasticsearch indices

Choose the index to configure, and then choose the Configuration tab.

Configure the connector:

Edit the name and description for the connector. Your team can use this information to differentiate this index from other connector indices. (These fields describe the connector and are independent of the Elasticsearch index name.)
Save your changes.
Edit the data source configuration. The fields here vary by connector. Refer to the documentation for each connector for details (see list of native connectors, above). See Security for security considerations.
Save your changes.

Optionally choose Edit sync schedule to begin managing the connector.

This operation requires:

Access to Kibana
Permission to create or manage the index
write indices privilege for the .elastic-connectors index
manage_api_key cluster privilege to allow the API key to be created
write_connector_secrets cluster privilege to allow the API key to be stored as a secret

Manage connector

edit

To manage documents, syncs, sync rules, ingest pipelines, and other connector features, see Using connectors.

Manage API keys

edit

API keys for native connectors were introduced in Elastic version 8.13.0. Native connectors created in earlier versions will not automatically use API keys upon upgrading to 8.13.0. See enabling API keys for more information.

Native connectors communicate with Elasticsearch using API keys. When managing a native connector through the Connectors page in the Kibana UI, API key management is handled automatically. API keys for native connectors have the manage permission for three indices:

The attached index
The access control (ACL) index used for document level security
The internal .elastic-connectors index.

Changing the attached index through Kibana will automatically invalidate the existing API key and generate a new one. If you want to rotate an existing API key, navigate to the Configuration tab. Scroll down to Manage API key and select Generate API key. This action will invalidate the previous API key, create a new API key, and update the connector secret.

API keys for native connectors are stored on an internal system index called .connector-secrets. This index can only be written to through API requests by users with the write_connector-secrets cluster privilege. Only the Enterprise Search instance has permission to read from this index.

Users managing native connectors will need the write_connector_secrets cluster privilege assigned to their role. See Security for security considerations.

Manage API keys programmatically

edit

You can also create and store API keys programmatically.

Expand the following section for details.

The user needs the cluster privileges manage_api_key and write_connector_secrets to generate API keys programmatically.

To create an API key for the connector:

Run the following command, replacing values where indicated. Note the id and encoded return values from the response:

POST /_security/api_key
{
  "name": "<connector_name>-connector-api-key",
  "role_descriptors": {
    "<connector_name>-connector-role": {
      "cluster": [
        "monitor"
      ],
      "indices": [
        {
          "names": [
            "<index_name>",
            ".search-acl-filter-<index_name>",
            ".elastic-connectors*"
          ],
          "privileges": [
            "all"
          ],
          "allow_restricted_indices": false
        }
      ]
    }
  }
}

Use the encoded value to store a connector secret, and note the id return value from this response:
```
POST _connector/_secret
{
  "value": <encoded_api_key>
}
```

Use the API key id and the connector secret id to update the connector:

PUT /_connector/<connector_id>/_api_key_id
{
  "api_key_id": "<API key id>",
  "api_key_secret_id": "<secret id>"
}

Enabling API keys for upgraded connectors

edit

Native connectors created before 8.13.0 do not initially have API keys upon upgrading. The attached indices of these connectors cannot be changed until the connector has been converted to use an API key.

Expand the following section for steps on enabling API keys for upgraded native connectors.

Run the following command in Dev Tools to enable API keys for the connector, replacing values where indicated.

POST .elastic-connectors/_update/<connector_id>
{
  "doc": {
    "features": {
      "native_connector_api_keys": {
        "enabled": true
      }
    }
  }
}

Go back to the Connectors page and navigate to the Configuration tab.
Scroll down to Manage API key and select Generate API key.

Your native connector is now using API keys to authorize ingestion into Elasticsearch.

Usage via API

edit

In 8.12 we introduced a set of Connector APIs to create and manage Elastic connectors and sync jobs, along with a CLI tool. Use these tools if you’d like to work with connectors and sync jobs programmatically.

End-to-end example

edit

The following example demonstrates how to use a native connector: Native connector tutorial (MongoDB).

Convert a native connector

edit

You can convert a native connector to a self-managed connector client to be run on your own infrastructure. You’ll find instructions in the UI on the connector index’s overview page.

Converting a native connector to a self-managed connector client is an irreversible operation!

« Internal knowledge search architecture MongoDB native connector tutorial »