Native connectors (managed service)
editNative connectors (managed service)
editNative connectors are Elastic connectors available directly within your Elastic Cloud deployment. No additional infrastructure is required.
Native connectors sync data sources directly to Elasticsearch indices. Create these indices using the Connector workflow within Kibana.
The following connectors are available as native connectors. See each connector reference for additional information specific to each connector.
- Azure Blob Storage
- Box
- Confluence (including Confluence Data Center)
- Dropbox
- GitHub
- Gmail
- Google Cloud Storage
- Google Drive
- Jira (including Jira Data Center)
- Microsoft SQL
- MongoDB
- MySQL
- Network Drive
- Notion
- OneDrive
- Oracle
- Outlook
- PostgreSQL
- Amazon S3
- Salesforce
- ServiceNow
- SharePoint Online
- Slack
- Teams
- Zoom
Availability and prerequisites
editNative connectors were introduced in Elastic version 8.5.0.
Your Elastic cloud deployment must include the following Elastic services:
- Elasticsearch
- Kibana
- Enterprise Search
Refer to Native Integrations on the Elastic subscriptions page, in the Elastic Search section for native connector licensing requirements.
Usage in Kibana UI
editFollow the Connector workflow in Kibana to select the Connector ingestion method. Choose a data source, create an Elasticsearch index, and configure a native connector to manage the index.
Select a connector
editChoose the data source to sync from the available options and select Continue.
Create index
editCreate a new index to be managed by the connector:
-
Name your index and optionally change the language analyzer to match the human language of your data source.
(The index name will be automatically prefixed with
search-
.) - Select Create index.
The index is created and ready to configure.
This operation requires:
- Access to Kibana
- Permission to create or manage the index
-
write
indices privilege for the.elastic-connectors
index -
manage_api_key
cluster privilege to allow the API key to be created -
write_connector_secrets
cluster privilege to allow the API key to be stored as a secret
Configure connector
editCreate a new index to be managed by the connector.
Continue from above, or navigate to the following location within the Kibana UI:
Search > Content > Elasticsearch indices
Choose the index to configure, and then choose the Configuration tab.
Configure the connector:
- Edit the name and description for the connector. Your team can use this information to differentiate this index from other connector indices. (These fields describe the connector and are independent of the Elasticsearch index name.)
- Save your changes.
- Edit the data source configuration. The fields here vary by connector. Refer to the documentation for each connector for details (see list of native connectors, above). See Security for security considerations.
- Save your changes.
Optionally choose Edit sync schedule to begin managing the connector.
This operation requires:
- Access to Kibana
- Permission to create or manage the index
-
write
indices privilege for the.elastic-connectors
index -
manage_api_key
cluster privilege to allow the API key to be created -
write_connector_secrets
cluster privilege to allow the API key to be stored as a secret
Manage connector
editTo manage documents, syncs, sync rules, ingest pipelines, and other connector features, see Using connectors.
Manage API keys
editAPI keys for native connectors were introduced in Elastic version 8.13.0. Native connectors created in earlier versions will not automatically use API keys upon upgrading to 8.13.0. See enabling API keys for more information.
Native connectors communicate with Elasticsearch using API keys.
When managing a native connector through the Connectors page in the Kibana UI, API key management is handled automatically.
API keys for native connectors have the manage
permission for three indices:
- The attached index
- The access control (ACL) index used for document level security
-
The internal
.elastic-connectors
index.
Changing the attached index through Kibana will automatically invalidate the existing API key and generate a new one. If you want to rotate an existing API key, navigate to the Configuration tab. Scroll down to Manage API key and select Generate API key. This action will invalidate the previous API key, create a new API key, and update the connector secret.
API keys for native connectors are stored on an internal system index called .connector-secrets
.
This index can only be written to through API requests by users with the write_connector-secrets
cluster privilege.
Only the Enterprise Search instance has permission to read from this index.
Users managing native connectors will need the write_connector_secrets
cluster privilege assigned to their role.
See Security for security considerations.
Manage API keys programmatically
editYou can also create and store API keys programmatically.
Expand the following section for details.
The user needs the cluster privileges manage_api_key
and write_connector_secrets
to generate API keys programmatically.
To create an API key for the connector:
-
Run the following command, replacing values where indicated. Note the
id
andencoded
return values from the response:POST /_security/api_key { "name": "<connector_name>-connector-api-key", "role_descriptors": { "<connector_name>-connector-role": { "cluster": [ "monitor" ], "indices": [ { "names": [ "<index_name>", ".search-acl-filter-<index_name>", ".elastic-connectors*" ], "privileges": [ "all" ], "allow_restricted_indices": false } ] } } }
-
Use the
encoded
value to store a connector secret, and note theid
return value from this response:POST _connector/_secret { "value": <encoded_api_key> }
-
Use the API key
id
and the connector secretid
to update the connector:PUT /_connector/<connector_id>/_api_key_id { "api_key_id": "<API key id>", "api_key_secret_id": "<secret id>" }
Enabling API keys for upgraded connectors
editNative connectors created before 8.13.0 do not initially have API keys upon upgrading. The attached indices of these connectors cannot be changed until the connector has been converted to use an API key.
Expand the following section for steps on enabling API keys for upgraded native connectors.
-
Run the following command in Dev Tools to enable API keys for the connector, replacing values where indicated.
POST .elastic-connectors/_update/<connector_id> { "doc": { "features": { "native_connector_api_keys": { "enabled": true } } } }
- Go back to the Connectors page and navigate to the Configuration tab.
- Scroll down to Manage API key and select Generate API key.
Your native connector is now using API keys to authorize ingestion into Elasticsearch.
Usage via API
editIn 8.12 we introduced a set of Connector APIs to create and manage Elastic connectors and sync jobs, along with a CLI tool. Use these tools if you’d like to work with connectors and sync jobs programmatically.
End-to-end example
editThe following example demonstrates how to use a native connector: Native connector tutorial (MongoDB).
Convert a native connector
editYou can convert a native connector to a self-managed connector client to be run on your own infrastructure. You’ll find instructions in the UI on the connector index’s overview page.
Converting a native connector to a self-managed connector client is an irreversible operation!