Elastic managed connectors
editElastic managed connectors
editManaged Elastic connectors are available directly within your Elastic Cloud deployment. No additional infrastructure is required.
Managed connectors sync data sources directly to Elasticsearch indices. Create these indices using the Connector workflow within Kibana.
The following connectors are available as Elastic managed connectors. Refer to each connector reference for additional information specific to each connector.
- Azure Blob Storage
- Box
- Confluence (including Confluence Data Center)
- Dropbox
- GitHub
- Gmail
- Google Cloud Storage
- Google Drive
- Jira (including Jira Data Center)
- Microsoft SQL
- MongoDB
- MySQL
- Network Drive
- Notion
- OneDrive
- Oracle
- Outlook
- PostgreSQL
- Amazon S3
- Salesforce
- ServiceNow
- SharePoint Online
- SharePoint Server
- Slack
- Teams
- Zoom
Availability and prerequisites
editManaged connectors were introduced in Elastic version 8.5.0.
Your Elastic Cloud deployment must include the following Elastic services:
- Elasticsearch
- Kibana
- Enterprise Search
Refer to Native Integrations on the Elastic subscriptions page, in the Elastic Search section for managed connector licensing requirements.
Usage in Kibana UI
editFollow the Connector workflow in Kibana to select the Connector ingestion method. Choose a data source, create an Elasticsearch index, and configure a managed connector to manage the index.
Select a connector
editChoose the data source to sync from the available options and select Continue.
Create index
editCreate a new index to be managed by the connector:
-
Name your index and optionally change the language analyzer to match the human language of your data source.
(The index name will be automatically prefixed with
search-
.) - Select Create index.
The index is created and ready to configure.
This operation requires:
- Access to Kibana
- Permission to create or manage the index
-
write
indices privilege for the.elastic-connectors
index -
manage_api_key
cluster privilege to allow the API key to be created -
write_connector_secrets
cluster privilege to allow the API key to be stored as a secret
Configure connector
editCreate a new index to be managed by the connector.
Continue from above, or navigate to the following location within the Kibana UI:
Search > Content > Elasticsearch indices
Choose the index to configure, and then choose the Configuration tab.
Configure the connector:
- Edit the name and description for the connector. Your team can use this information to differentiate this index from other connector indices. (These fields describe the connector and are independent of the Elasticsearch index name.)
- Save your changes.
- Edit the data source configuration. The fields here vary by connector. Refer to the documentation for each connector for details (refer to list of Elastic managed connectors, above). Refer to Security for security considerations.
- Save your changes.
Optionally choose Edit sync schedule to begin managing the connector.
This operation requires:
- Access to Kibana
- Permission to create or manage the index
-
write
indices privilege for the.elastic-connectors
index -
manage_api_key
cluster privilege to allow the API key to be created -
write_connector_secrets
cluster privilege to allow the API key to be stored as a secret
Manage connector
editTo manage documents, syncs, sync rules, ingest pipelines, and other connector features, refer to Connectors UI in Kibana.
Manage API keys
editAPI keys for Elastic managed connectors were introduced in Elastic version 8.13.0. Managed connectors created in earlier versions will not automatically use API keys upon upgrading to 8.13.0. Refer to enabling API keys for more information.
Managed connectors communicate with Elasticsearch using API keys.
When managing a managed connector through the Connectors page in the Kibana UI, API key management is handled automatically.
API keys for Elastic managed connectors have the manage
permission for three indices:
- The attached index
- The access control (ACL) index used for document level security
-
The internal
.elastic-connectors
index.
Changing the attached index through Kibana will automatically invalidate the existing API key and generate a new one. If you want to rotate an existing API key, navigate to the Configuration tab. Scroll down to Manage API key and select Generate API key. This action will invalidate the previous API key, create a new API key, and update the connector secret.
API keys for Elastic managed connectors are stored on an internal system index called .connector-secrets
.
This index can only be written to through API requests by users with the write_connector-secrets
cluster privilege.
Only the Enterprise Search instance has permission to read from this index.
Users managing Elastic managed connectors will need the write_connector_secrets
cluster privilege assigned to their role.
Refer to Security for security considerations.
Manage API keys programmatically
editYou can also create and store API keys programmatically.
Expand the following section for details.
The user needs the cluster privileges manage_api_key
, manage_connector
and write_connector_secrets
to generate API keys programmatically.
To create an API key for the connector:
-
Run the following command, replacing values where indicated. Note the
id
andencoded
return values from the response:const response = await client.security.createApiKey({ name: "my-connector-api-key", role_descriptors: { "my-connector-connector-role": { cluster: ["monitor", "manage_connector"], indices: [ { names: [ "my-index_name", ".search-acl-filter-my-index_name", ".elastic-connectors*", ], privileges: ["all"], allow_restricted_indices: false, }, ], }, }, }); console.log(response);
POST /_security/api_key { "name": "my-connector-api-key", "role_descriptors": { "my-connector-connector-role": { "cluster": [ "monitor", "manage_connector" ], "indices": [ { "names": [ "my-index_name", ".search-acl-filter-my-index_name", ".elastic-connectors*" ], "privileges": [ "all" ], "allow_restricted_indices": false } ] } } }
-
Use the
encoded
value to store a connector secret, and note theid
return value from this response:const response = await client.connector.secretPost({ body: { value: "encoded_api_key", }, }); console.log(response);
POST _connector/_secret { "value": "encoded_api_key" }
-
Use the API key
id
and the connector secretid
to update the connector:const response = await client.connector.updateApiKeyId({ connector_id: "my_connector_id>", api_key_id: "API key_id", api_key_secret_id: "secret_id", }); console.log(response);
PUT /_connector/my_connector_id>/_api_key_id { "api_key_id": "API key_id", "api_key_secret_id": "secret_id" }
Enabling API keys for upgraded connectors
editManaged connectors created before 8.13.0 do not initially have API keys upon upgrading. The attached indices of these connectors cannot be changed until the connector has been converted to use an API key.
Expand the following section for steps on enabling API keys for upgraded Elastic managed connectors.
-
Run the following command in Dev Tools to enable API keys for the connector, replacing values where indicated.
const response = await client.update({ index: ".elastic-connectors", id: "connector_id", doc: { features: { native_connector_api_keys: { enabled: true, }, }, }, }); console.log(response);
POST .elastic-connectors/_update/connector_id { "doc": { "features": { "native_connector_api_keys": { "enabled": true } } } }
- Go back to the Connectors page and navigate to the Configuration tab.
- Scroll down to Manage API key and select Generate API key.
Your managed connector is now using API keys to authorize ingestion into Elasticsearch.
Usage via API
editIn 8.12 we introduced a set of Connector APIs to create and manage Elastic connectors and sync jobs, along with a CLI tool. Use these tools if you’d like to work with connectors and sync jobs programmatically.
End-to-end example
editThe following example demonstrates how to use a managed connector on Elastic Cloud: Managed connector tutorial (MongoDB).
Convert a managed connector
editYou can convert a managed connector to a self-managed connector to be run on your own infrastructure. You’ll find instructions in the UI on the connector index’s overview page.
Converting a managed connector to a self-managed connector is an irreversible operation!