« PostgreSQL self-managed connector tutorial MongoDB managed connector tutorial »
Elastic Docs ›Elasticsearch Guide [8.16] ›Ingest content with Elastic connectors

Elastic managed connectors

edit

Elastic managed connectors

edit

Naming history

Elastic managed connectors were initially known as "native connectors". You might find this term in older documentation.

Managed Elastic connectors are available directly within your Elastic Cloud deployment. No additional infrastructure is required.

Managed connectors sync data sources directly to Elasticsearch indices. Create these indices using the Connector workflow within Kibana.

The following connectors are available as Elastic managed connectors. Refer to each connector reference for additional information specific to each connector.

Availability and prerequisites

edit

Managed connectors were introduced in Elastic version 8.5.0.

Your Elastic Cloud deployment must include the following Elastic services:

Refer to Native Integrations on the Elastic subscriptions page, in the Elastic Search section for managed connector licensing requirements.

Elastic Cloud IPs

Using our Elastic managed connectors involves outbound data transfer (egress) from your Elastic Cloud deployment. If you have IP/firewall rules on your third party service, you’ll need to add the Elastic Cloud egress static IP ranges to your service’s configuration.

Refer to static IP ranges in the Elastic Cloud documentation for the latest list of IP addresses.

Usage in Kibana UI

edit

Follow the Connector workflow in Kibana to select the Connector ingestion method. Choose a data source, create an Elasticsearch index, and configure a managed connector to manage the index.

use a connector workflow

Select a connector

edit

Choose the data source to sync from the available options and select Continue.

Create index

edit

Create a new index to be managed by the connector:

  1. Name your index and optionally change the language analyzer to match the human language of your data source. (The index name will be automatically prefixed with search-.)
  2. Select Create index.

The index is created and ready to configure.

This operation requires:

  1. Access to Kibana
  2. Permission to create or manage the index
  3. write indices privilege for the .elastic-connectors index
  4. manage_api_key cluster privilege to allow the API key to be created
  5. write_connector_secrets cluster privilege to allow the API key to be stored as a secret

Configure connector

edit

Create a new index to be managed by the connector.

Continue from above, or navigate to the following location within the Kibana UI:

Search > Content > Elasticsearch indices from the main menu, or use the global search field.

Choose the index to configure, and then choose the Configuration tab.

Configure the connector:

  1. Edit the name and description for the connector. Your team can use this information to differentiate this index from other connector indices. (These fields describe the connector and are independent of the Elasticsearch index name.)
  2. Save your changes.
  3. Edit the data source configuration. The fields here vary by connector. Refer to the documentation for each connector for details (refer to list of Elastic managed connectors, above). Refer to Security for security considerations.
  4. Save your changes.

Optionally choose Edit sync schedule to begin managing the connector.

This operation requires:

  1. Access to Kibana
  2. Permission to create or manage the index
  3. write indices privilege for the .elastic-connectors index
  4. manage_api_key cluster privilege to allow the API key to be created
  5. write_connector_secrets cluster privilege to allow the API key to be stored as a secret

Manage connector

edit

To manage documents, syncs, sync rules, ingest pipelines, and other connector features, refer to Connectors UI in Kibana.

Manage API keys

edit

API keys for Elastic managed connectors were introduced in Elastic version 8.13.0. Managed connectors created in earlier versions will not automatically use API keys upon upgrading to 8.13.0. Refer to enabling API keys for more information.

Managed connectors communicate with Elasticsearch using API keys. When managing a managed connector through the Connectors page in the Kibana UI, API key management is handled automatically. API keys for Elastic managed connectors have the manage permission for three indices:

  1. The attached index
  2. The access control (ACL) index used for document level security
  3. The internal .elastic-connectors index.

Changing the attached index through Kibana will automatically invalidate the existing API key and generate a new one. If you want to rotate an existing API key, navigate to the Configuration tab. Scroll down to Manage API key and select Generate API key. This action will invalidate the previous API key, create a new API key, and update the connector secret.

API keys for Elastic managed connectors are stored on an internal system index called .connector-secrets. This index can only be written to through API requests by users with the write_connector-secrets cluster privilege. Only the Enterprise Search instance has permission to read from this index.

Users managing Elastic managed connectors will need the write_connector_secrets cluster privilege assigned to their role. Refer to Security for security considerations.

Manage API keys programmatically

edit

You can also create and store API keys programmatically.

Expand the following section for details.

The user needs the cluster privileges manage_api_key, manage_connector and write_connector_secrets to generate API keys programmatically.

To create an API key for the connector:

  1. Run the following command, replacing values where indicated. Note the id and encoded return values from the response:

    resp = client.security.create_api_key(
    name="my-connector-api-key",
    role_descriptors={
        "my-connector-connector-role": {
            "cluster": [
                "monitor",
                "manage_connector"
            ],
            "indices": [
                {
                    "names": [
                        "my-index_name",
                        ".search-acl-filter-my-index_name",
                        ".elastic-connectors*"
                    ],
                    "privileges": [
                        "all"
                    ],
                    "allow_restricted_indices": False
                }
            ]
        }
    },
)
print(resp)
    const response = await client.security.createApiKey({
  name: "my-connector-api-key",
  role_descriptors: {
    "my-connector-connector-role": {
      cluster: ["monitor", "manage_connector"],
      indices: [
        {
          names: [
            "my-index_name",
            ".search-acl-filter-my-index_name",
            ".elastic-connectors*",
          ],
          privileges: ["all"],
          allow_restricted_indices: false,
        },
      ],
    },
  },
});
console.log(response);
    POST /_security/api_key
{
  "name": "my-connector-api-key",
  "role_descriptors": {
    "my-connector-connector-role": {
      "cluster": [
        "monitor",
        "manage_connector"
      ],
      "indices": [
        {
          "names": [
            "my-index_name",
            ".search-acl-filter-my-index_name",
            ".elastic-connectors*"
          ],
          "privileges": [
            "all"
          ],
          "allow_restricted_indices": false
        }
      ]
    }
  }
}

  2. Use the encoded value to store a connector secret, and note the id return value from this response:

    resp = client.connector.secret_post(
    body={
        "value": "encoded_api_key"
    },
)
print(resp)
    const response = await client.connector.secretPost({
  body: {
    value: "encoded_api_key",
  },
});
console.log(response);
    POST _connector/_secret
{
  "value": "encoded_api_key"
}

  3. Use the API key id and the connector secret id to update the connector:

    resp = client.connector.update_api_key_id(
    connector_id="my_connector_id>",
    api_key_id="API key_id",
    api_key_secret_id="secret_id",
)
print(resp)
    const response = await client.connector.updateApiKeyId({
  connector_id: "my_connector_id>",
  api_key_id: "API key_id",
  api_key_secret_id: "secret_id",
});
console.log(response);
    PUT /_connector/my_connector_id>/_api_key_id
{
  "api_key_id": "API key_id",
  "api_key_secret_id": "secret_id"
}

Enabling API keys for upgraded connectors

edit

Managed connectors created before 8.13.0 do not initially have API keys upon upgrading. The attached indices of these connectors cannot be changed until the connector has been converted to use an API key.

Expand the following section for steps on enabling API keys for upgraded Elastic managed connectors.

  1. Run the following command in Dev Tools to enable API keys for the connector, replacing values where indicated.

    resp = client.update(
    index=".elastic-connectors",
    id="connector_id",
    doc={
        "features": {
            "native_connector_api_keys": {
                "enabled": True
            }
        }
    },
)
print(resp)
    const response = await client.update({
  index: ".elastic-connectors",
  id: "connector_id",
  doc: {
    features: {
      native_connector_api_keys: {
        enabled: true,
      },
    },
  },
});
console.log(response);
    POST .elastic-connectors/_update/connector_id
{
  "doc": {
    "features": {
      "native_connector_api_keys": {
        "enabled": true
      }
    }
  }
}
  2. Go back to the Connectors page and navigate to the Configuration tab.
  3. Scroll down to Manage API key and select Generate API key.

Your managed connector is now using API keys to authorize ingestion into Elasticsearch.

Usage via API

edit

In 8.12 we introduced a set of Connector APIs to create and manage Elastic connectors and sync jobs, along with a CLI tool. Use these tools if you’d like to work with connectors and sync jobs programmatically.

End-to-end example

edit

The following example demonstrates how to use a managed connector on Elastic Cloud: Managed connector tutorial (MongoDB).

Convert a managed connector

edit

You can convert a managed connector to a self-managed connector to be run on your own infrastructure. You’ll find instructions in the UI on the connector index’s overview page.

Converting a managed connector to a self-managed connector is an irreversible operation!

« PostgreSQL self-managed connector tutorial MongoDB managed connector tutorial »