IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Native connectors Workplace Search connectors »
Elastic Docs › ›Elastic connectors

Connector clients and frameworks

edit

Connector clients and frameworks

edit

Connector clients are open-code connector implementations you can deploy to your own infrastructure. Connector clients allow you to customize connectors to satisfy your needs, or use new connectors before they are available natively within Elastic Cloud.

Connector clients sync data sources directly to Elasticsearch indices. Create these indices using the Build a connector workflow within Kibana. See Connector clients and frameworks.

The following connectors are available as connector clients:

Connector clients are built using connector frameworks. Use these frameworks to deploy existing connector clients, or create your own clients to sync from custom data sources.

The following connector frameworks are available:

  • Python (branch 8.7, compatible with Elastic 8.7)
  • Ruby (branch 8.7, compatible with Elastic 8.7)

Availability and prerequisites

edit

Connector clients and frameworks were introduced in Elastic version 8.4.0.

Refer to the individual connectors references for connector-specific compatibility information.

The connector frameworks are beta features and are subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.

Your Elastic deployment must include the following Elastic services:

  • Elasticsearch
  • Kibana

  • Enterprise Search with at least 4GB RAM per zone

    Connectors are not available to Elastic Cloud trial deployments. For trial deployments, the Enterprise Search service is limited to 2GB RAM.

(A new Elastic Cloud deployment includes these services by default, however, you must increase the RAM for the Enterprise Search service. See Infrastructure requirements.)

You must additionally deploy and operate one or more connector services with native mode set to false. See Connector service.

Usage

edit

Use the Build a connector workflow to create an Elasticsearch index and configure a connector client to manage the index.

build a connector workflow

Create index

edit

Create a new index to be managed by the connector.

Within Kibana, visit:

Enterprise Search > Content > Elasticsearch indices

Create a new connector index:

  1. Choose Create new index.
  2. Choose Build a connector.
  3. Provide the connector name and optionally change the language analyzer to match the human language of your data source. (The name you provide is automatically prefixed with search-.)
  4. Save your changes.

The index is created and ready to configure.

This operation requires access to Kibana and the write indices privilege for the .elastic-connectors index.

Configure connector

edit

Configure the connector to manage the index’s documents.

Continue from above, or navigate to the following location within Kibana:

Enterprise Search > Content > Elasticsearch indices

Choose the index to configure, and then choose the Configuration tab.

Configure the connector:

  1. Choose Generate API key. An API key is printed to the screen.
  2. Edit the name and description for the connector. Your team can use this information to differentiate this index from other connector indices. (These fields describe the connector and are independent of the Elasticsearch index name.)
  3. Save your changes.
  4. Locate the API key and connector ID, which are both printed to the screen. Use these values to configure the connector service you are running within your own infrastructure. Refer to the documentation for your connector service.
  5. If necessary, choose Recheck now to display additional configuration fields for the connector. Edit these fields, referring to the connector’s reference documentation as needed.
  6. Save your changes.

Optionally choose Edit sync schedule to begin managing the connector.

This operation requires access to Kibana and the write indices privilege for the .elastic-connectors index.

Manage connector

edit

To manage documents, syncs, sync rules, ingest pipelines, and other connector features, see Usage.

These processes are identical for connector clients and native connectors.

Example

edit

The following example demonstrates how to use a native connector: PostgreSQL connector client tutorial.

Connector service

edit

To use connector clients, you must deploy and operate one or more connector services. Connector services are available through the following connector frameworks:

  • Python (branch 8.7, compatible with Elastic 8.7)
  • Ruby (branch 8.7, compatible with Elastic 8.7)

Refer to this guide in the Python framework repository for an example of how to deploy a connector service with Docker.

For each connector you’d like to use, review its reference documentation to determine which connector service is required.

Deploy and operate that service with native mode set to false to use it as a connector client.

See the frameworks listed above for connector service documentation.

Connector testing

edit

The connector framework enables you to run end-to-end (E2E) tests on your connector clients, against a real data source.

To avoid tampering with a real Elasticsearch instance, E2E tests run an isolated Elasticsearch instance in Docker. Configuration values are set in your docker-compose.yml file. Docker Compose manages the setup of the development environment, including both the mock Elastic instance and mock data source.

E2E tests use default configuration values for the connector.

Example

edit

To execute a functional test for the Amazon S3 connector, run the following command:

$ make ftest NAME=s3

By default, this will use a medium-sized dataset. For faster tests add the DATA_SIZE=small flag:

$ make ftest NAME=s3 DATA_SIZE=small
« Native connectors Workplace Search connectors »