IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Native connectors Workplace Search connectors »
Elastic Docs › ›Elastic connectors

Connector clients and frameworks

edit

Connector clients and frameworks

edit

Connector clients are open-code connector implementations you can deploy to your own infrastructure. Connector clients allow you to customize connectors to satisfy your needs, or use new connectors before they are available natively within Elastic Cloud.

Connector clients sync data sources directly to Elasticsearch indices. Create these indices using the Connector workflow within Kibana. See Connector clients and frameworks.

The following connectors are available as connector clients:

Connector clients are built using the Python connector framework (branch 8.8, compatible with Elastic 8.8). Use this framework to deploy existing connector clients, or create your own clients to sync from custom data sources.

Availability and prerequisites

edit

Connector clients and frameworks were introduced in Elastic version 8.4.0.

Refer to the individual connectors references for connector-specific compatibility information.

Your Elastic deployment must include the following Elastic services:

  • Elasticsearch
  • Kibana
  • Enterprise Search

(A new Elastic Cloud deployment includes these services by default.)

You must additionally deploy and operate one or more connector services with native mode set to false. See Connector service.

Usage

edit

Follow the Connector workflow in Kibana to select the Connector ingestion method. Choose a data source, create an Elasticsearch index, and configure a connector client to manage the index.

build a connector workflow

Select a connector

edit

Choose the data source to sync from the available options and select Continue.

Create index

edit

Create a new index to be managed by the connector:

  1. Provide the connector name and optionally change the language analyzer to match the human language of your data source. (The index name will be automatically prefixed with search-.)
  2. Select Create index.

The index is created and ready to configure.

This operation requires access to Kibana and the write indices privilege for the .elastic-connectors index.

Configure connector

edit

Configure the connector to manage the index’s documents.

Continue from above, or navigate to the following location within Kibana:

Enterprise Search > Content > Elasticsearch indices

Choose the index to configure, and then choose the Configuration tab.

Configure the connector:

  1. Choose Generate API key. An API key is printed to the screen.
  2. Edit the name and description for the connector. Your team can use this information to differentiate this index from other connector indices. (These fields describe the connector and are independent of the Elasticsearch index name.)
  3. Save your changes.
  4. Locate the API key and connector ID, which are both printed to the screen. Use these values to configure the connector service you are running on your own infrastructure. Refer to the documentation for your connector service.
  5. If necessary, choose Recheck now to display additional configuration fields for the connector. Edit these fields, referring to the connector’s reference documentation as needed.
  6. Save your changes.

Optionally choose Edit sync schedule to begin managing the connector.

This operation requires access to Kibana and the write indices privilege for the .elastic-connectors index.

Manage connector

edit

To manage documents, syncs, sync rules, ingest pipelines, and other connector features, see Usage.

These processes are identical for connector clients and native connectors.

Example

edit

The following example demonstrates how to deploy the connector service and a connector client on your on infrastructure: PostgreSQL connector client tutorial.

Connector service

edit

To use connector clients, you must deploy and operate the connectors-python connector service. The connector service is available through the following connector framework:

  • Python (branch 8.8, compatible with Elastic 8.8)

Each connector reference includes a section called Deployment using Docker with instructions on how to deploy the connector service and connector client using Docker.

Deploy and operate the connector service with native mode set to false to use it as a connector client.

Connector service documentation is available in the connectors-python repository linked above.

Connector testing

edit

The connector framework enables you to run end-to-end (E2E) tests on your connector clients, against a real data source.

To avoid tampering with a real Elasticsearch instance, E2E tests run an isolated Elasticsearch instance in Docker. Configuration values are set in your docker-compose.yml file. Docker Compose manages the setup of the development environment, including both the mock Elastic instance and mock data source.

E2E tests use default configuration values for the connector.

Example

edit

To execute a functional test for the Amazon S3 connector, run the following command:

$ make ftest NAME=s3

By default, this will use a medium-sized dataset. For faster tests add the DATA_SIZE=small flag:

$ make ftest NAME=s3 DATA_SIZE=small
« Native connectors Workplace Search connectors »