Connector clients

A connector is a type of Elastic integration that syncs data from an original data source to Elasticsearch. Each connector extracts the original files, records, or objects; and transforms them into documents within Elasticsearch.

Connector clients are self-managed connectors that you run on your own infrastructure. These connectors are written in Python and the source code is available in the elastic/connectors repo.

Connector clients are available for the following third-party data sources:

Because connector clients are self-managed on your own infrastructure, they run outside of your Elasticsearch Serverless project.

You can run them from source or in a Docker container.

The first decision you need to make before deploying a connector is which third party service (data source) you want to sync to Elasticsearch. See the list of available connectors.

Note that each data source will have specific prerequisites you’ll need to meet to authorize the connector to access its data. For example, certain data sources may require you to create an OAuth application, or create a service account. You’ll need to check the individual connector documentation for these details.

In your project’s UI, go to Elasticsearch → Connectors. Follow these steps:

To use connector clients, you must deploy the connector service so your connector can talk to your Elasticsearch instance. The source code is hosted in the elastic/connectors repository.

You have two deployment options:

You can deploy connector clients using Docker. Follow these instructions.

Step 1: Download sample configuration file

You can either download the configuration file manually or run the following command:

curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output </absolute/path/to>/connectors-config/config.yml

Change the --output argument value to the path where you want to save the configuration file.

Step 2: Update the configuration file for your self-managed connector

Your configuration file should look like this:

elasticsearch.host: <ELASTICSEARCH_ENDPOINT>
elasticsearch.api_key: <ELASTICSEARCH_API_KEY>

connectors:
  -
    connector_id: <CONNECTOR_ID_FROM_UI>
    service_type: <SERVICE-NAME> # sharepoint_online (example)
    api_key: <CONNECTOR_API_KEY> # Optional. If not provided, the connector will use the elasticsearch.api_key instead

Step 3: Run the Docker image

Use the following command, substituting values where necessary:

docker run \
-v "</absolute/path/to>/connectors-config:/config" \ # NOTE: change absolute path to match where config.yml is located on your machine
--tty \
--rm \
docker.elastic.co/enterprise-search/elastic-connectors:{version}.0 \
/app/bin/elastic-ingest \
-c /config/config.yml # Path to your configuration file in the container

Find all available Docker images in the official Elastic Docker registry.

Running from source requires cloning the repository and running the code locally. Use this approach if you’re actively customizing connectors.

Follow these steps:

Your configuration file should look like this:

elasticsearch.host: <ELASTICSEARCH_ENDPOINT>
elasticsearch.api_key: <ELASTICSEARCH_API_KEY>

connectors:
  -
    connector_id: <CONNECTOR_ID_FROM_UI>
    service_type: <SERVICE-NAME> # sharepoint_online (example)
    api_key: <CONNECTOR_API_KEY> # Optional. If not provided, the connector will use the elasticsearch.api_key instead

Run the connector service

Once you’ve configured the connector code, you can run the connector service.

In your terminal or IDE:

The connector service should now be running in your terminal. If the connection to your Elasticsearch instance was successful, the Configure your connector step will be activated in the project’s UI.

Here we’re working locally. In a production setup, you’ll deploy the connector service to your own infrastructure.

Once the connector service is running, it’s time to head back to the UI to finalize the connector configuration. You should now see the Configure your connector step in your project’s UI.

In this step, you need to add the specific connection details about your data source instance, like URL, authorization credentials, etc. These details will vary based on the third-party data source you’re connecting to.

For example, the Sharepoint Online connector requires the following details about your Sharepoint instance:

Once you’ve entered the data source details, you need to connect to an index. This is the final step in your project’s UI, before you can run a sync.

You can choose to sync to an existing Elasticsearch index, or create a new index for your connector. You can also create an API key that is limited to only access your selected index.

Once this step is completed, you’re ready to run a sync. When a sync is launched you’ll start to see documents being added to your Elasticsearch index.

Learn how syncing works in the elastic/connectors repo docs.