For new users, we recommend using our Elasticsearch-native tools, rather than the standalone Workplace Search product. See this blog post for more information about upgrading your internal knowledge search, to make it an amazing experience for your users!
« Content Sources Overview Connecting custom sources »
Elastic Docs › ›Content Sources Overview

Custom connector packages

edit

Custom connector packages

edit

This feature is in technical preview. Features in technical preview are subject to change and are not covered by the service level agreement (SLA) of features that have reached general availability (GA).

From the beginning, Elastic Workplace Search has provided content sources with integrated connectors for popular content services such as Dropbox, GitHub, Google Drive, and many more. These connectors automatically pull data from these services, according to a schedule. From this data, they extract and index content to optimize it for search.

Although these connectors provide some flexibility through the Workplace Search API and the Kibana UI, they trade off additional flexibility to be as easy as possible to use. Users are limited to the features of these connectors and to the set of content services to which they connect.

To offer more flexibility, Workplace Search has always provided custom sources. A custom source does not provide a connector. It is essentially an index into which you can push content that you extract on your own. Users have complete flexibility but they must write their own code to manage scheduling, extracting, and indexing.

With version 8.2, Workplace Search introduced connector packages. A connector package pairs a custom source with a connector that you run on your own infrastructure. Because the code is open, users can modify the connector for greater flexibility. At the same time, the solution provides many conveniences of a connector, such as automatic scheduling, extraction, and indexing. However, users are still limited to the set of content services for which Elastic provides these connector packages.

With version 8.3, Workplace Search introduced custom connector packages (in technical preview). Using a software development kit (SDK) for Ruby, users can create their own connector packages, enabling connections to any content service. The SDK provides a partial connector implementation. Complete the implementation to create a connector that uses the familiar Workplace Search model for scheduling, extraction, and indexing.

Before using this feature, review the known issues and limitations and learn how to get help and provide feedback. When you’re ready to get started, see Creating a custom connector package.

Known issues and limitations

edit

Custom connector packages are available now in technical preview. This preview enables you to experiment with this feature and evaluate it for your use case. While in preview, there are several ways to get help and provide feedback.

The following connector packages provided by Elastic were built using the same SDK provided for custom connector packages:

Each Elastic deployment can connect only one connector using this SDK. Therefore, when using a custom connector package, you cannot use any of the above connector packages.

(Elastic also has connector packages implemented in Python, such as SharePoint Server and network drives. These packages may still be used when using the SDK for Ruby.)

The elastic/connectors GitHub repository is under active development. When developing a custom connector package for Workplace Search, use the 8.3 branch of the repository. All links within this documentation point to the 8.3 branch.

Getting help and providing feedback

edit

While using this preview feature, use any of the following resources to request help and provide feedback:

Creating a custom connector package

edit

The following sections provide enough information to get started. As you develop, there are several ways to get help and provide feedback.

The process to create a custom connector package is described in the following steps:

  1. Set up and learn your development environment
  2. Develop and test the connector
  3. Move to production
  4. (Optionally) share your connector package

Additionally, the following Elastic blog post provides a walkthrough for creating a specific connector (GitLab): You have more ways to bake this cake: Bring your own Enterprise Search connector. The blog post goes into much more detail than this documentation, but its examples are limited to one content service. The following documentation provides examples for multiple content services.

Set up and learn your development environment

edit

The elastic/connectors GitHub repository (8.3 branch) contains the connector SDK for Ruby and example connector implementations. To create your own connector package, fork the repository and refer to the following documentation for connector developers:

These docs explain how to set up and confirm your development environment to ensure you can run the connectors server. After you have this application running, you can develop your own connector and use this application to run your connector.

In short, you should complete the following steps to confirm your development environment:

  1. Fork the repository
  2. Install dependencies
  3. Generate an API key
  4. Create the connectors configuration file
  5. Start the connectors application server
  6. Send an HTTP request to the currently running connector
  7. Stop the connectors application server

You may also want to complete the following to get more familiar with the environment:

  • Run the connectors test suite
  • Open a Ruby console
  • Review all make targets in the Makefile

You will also need access to the content service you are targeting (example: GitLab) and you’ll need an Elastic deployment. While developing a connector, we recommend running your Elastic deployment using Docker. See Running Enterprise Search using Docker in the Enterprise Search documentation.

Configure the following Enterprise Search settings to bypass SSL during local development:

Develop and test the connector

edit

After you are familiar with the development environment, you can begin developing your connector. The elastic/connectors GitHub repository (8.3 branch) contains several complete connector implementations and code shared by all connectors developed with this SDK. The shared code acts as a partial connector implementation. You will need to complete your connector using the other connector implementations as examples.

At a high level, develop your connector using the following workflow:

  1. Set your custom connector as the current connector within the connectors configuration file
  2. Set any additional configuration required for your connector in the connectors configuration file
  3. Register your connector with the SDK
  4. Iteratively implement the connector interface in Ruby (see below)
  5. Iteratively test the HTTP responses from the connector (see below)
  6. Connect your Enterprise Search deployment to the connector and test the integration

Additionally, the following Elastic blog post provides a walkthrough for creating a specific connector (GitLab): You have more ways to bake this cake: Bring your own Enterprise Search connector. The blog post goes into much more detail than this documentation, but its examples are limited to one content service. The following documentation provides examples for multiple content services.

To implement the connector interface in Ruby, review the existing connector implementations and shared code. Refer to the following examples:

The SDK:

Inclues a connector registry:

That registers connectors:

Each of which may use an authorization implementation:

And a configuration:

And a custom client:

And an extractor:

Each extractor may make use of an adapter:

All of the above rely on shared code, some of which you may want to use in your own code. The shared code includes:

Async behavior (jobs):

And library code, such as utilities, constants, and error classes:

And additional files left over from the extraction of these connectors from Enterprise Search:

You may also want to review the connectors server implementation:

Which implements the following HTTP operations:

GET  /
POST /status
POST /start_sync
POST /documents
POST /download
POST /deleted
POST /permissions
POST /oauth2/init
POST /oauth2/exchange
POST /oauth2/refresh
POST /secrets/compare

While developing, iteratively test your connector’s HTTP responses to these operations.

After your connector is implemented, you can connect your Enterprise Search deployment to the connector to test the integration. Refer to the repository’s README for details.

If needed, several debugging libraries are installed by default. Refer to the Developers guide for details.

Move to production

edit

You can consider development complete after you’ve confirmed the integration between the content service, the connector, and Enterprise Search. At this point, you are ready to deploy your connector to a production environment and integrate it with the production content service and production Elastic deployment.

This documentation does not describe how to run a Ruby application in production. Choose a Ruby hosting provider and consult their deployment documentation.

Remember to remove all development configuration settings or update them to production values.

After deploying the connector, test the integration between the three production services.

(Optionally) share your connector package

edit

We encourage you to share your connector with others. You can distribute your connector as a Ruby gem, and you can contribute your connector to the upstream elastic/connectors repository. Refer to the following guides within the repository:

« Content Sources Overview Connecting custom sources »