Native connectors
editNative connectors
editNative connectors are a collection of connectors that are available through Elastic Cloud. Use native connectors to integrate with data sources using only the Kibana UI—no development required.
View Available connectors and Availability and prerequisites to evaluate native connectors for your use case.
See Usage for a complete reference to using these connectors, and review Troubleshooting if you encounter issues.
Available connectors
editThe following native connectors are available:
Availability and prerequisites
editNative connectors were introduced in Elastic version 8.5.0.
Currently, all native connectors are in technical preview. Features in technical preview are subject to change and are not covered by the service level agreement (SLA) of features that have reached general availability (GA).
Native connectors are available to all Elastic Cloud deployments. Your deployment must include the Elasticsearch, Kibana, and Enterprise Search services. Your Enterprise Search service should have 4+ GB RAM per zone to avoid out of memory errors. See Infrastructure requirements to learn how to verify and change the RAM for your Enterprise Search service.
Native connectors are also available to self-managed deployments if the subscription requirements are satisfied. View the requirements for this feature under the Elastic Enterprise Search section of the Elastic Stack subscriptions page.
Self-managed deployments must additionally include a connector service, running in native mode. Connector services are available in multiple programming languages. Refer to the following repositories for documentation:
Usage
editKibana provides an interface to manage all connector indices. From there, you can manage existing connectors and set up additional connectors.
For each connector you’d like to use, complete the following:
- Gather the information about the data source you will need to configure the index, such as host and access credentials. Refer to each connector’s reference documentation for details.
- Create a new connector index.
- Configure the index with information about the data source.
- Manage recurring syncs or sync once to begin syncing data to the index and manage ongoing updates.
- View the index status and details and synced documents.
- Optionally use ingest pipelines to post-process or transform document fields.
If you encounter issues, see Troubleshooting for help.
All native connectors share several prerequisites.
Each native connector has its own reference, which may include additional prerequisites or usage instructions.
Manage connector indices
editWithin Kibana, navigate to:
Enterprise Search → Content → Elasticsearch indices
View a list of indices and their attributes, including index health and ingestion status. Use the Ingestion method field to identify which indices are associated with connectors.
Within this interface, you can choose to view the details for each existing index or delete an index. Or, you can create a new index.
These operations require access to Kibana and additional index privileges.
Create index
editContinue from above, or navigate to the following breadcrumb within Kibana:
Enterprise Search → Content → Elasticsearch indices
Create a new connector index:
- Choose Create new index.
-
Provide the connector name and optionally change the language analyzer to match the human language of your data source.
(The name you provide is automatically prefixed with
search-
.) - Save your changes.
- When prompted, choose from the available connectors.
- Save your changes.
The index is created and ready to configure.
This operation requires access to Kibana and the write
indices privilege for the .elastic-connectors
index.
Configure index
editContinue from above, or navigate to the following breadcrumb within Kibana:
Enterprise Search → Content → Elasticsearch indices
Choose the index to configure, and then choose the Configuration tab.
Configure the index:
- Edit the name and description for the connector. Your team can use this information to differentiate this index from other connector indices. (These fields describe the connector and are independent of the Elasticsearch index name.)
- Save your changes.
- Edit the data source configuration. The fields here vary by connector. Refer to the documentation for each connector for details.
- Save your changes.
- Optionally choose Edit sync schedule to move to the Scheduling tab to manage recurring syncs.
This operation requires access to Kibana and the write
indices privilege for the .elastic-connectors
index.
See Security for security details.
Manage recurring syncs
editContinue from above, or navigate to the following breadcrumb within Kibana:
Enterprise Search → Content → Elasticsearch indices
Choose the index to configure, and then choose the Scheduling tab.
Within this interface, you can enable or disable recurring syncs. When enabled, you can additionally manage the sync schedule.
This operation requires access to Kibana and the write
indices privilege for the .elastic-connectors
index.
Alternatively, you can sync once.
After you enable recurring syncs or sync once, the first sync will begin. (There may be a short delay before the connector service begins the first sync.) You may want to view the index details to see the status or errors, or view the synced documents.
Sync once
editAfter a connector index is configured, you can request a sync at any time.
Within Kibana, navigate to:
Enterprise Search → Content → Elasticsearch indices
Then choose the index to sync.
Regardless of which tab is active, the Sync button is always visible in the top right. Choose this button to request a sync. (There may be a short delay before the connector service begins the sync.)
This operation requires access to Kibana and the write
indices privilege for the .elastic-connectors
index.
View status
editView the index details to see a variety of information that communicate the status of the index and connector.
Within Kibana, navigate to:
Enterprise Search → Content → Elasticsearch indices
Then choose the index to view.
The Overview tab presents a variety of information, including:
- Any errors affecting the connector or sync process.
- The current ingestion status.
- The current document count.
- Recent sync history, including sync status.
This operation requires access to Kibana and the read
indices privilege for the .elastic-connectors
index.
View documents
editView the documents the connector has synced from the data source. Additionally view the index mappings to determine the current document schema.
Within Kibana, navigate to:
Enterprise Search → Content → Elasticsearch indices
Then choose the index to view.
Choose the Documents tab to view the synced documents. Choose the Index Mappings tab to view the index mappings that were created by the connector.
When setting up a new connector, ensure you are getting the documents and fields you were expecting from the data source. If not, see Troubleshooting for help.
These operations require access to Kibana and the read
and manage
indices privileges for the index containing the documents.
See Security for security details.
Manage ingest pipelines
editEach connector determines how it extracts data from its data source and transforms that data into Elasticsearch documents. However, you can use Ingest pipelines to perform additional processing and transformation on each document, before it is written to Elasticsearch.
Each connector index has a default ingest pipeline, which you can customize or replace through Kibana.
Within Kibana, navigate to:
Enterprise Search → Content → Elasticsearch indices
Then choose the index to manage and choose the Pipelines tab.
This functionality applies to all search indices, including crawler and API indices. Refer to the following documentation for details: Ingest pipelines.
Troubleshooting
editUse the following actions to help diagnose and resolve connector issues:
Security
editElastic Cloud deployments have strong security defaults. For example, data is encrypted by default, whether at rest or in transit.
Self-managed deployments require more upfront work to ensure strong security. Refer to Secure the Elastic Stack in the Elasticsearch documentation for more information.
Access to credentials
editCredentials for the data source — such as API keys or username/password pair— are stored in your deployment’s .elastic-connectors
Elasticsearch index.
Therefore, the credentials are visible to all Elastic users with the read
indices privilege for that index.
By default, the following Elastic users have this privilege: the elastic
superuser and the kibana_system
user.
Enterprise Search service account tokens can also read the .elastic-connectors
index.
Access to documents
editData synced from your data source are stored as documents in the Elasticsearch index you created.
This data is visible to all Elastic users with the read
indices privilege for that index.
Be careful to ensure that access to this index is at least as restrictive as access to the original data source.
Encryption
editElastic Cloud automatically encrypts data at rest.
Data in transit is automatically encrypted using https
.
Self-managed deployments must implement encryption at rest. See Configure security for the Elastic Stack in the Elasticsearch documentation for more information.
Request help or support
editElastic provides several resources for help, and your Elastic subscription may include Elastic support for this feature.
Use the following resources to request help from the Elastic community:
Use the following resources to request Elastic support: