Ingestion overview
editIngestion overview
editIngestion is the critical first step in building search into your application or website. On the Elastic platform, ingestion is all about getting your data into Elasticsearch.
The "Create index" UI in Enterprise Search guides you through creating an Elasticsearch index that is optimized for search use cases.
When you create an index using the UI, you are prompted to choose an ingestion method for adding documents to your index.
Enterprise Search provides 3 ingestion methods:
When using the web crawler or connectors, your Elasticsearch index is managed by Enterprise Search. You don’t need to modify the documents yourself.
When using the API, you (or your application) are in control of managing the documents.
Ingest pipelines are another important ingestion tool. They enable you to transform your data at ingestion time, before it gets indexed into Elasticsearch.
Elastic web crawler
editUse the Elastic web crawler to programmatically discover, extract, and index searchable content from websites and knowledge bases. The web crawler is available out of the box on Elastic Cloud, and can be managed entirely through the Kibana UI. Get started indexing data into Elasticsearch in just a few clicks.
Elastic connectors
editA connector is an Elastic integration that syncs data from an original data source to an Elastic deployment. Enterprise Search provides connectors for a range of popular third-party sources, such as MongoDB, MySQL, and S3 to name a few.
These connectors come in two flavors:
- Native connectors: Fully managed connectors, available out of the box on Elastic Cloud.
- Connector clients: Self-managed open source connectors run on your own infrastructure. Customize connectors to your needs, use new connectors before they are available natively within Elastic Cloud, and build your own connectors for custom data sources.
Workplace Search connectors
editWorkplace Search connectors are available directly within your Elastic deployment, through the Workplace Search product. No additional infrastructure is required.
Workplace Search connector packages are open-code connector implementations you can deploy to your own infrastructure.
Ingestion APIs
editIn Kibana, the Use the API ingestion method enables you to programmatically add documents to your search-optimized Elasticsearch index. A number of APIs can be used to ingest data into Elasticsearch.
Refer to the Ingestion APIs overview for full details. Learn about using Elasticsearch APIs for optimized ingestion, and check out the official Elastic clients available for many popular programming languages.
Ingest pipelines
editUse ingest pipelines to transform your data before it is indexed into Elasticsearch. Refer to Ingest pipelines for an overview of ingest pipelines in Enterprise Search.
Machine learning document enrichment is powered by ingest pipelines. If you want to use ML to transform your Elasticsearch documents, it’s important to understand how ingest pipelines and ingest processors work.