Elastic MongoDB connector reference

This connector is available as a managed connector in Elastic versions 8.5.0 and later.

To use this connector natively in Elastic Cloud, satisfy all managed connector requirements.

This connector is compatible with MongoDB Atlas and MongoDB 3.6 and later.

The data source and your Elastic deployment must be able to communicate with each other over a network.

Each time you create an index to be managed by this connector, you will create a new connector configuration. You will need some or all of the following information about the data source.

To create a new MongoDB connector:

For additional operations, see Connectors UI in Kibana.

You can use the Elasticsearch Create connector API to create a new native MongoDB connector.

For example:

resp = client.connector.put(
    connector_id="my-{service-name-stub}-connector",
    index_name="my-elasticsearch-index",
    name="Content synced from {service-name}",
    service_type="{service-name-stub}",
    is_native=True,
)
print(resp)

PUT _connector/my-mongodb-connector
{
  "index_name": "my-elasticsearch-index",
  "name": "Content synced from MongoDB",
  "service_type": "mongodb",
  "is_native": true
}

Refer to the Elasticsearch API documentation for details of all available Connector APIs.

To use this connector as a managed connector, use the Connector workflow. See Elastic managed connectors.

For additional operations, see Connectors UI in Kibana.

An example is available for this connector. See Managed connector tutorial (MongoDB).

It’s not possible to use expressions like new Date() inside an aggregation pipeline. These expressions won’t be evaluated by the underlying MongoDB client, but will be passed as a string to the MongoDB instance. A possible workaround is to use aggregation variables.

Incorrect (new Date() will be interpreted as string):

{
    "aggregate": {
        "pipeline": [
            {
                "$match": {
                  "expiresAt": {
                    "$gte": "new Date()"
                  }
                }
            }
        ]
    }
}

Correct (usage of $$NOW):

{
  "aggregate": {
    "pipeline": [
      {
        "$addFields": {
          "current_date": {
            "$toDate": "$$NOW"
          }
        }
      },
      {
        "$match": {
          "$expr": {
            "$gte": [
              "$expiresAt",
              "$current_date"
            ]
          }
        }
      }
    ]
  }
}

Currently, the MongoDB connector does not support working with self-signed or custom CA certs when connecting to your self-managed MongoDB host.

This can be worked around in development environments, by appending certain query parameters to the configured host.

For example, if your host is mongodb+srv://my.mongo.host.com, appending ?tls=true&tlsAllowInvalidCertificates=true will allow disabling TLS certificate verification.

The full host in this example will look like this:

mongodb+srv://my.mongo.host.com/?tls=true&tlsAllowInvalidCertificates=true

See Known issues for any issues affecting all connectors.

See Troubleshooting.

See Security.

The following describes the default syncing behavior for this connector. Use sync rules and ingest pipelines to customize syncing for specific indices.

All documents in the configured MongoDB database and collection are extracted and transformed into documents in your Elasticsearch index.

This results in Elasticsearch documents that closely match the original MongoDB documents.

The Elasticsearch mapping is created when the first document is created.

Each sync is a "full" sync. For each MongoDB document discovered:

This is recursive, because embedded documents can themselves contain embedded documents.

The following sections describe Sync rules for this connector.

Basic sync rules are identical for all connectors and are available by default.

Advanced rules for MongoDB can be used to express either find queries or aggregation pipelines. They can also be used to tune options available when issuing these queries/pipelines.

For find queries, the structure of this JSON DSL should look like:

{
	"find":{
		"filter": {
			// find query goes here
		},
		"options":{
			// query options go here
		}
	}
}

For example:

{
	"find": {
		"filter": {
			"$text": {
				"$search": "garden",
				"$caseSensitive": false
			}
		},
		"skip": 10,
		"limit": 1000
	}
}

find queries also support additional options, for example the projection object:

{
  "find": {
    "filter": {
      "languages": [
        "English"
      ],
      "runtime": {
        "$gt":90
      }
    },
    "projection":{
      "tomatoes": 1
    }
  }
}

Where the available options are:

Similarly, for aggregation pipelines, the structure of the JSON DSL should look like:

{
	"aggregate":{
		"pipeline": [
			// pipeline elements go here
		],
		"options": {
            // pipeline options go here
		}
    }
}

Where the available options are:

As part of the 8.8.0 release the MongoDB connector was moved from the Ruby connectors framework to the Elastic connector framework.

This change introduces minor formatting modifications to data ingested from MongoDB:

If your MongoDB connector stopped working after migrating from 8.7.x to 8.8.x, read the workaround outlined in Known issues. If that does not work, we recommend deleting the search index attached to this connector and re-creating a MongoDB connector from scratch.

This connector is also available as a self-managed connector from the Elastic connector framework. To use this connector as a self-managed connector, satisfy all self-managed connector requirements.

This connector is compatible with MongoDB Atlas and MongoDB 3.6 and later.

The data source and your Elastic deployment must be able to communicate with each other over a network.

The following configuration fields are required to set up the connector:

To create a new MongoDB connector:

You can use the Elasticsearch Create connector API to create a new self-managed MongoDB self-managed connector.

For example:

resp = client.connector.put(
    connector_id="my-{service-name-stub}-connector",
    index_name="my-elasticsearch-index",
    name="Content synced from {service-name}",
    service_type="{service-name-stub}",
)
print(resp)

PUT _connector/my-mongodb-connector
{
  "index_name": "my-elasticsearch-index",
  "name": "Content synced from MongoDB",
  "service_type": "mongodb"
}

Refer to the Elasticsearch API documentation for details of all available Connector APIs.

To use this connector as a self-managed connector, see Self-managed connectors For additional usage operations, see Connectors UI in Kibana.

An example is available for this connector. See Managed connector tutorial (MongoDB).

It’s not possible to use expressions like new Date() inside an aggregation pipeline. These expressions won’t be evaluated by the underlying MongoDB client, but will be passed as a string to the MongoDB instance. A possible workaround is to use aggregation variables.

Incorrect (new Date() will be interpreted as string):

{
    "aggregate": {
        "pipeline": [
            {
                "$match": {
                  "expiresAt": {
                    "$gte": "new Date()"
                  }
                }
            }
        ]
    }
}

Correct (usage of $$NOW):

{
  "aggregate": {
    "pipeline": [
      {
        "$addFields": {
          "current_date": {
            "$toDate": "$$NOW"
          }
        }
      },
      {
        "$match": {
          "$expr": {
            "$gte": [
              "$expiresAt",
              "$current_date"
            ]
          }
        }
      }
    ]
  }
}

Currently, the MongoDB connector does not support working with self-signed or custom CA certs when connecting to your self-managed MongoDB host.

This can be worked around in development environments, by appending certain query parameters to the configured host.

For example, if your host is mongodb+srv://my.mongo.host.com, appending ?tls=true&tlsAllowInvalidCertificates=true will allow disabling TLS certificate verification.

The full host in this example will look like this:

mongodb+srv://my.mongo.host.com/?tls=true&tlsAllowInvalidCertificates=true

A bug introduced in 8.12.0 causes the Connectors docker image to error out if run using MongoDB as its source. The command line will output the error cannot import name 'coroutine' from 'asyncio'. This issue is fixed in versions 8.12.2 and 8.13.0. This bug does not affect Elastic managed connectors.

See Known issues for any issues affecting all connectors.

See Troubleshooting.

See Security.

You can deploy the MongoDB connector as a self-managed connector using Docker. Follow these instructions.

curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml

# When connecting to your cloud deployment you should edit the host value
elasticsearch.host: http://host.docker.internal:9200
elasticsearch.api_key: <ELASTICSEARCH_API_KEY>

connectors:
  -
    connector_id: <CONNECTOR_ID_FROM_KIBANA>
    service_type: mongodb
    api_key: <CONNECTOR_API_KEY_FROM_KIBANA> # Optional. If not provided, the connector will use the elasticsearch.api_key instead

docker run \
-v ~/connectors-config:/config \
--network "elastic" \
--tty \
--rm \
docker.elastic.co/integrations/elastic-connectors:9.0.0-beta1.0 \
/app/bin/elastic-ingest \
-c /config/config.yml

Refer to DOCKER.md in the elastic/connectors repo for more details.

Find all available Docker images in the official registry.

The following describes the default syncing behavior for this connector. Use sync rules and ingest pipelines to customize syncing for specific indices.

All documents in the configured MongoDB database and collection are extracted and transformed into documents in your Elasticsearch index.

This results in Elasticsearch documents that closely match the original MongoDB documents.

The Elasticsearch mapping is created when the first document is created.

Each sync is a "full" sync. For each MongoDB document discovered:

This is recursive, because embedded documents can themselves contain embedded documents.

The following sections describe Sync rules for this connector.

Basic sync rules are identical for all connectors and are available by default.

Advanced rules for MongoDB can be used to express either find queries or aggregation pipelines. They can also be used to tune options available when issuing these queries/pipelines.

For find queries, the structure of this JSON DSL should look like:

{
	"find":{
		"filter": {
			// find query goes here
		},
		"options":{
			// query options go here
		}
	}
}

For example:

{
	"find": {
		"filter": {
			"$text": {
				"$search": "garden",
				"$caseSensitive": false
			}
		},
		"skip": 10,
		"limit": 1000
	}
}

find queries also support additional options, for example the projection object:

{
  "find": {
    "filter": {
      "languages": [
        "English"
      ],
      "runtime": {
        "$gt":90
      }
    },
    "projection":{
      "tomatoes": 1
    }
  }
}

Where the available options are:

Similarly, for aggregation pipelines, the structure of the JSON DSL should look like:

{
	"aggregate":{
		"pipeline": [
			// pipeline elements go here
		],
		"options": {
            // pipeline options go here
		}
    }
}

Where the available options are:

As part of the 8.8.0 release the MongoDB connector was moved from the Ruby connectors framework to the Elastic connector framework.

This change introduces minor formatting modifications to data ingested from MongoDB:

If your MongoDB connector stopped working after migrating from 8.7.x to 8.8.x, read the workaround outlined in Known issues. If that does not work, we recommend deleting the search index attached to this connector and re-creating a MongoDB connector from scratch.