Update connector filtering API
editUpdate connector filtering API
editThis functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.
Updates the draft filtering
configuration of a connector and marks the draft validation state as edited
. The filtering draft is activated once validated by the running Elastic connector service.
The filtering property is used to configure sync rules (both basic and advanced) for a connector. Learn more in the Sync rules.
To get started with Connector APIs, check out our tutorial.
Request
editPUT _connector/<connector_id>/_filtering
Prerequisites
edit- To sync data using self-managed connectors, you need to deploy the Elastic connector service. on your own infrastructure. This service runs automatically on Elastic Cloud for Elastic managed connectors.
-
The
connector_id
parameter should reference an existing connector. -
Filtering draft is activated once validated by the running Elastic connector service, the
draft.validation.state
must bevalid
. -
If, after a validation attempt, the
draft.validation.state
equals toinvalid
, inspectdraft.validation.errors
and fix any issues.
Path parameters
edit-
<connector_id>
- (Required, string)
Request body
edit-
rules
-
(Optional, array of objects) An array of basic sync rules, each with the following sub-attributes:
-
id
(Required, string)
A unique identifier for the rule. -
policy
(Required, string)
Specifies the policy, such asinclude
orexclude
. -
field
(Required, string)
The field in the document to which this rule applies. -
rule
(Required, string)
The type of rule, such asregex
,starts_with
,ends_with
,contains
,equals
,<
,>
, etc. -
value
(Required, string)
The value to be used in conjunction with the rule for matching the contents of the document’s field. -
order
(Required, number)
The order in which the rules are applied. The first rule to match has its policy applied. -
created_at
(Optional, datetime)
The timestamp when the rule was added. Defaults tonow
UTC timestamp. -
updated_at
(Optional, datetime)
The timestamp when the rule was last edited. Defaults tonow
UTC timestamp.
-
-
advanced_snippet
-
(Optional, object) Used for advanced filtering at query time, with the following sub-attributes:
-
value
(Required, object or array)
A JSON object/array passed directly to the connector for advanced filtering. -
created_at
(Optional, datetime)
The timestamp when this JSON object was created. Defaults tonow
UTC timestamp. -
updated_at
(Optional, datetime)
The timestamp when this JSON object was last edited. Defaults tonow
UTC timestamp.
-
Response codes
edit-
200
- Connector draft filtering was successfully updated.
-
400
-
The
connector_id
was not provided or the request payload was malformed. -
404
(Missing resources) -
No connector matching
connector_id
could be found.
Examples
editThe following example updates the draft basic sync rules for a Google Drive connector with ID my-g-drive-connector
. All Google Drive files with .txt
extension will be skipped:
resp = client.connector.update_filtering( connector_id="my-g-drive-connector", rules=[ { "field": "file_extension", "id": "exclude-txt-files", "order": 0, "policy": "exclude", "rule": "equals", "value": "txt" }, { "field": "_", "id": "DEFAULT", "order": 1, "policy": "include", "rule": "regex", "value": ".*" } ], ) print(resp)
const response = await client.connector.updateFiltering({ connector_id: "my-g-drive-connector", rules: [ { field: "file_extension", id: "exclude-txt-files", order: 0, policy: "exclude", rule: "equals", value: "txt", }, { field: "_", id: "DEFAULT", order: 1, policy: "include", rule: "regex", value: ".*", }, ], }); console.log(response);
PUT _connector/my-g-drive-connector/_filtering { "rules": [ { "field": "file_extension", "id": "exclude-txt-files", "order": 0, "policy": "exclude", "rule": "equals", "value": "txt" }, { "field": "_", "id": "DEFAULT", "order": 1, "policy": "include", "rule": "regex", "value": ".*" } ] }
{ "result": "updated" }
The following example updates the draft advanced sync rules for a MySQL connector with id my-sql-connector
. Advanced sync rules are specific to each connector type. Refer to the references for connectors that support advanced sync rules for syntax and examples.
resp = client.connector.update_filtering( connector_id="my-sql-connector", advanced_snippet={ "value": [ { "tables": [ "users", "orders" ], "query": "SELECT users.id AS id, orders.order_id AS order_id FROM users JOIN orders ON users.id = orders.user_id" } ] }, ) print(resp)
const response = await client.connector.updateFiltering({ connector_id: "my-sql-connector", advanced_snippet: { value: [ { tables: ["users", "orders"], query: "SELECT users.id AS id, orders.order_id AS order_id FROM users JOIN orders ON users.id = orders.user_id", }, ], }, }); console.log(response);
PUT _connector/my-sql-connector/_filtering { "advanced_snippet": { "value": [{ "tables": [ "users", "orders" ], "query": "SELECT users.id AS id, orders.order_id AS order_id FROM users JOIN orders ON users.id = orders.user_id" }] } }
{ "result": "updated" }
Note, you can also update draft rules
and advanced_snippet
in a single request.
Once the draft is updated, its validation state is set to edited
. The connector service will then validate the rules and report the validation state as either invalid
or valid
. If the state is valid
, the draft filtering will be activated by the running Elastic connector service.