Update connector filtering API

This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.

Updates the draft filtering configuration of a connector and marks the draft validation state as edited. The filtering draft is activated once validated by the running Elastic connector service.

The filtering property is used to configure sync rules (both basic and advanced) for a connector. Learn more in the Sync rules.

To get started with Connector APIs, check out our tutorial.

Request

edit

PUT _connector/<connector_id>/_filtering

Prerequisites

edit

To sync data using self-managed connectors, you need to deploy the Elastic connector service. on your own infrastructure. This service runs automatically on Elastic Cloud for Elastic managed connectors.
The connector_id parameter should reference an existing connector.
Filtering draft is activated once validated by the running Elastic connector service, the draft.validation.state must be valid.
If, after a validation attempt, the draft.validation.state equals to invalid, inspect draft.validation.errors and fix any issues.

Path parameters

edit

<connector_id>: (Required, string)

Request body

edit

rules

(Optional, array of objects) An array of basic sync rules, each with the following sub-attributes:

id (Required, string)
A unique identifier for the rule.
policy (Required, string)
Specifies the policy, such as include or exclude.
field (Required, string)
The field in the document to which this rule applies.
rule (Required, string)
The type of rule, such as regex, starts_with, ends_with, contains, equals, <, >, etc.
value (Required, string)
The value to be used in conjunction with the rule for matching the contents of the document’s field.
order (Required, number)
The order in which the rules are applied. The first rule to match has its policy applied.
created_at (Optional, datetime)
The timestamp when the rule was added. Defaults to now UTC timestamp.
updated_at (Optional, datetime)
The timestamp when the rule was last edited. Defaults to now UTC timestamp.

advanced_snippet

(Optional, object) Used for advanced filtering at query time, with the following sub-attributes:

value (Required, object or array)
A JSON object/array passed directly to the connector for advanced filtering.
created_at (Optional, datetime)
The timestamp when this JSON object was created. Defaults to now UTC timestamp.
updated_at (Optional, datetime)
The timestamp when this JSON object was last edited. Defaults to now UTC timestamp.

Response codes

edit

200: Connector draft filtering was successfully updated.
400: The connector_id was not provided or the request payload was malformed.
404 (Missing resources): No connector matching connector_id could be found.

Examples

edit

The following example updates the draft basic sync rules for a Google Drive connector with ID my-g-drive-connector. All Google Drive files with .txt extension will be skipped:

resp = client.connector.update_filtering(
    connector_id="my-g-drive-connector",
    rules=[
        {
            "field": "file_extension",
            "id": "exclude-txt-files",
            "order": 0,
            "policy": "exclude",
            "rule": "equals",
            "value": "txt"
        },
        {
            "field": "_",
            "id": "DEFAULT",
            "order": 1,
            "policy": "include",
            "rule": "regex",
            "value": ".*"
        }
    ],
)
print(resp)

const response = await client.connector.updateFiltering({
  connector_id: "my-g-drive-connector",
  rules: [
    {
      field: "file_extension",
      id: "exclude-txt-files",
      order: 0,
      policy: "exclude",
      rule: "equals",
      value: "txt",
    },
    {
      field: "_",
      id: "DEFAULT",
      order: 1,
      policy: "include",
      rule: "regex",
      value: ".*",
    },
  ],
});
console.log(response);

PUT _connector/my-g-drive-connector/_filtering
{
    "rules": [
         {
            "field": "file_extension",
            "id": "exclude-txt-files",
            "order": 0,
            "policy": "exclude",
            "rule": "equals",
            "value": "txt"
        },
        {
            "field": "_",
            "id": "DEFAULT",
            "order": 1,
            "policy": "include",
            "rule": "regex",
            "value": ".*"
        }
    ]
}

{
    "result": "updated"
}

The following example updates the draft advanced sync rules for a MySQL connector with id my-sql-connector. Advanced sync rules are specific to each connector type. Refer to the references for connectors that support advanced sync rules for syntax and examples.

resp = client.connector.update_filtering(
    connector_id="my-sql-connector",
    advanced_snippet={
        "value": [
            {
                "tables": [
                    "users",
                    "orders"
                ],
                "query": "SELECT users.id AS id, orders.order_id AS order_id FROM users JOIN orders ON users.id = orders.user_id"
            }
        ]
    },
)
print(resp)

const response = await client.connector.updateFiltering({
  connector_id: "my-sql-connector",
  advanced_snippet: {
    value: [
      {
        tables: ["users", "orders"],
        query:
          "SELECT users.id AS id, orders.order_id AS order_id FROM users JOIN orders ON users.id = orders.user_id",
      },
    ],
  },
});
console.log(response);

PUT _connector/my-sql-connector/_filtering
{
    "advanced_snippet": {
        "value": [{
            "tables": [
                "users",
                "orders"
            ],
            "query": "SELECT users.id AS id, orders.order_id AS order_id FROM users JOIN orders ON users.id = orders.user_id"
        }]
    }
}

{
    "result": "updated"
}

Note, you can also update draft rules and advanced_snippet in a single request.

Once the draft is updated, its validation state is set to edited. The connector service will then validate the rules and report the validation state as either invalid or valid. If the state is valid, the draft filtering will be activated by the running Elastic connector service.

« Update connector features API Update connector name and description API »