Elastic SharePoint Online connector reference

This connector is available as a managed connector in Elastic versions 8.9.0 and later. To use this connector natively in Elastic Cloud, satisfy all managed connector requirements.

To use this connector as a managed connector, see Elastic managed connectors.

For additional operations, see Connectors UI in Kibana.

Before you can configure the connector, you must create an OAuth App in the SharePoint Online platform. Your connector will authenticate to SharePoint as the registered OAuth application/client. You’ll collect values (client ID, tenant ID, and client secret) during this process that you’ll need for the configuration step in Kibana.

To get started, first log in to SharePoint Online and access your administrative dashboard. Ensure you are logged in as the Azure Portal service account.

Follow these steps:

Refer to the following documentation for setting SharePoint permissions.

This connector is compatible with SharePoint Online.

Use the following configuration fields to set up the connector:

The connector syncs the following SharePoint object types:

Basic sync rules are identical for all connectors and are available by default. For more information read Types of sync rule.

The following section describes advanced sync rules for this connector. Advanced sync rules are defined through a source-specific DSL JSON snippet.

Advanced rules for the Sharepoint Online connector enable you to avoid extracting and syncing older data that might no longer be relevant for search.

Example:

{
	"skipExtractingDriveItemsOlderThan": 60
}

This rule will not extract content of any drive items (files in document libraries) that haven’t been modified for 60 days or more.

Limitations of sync rules with incremental syncs

Changing sync rules after Sharepoint Online content has already been indexed can bring unexpected results, when using incremental syncs.

Incremental syncs ensure updates from 3rd-party system, but do not modify existing documents in the index.

To avoid these issues, run a full sync after changing sync rules (basic or advanced).

Let’s take a look at several examples where incremental syncs might lead to inconsistent data on your index.

Example: Restrictive basic sync rule added after a full sync

Imagine your Sharepoint Online drive contains the following drive items:

/Documents/Report.doc
/Documents/Spreadsheet.xls
/Presentations/Q4-2020-Report.pdf
/Presentations/Q4-2020-Report-Data.xls
/Personal/Documents/Sales.xls

After a sync, all these drive items will be stored on your Elasticsearch index. Let’s add a basic sync rule, filtering files by their path:

Exclude WHERE path CONTAINS "Documents"

These filtering rules will exclude all files with "Documents" in their path, leaving only files in /Presentations directory:

/Presentations/Q4-2020-Report.pdf
/Presentations/Q4-2020-Report-Data.xls

If no files were changed, incremental sync will not receive information about changes from Sharepoint Online and won’t be able to delete any files, leaving the index in the same state it was before the sync.

After a full sync, the index will be updated and files that are excluded by sync rules will be removed.

Example: Restrictive basic sync rules removed after a full sync

Imagine that Sharepoint Online drive has the following drive items:

/Documents/Report.doc
/Documents/Spreadsheet.xls
/Presentations/Q4-2020-Report.pdf
/Presentations/Q4-2020-Report-Data.xls
/Personal/Documents/Sales.xls

Before doing a sync, we add a restrictive basic filtering rule:

Exclude WHERE path CONTAINS "Documents"

After a full sync, the index will contain only files in the /Presentations directory:

/Presentations/Q4-2020-Report.pdf
/Presentations/Q4-2020-Report-Data.xls

Afterwards, we can remove the filtering rule and run an incremental sync. If no changes happened to the files, incremental sync will not mirror these changes in the Elasticsearch index, because Sharepoint Online will not report any changes to the items. Only a full sync will include the items previously ignored by the sync rule.

Example: Advanced sync rules edge case

Advanced sync rules can be applied to limit which documents will have content extracted. For example, it’s possible to set a rule so that documents older than 180 days won’t have content extracted.

However, there is an edge case. Imagine a document that is 179 days old and its content is extracted and indexed into Elasticsearch. After 2 days, this document will be 181 days old. Since this document was already ingested it will not be modified. Therefore, the content will not be removed from the index, following an incremental sync.

In this situation, if you want older documents to be removed, you will need to clean the index up manually. For example, you can manually run an Elasticsearch query that removes drive item content older than 180 days:

resp = client.update_by_query(
    index="INDEX_NAME",
    conflicts="proceed",
    query={
        "bool": {
            "filter": [
                {
                    "match": {
                        "object_type": "drive_item"
                    }
                },
                {
                    "exists": {
                        "field": "file"
                    }
                },
                {
                    "range": {
                        "lastModifiedDateTime": {
                            "lte": "now-180d"
                        }
                    }
                }
            ]
        }
    },
    script={
        "source": "ctx._source.body = ''",
        "lang": "painless"
    },
)
print(resp)

const response = await client.updateByQuery({
  index: "INDEX_NAME",
  conflicts: "proceed",
  query: {
    bool: {
      filter: [
        {
          match: {
            object_type: "drive_item",
          },
        },
        {
          exists: {
            field: "file",
          },
        },
        {
          range: {
            lastModifiedDateTime: {
              lte: "now-180d",
            },
          },
        },
      ],
    },
  },
  script: {
    source: "ctx._source.body = ''",
    lang: "painless",
  },
});
console.log(response);

POST INDEX_NAME/_update_by_query?conflicts=proceed
{
  "query": {
    "bool": {
      "filter": [
        {
          "match": {
            "object_type": "drive_item"
          }
        },
        {
          "exists": {
            "field": "file"
          }
        },
        {
          "range": {
            "lastModifiedDateTime": {
              "lte": "now-180d"
            }
          }
        }
      ]
    }
  },
  "script": {
    "source": "ctx._source.body = ''",
    "lang": "painless"
  }
}

Document-level security (DLS) enables you to restrict access to documents based on a user’s permissions. This feature is available by default for this connector.

Refer to configuration on this page for how to enable DLS for this connector.

The default content extraction service is powered by the Enterprise Search default ingest pipeline. (See Ingest pipelines for Search indices.)

See Content extraction.

The SharePoint Online self-managed connector supports large file content extraction (> 100MB). This requires:

Refer to local content extraction for more information.

Refer to Known issues for a list of known issues for all connectors.

See Troubleshooting.

See Security.

This connector is available as a self-managed self-managed connector. To use this connector as a self-managed connector, satisfy all self-managed connector requirements.

To use this connector as a self-managed connector, see Self-managed connectors For additional operations, see Connectors UI in Kibana.

Before you can configure the connector, you must create an OAuth App in the SharePoint Online platform. Your connector will authenticate to SharePoint as the registered OAuth application/client. You’ll collect values (client ID, tenant ID, and client secret) during this process that you’ll need for the configuration step in Kibana.

To get started, first log in to SharePoint Online and access your administrative dashboard. Ensure you are logged in as the Azure Portal service account.

Follow these steps:

Refer to the following documentation for setting SharePoint permissions.

This connector is compatible with SharePoint Online.

Use the following configuration fields to set up the connector:

You can deploy the SharePoint Online connector as a self-managed connector using Docker. Follow these instructions.

curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml

# When connecting to your cloud deployment you should edit the host value
elasticsearch.host: http://host.docker.internal:9200
elasticsearch.api_key: <ELASTICSEARCH_API_KEY>

connectors:
  -
    connector_id: <CONNECTOR_ID_FROM_KIBANA>
    service_type: sharepoint_online
    api_key: <CONNECTOR_API_KEY_FROM_KIBANA> # Optional. If not provided, the connector will use the elasticsearch.api_key instead

docker run \
-v ~/connectors-config:/config \
--network "elastic" \
--tty \
--rm \
docker.elastic.co/integrations/elastic-connectors:8.16.2.0 \
/app/bin/elastic-ingest \
-c /config/config.yml

Refer to DOCKER.md in the elastic/connectors repo for more details.

Find all available Docker images in the official registry.

The connector syncs the following SharePoint object types:

Basic sync rules are identical for all connectors and are available by default. For more information read Types of sync rule.

The following section describes advanced sync rules for this connector. Advanced sync rules are defined through a source-specific DSL JSON snippet.

Advanced rules for the Sharepoint Online connector enable you to avoid extracting and syncing older data that might no longer be relevant for search.

Example:

{
	"skipExtractingDriveItemsOlderThan": 60
}

This rule will not extract content of any drive items (files in document libraries) that haven’t been modified for 60 days or more.

Limitations of sync rules with incremental syncs

Changing sync rules after Sharepoint Online content has already been indexed can bring unexpected results, when using incremental syncs.

Incremental syncs ensure updates from 3rd-party system, but do not modify existing documents in the index.

To avoid these issues, run a full sync after changing sync rules (basic or advanced).

Let’s take a look at several examples where incremental syncs might lead to inconsistent data on your index.

Example: Restrictive basic sync rule added after a full sync

Imagine your Sharepoint Online drive contains the following drive items:

/Documents/Report.doc
/Documents/Spreadsheet.xls
/Presentations/Q4-2020-Report.pdf
/Presentations/Q4-2020-Report-Data.xls
/Personal/Documents/Sales.xls

After a sync, all these drive items will be stored on your Elasticsearch index. Let’s add a basic sync rule, filtering files by their path:

Exclude WHERE path CONTAINS "Documents"

These filtering rules will exclude all files with "Documents" in their path, leaving only files in /Presentations directory:

/Presentations/Q4-2020-Report.pdf
/Presentations/Q4-2020-Report-Data.xls

If no files were changed, incremental sync will not receive information about changes from Sharepoint Online and won’t be able to delete any files, leaving the index in the same state it was before the sync.

After a full sync, the index will be updated and files that are excluded by sync rules will be removed.

Example: Restrictive basic sync rules removed after a full sync

Imagine that Sharepoint Online drive has the following drive items:

/Documents/Report.doc
/Documents/Spreadsheet.xls
/Presentations/Q4-2020-Report.pdf
/Presentations/Q4-2020-Report-Data.xls
/Personal/Documents/Sales.xls

Before doing a sync, we add a restrictive basic filtering rule:

Exclude WHERE path CONTAINS "Documents"

After a full sync, the index will contain only files in the /Presentations directory:

/Presentations/Q4-2020-Report.pdf
/Presentations/Q4-2020-Report-Data.xls

Afterwards, we can remove the filtering rule and run an incremental sync. If no changes happened to the files, incremental sync will not mirror these changes in the Elasticsearch index, because Sharepoint Online will not report any changes to the items. Only a full sync will include the items previously ignored by the sync rule.

Example: Advanced sync rules edge case

Advanced sync rules can be applied to limit which documents will have content extracted. For example, it’s possible to set a rule so that documents older than 180 days won’t have content extracted.

However, there is an edge case. Imagine a document that is 179 days old and its content is extracted and indexed into Elasticsearch. After 2 days, this document will be 181 days old. Since this document was already ingested it will not be modified. Therefore, the content will not be removed from the index, following an incremental sync.

In this situation, if you want older documents to be removed, you will need to clean the index up manually. For example, you can manually run an Elasticsearch query that removes drive item content older than 180 days:

resp = client.update_by_query(
    index="INDEX_NAME",
    conflicts="proceed",
    query={
        "bool": {
            "filter": [
                {
                    "match": {
                        "object_type": "drive_item"
                    }
                },
                {
                    "exists": {
                        "field": "file"
                    }
                },
                {
                    "range": {
                        "lastModifiedDateTime": {
                            "lte": "now-180d"
                        }
                    }
                }
            ]
        }
    },
    script={
        "source": "ctx._source.body = ''",
        "lang": "painless"
    },
)
print(resp)

const response = await client.updateByQuery({
  index: "INDEX_NAME",
  conflicts: "proceed",
  query: {
    bool: {
      filter: [
        {
          match: {
            object_type: "drive_item",
          },
        },
        {
          exists: {
            field: "file",
          },
        },
        {
          range: {
            lastModifiedDateTime: {
              lte: "now-180d",
            },
          },
        },
      ],
    },
  },
  script: {
    source: "ctx._source.body = ''",
    lang: "painless",
  },
});
console.log(response);

POST INDEX_NAME/_update_by_query?conflicts=proceed
{
  "query": {
    "bool": {
      "filter": [
        {
          "match": {
            "object_type": "drive_item"
          }
        },
        {
          "exists": {
            "field": "file"
          }
        },
        {
          "range": {
            "lastModifiedDateTime": {
              "lte": "now-180d"
            }
          }
        }
      ]
    }
  },
  "script": {
    "source": "ctx._source.body = ''",
    "lang": "painless"
  }
}

Document-level security (DLS) enables you to restrict access to documents based on a user’s permissions. This feature is available by default for this connector.

Refer to configuration on this page for how to enable DLS for this connector.

The default content extraction service is powered by the Enterprise Search default ingest pipeline. (See Ingest pipelines for Search indices.)

See Content extraction.

The SharePoint Online self-managed connector supports large file content extraction (> 100MB). This requires:

Refer to local content extraction for more information.

The connector framework enables operators to run functional tests against a real data source. Refer to Connector testing for more details.

To perform E2E testing for the SharePoint Online connector, run the following command:

$ make ftest NAME=sharepoint_online

For faster tests, add the DATA_SIZE=small flag:

make ftest NAME=sharepoint_online DATA_SIZE=small

Refer to Known issues for a list of known issues for all connectors.

See Troubleshooting.

See Security.