Customize indexing for a content source

Indexing rules can be configured for each content source. You may be able to use custom indexing rules to speed up content source synchronization and decrease storage costs. You can also exclude content that shouldn’t be processed, such as binary files.

Indexing rules apply at the time of full source synchronization. This means that they will only take effect when the content from a source is re-synchronized. Changing indexing rules automatically triggers a synchronization process, but this process might take some time, especially when dealing with content sources that have large sets of documents.

Indexing rules are evaluated in order. Any object is evaluated against each rule in order, and the first rule it matches will be applied to it. This means that if the first rule excludes a file, that file will be excluded from indexing even if later rules include it. Similarly, if the first rule includes a file that file will be included even if later rules exclude it.

In this guide, you will see how to change the indexing rules on a content source. Here are some examples of sources that support custom indexing rules:

The examples in this guide use a Dropbox content source.

These steps assume you have already connected a source. Here we will modify the indexing rules on a Dropbox source to exclude content from a top-level folder named Legal.

Step 1. First, we need to locate the content source in Kibana, and open the Assets and objects page. From the administrative dashboard, navigate to Sources:

Figure 123. Content sources

From here, open the Dropbox content source by clicking on Details, then click on Synchronization in the sidebar and open the Assets and objects page.

Step 2. Next, we need to identify the filter rule we wish to use when indexing documents. If we assume our Dropbox documents exist at the following paths:

/employee-directory.pdf
/Documentation/enabling-saml-auth.pdf
/Documentation/customizing-content-sources.pdf
/Legal/clients-list.xls
/Legal/Contracts/business.docx

If we want to exclude every document contained in the top-level Legal folder, we can use a pattern like:

/Legal/**/*

This pattern is created using glob patterns.

Step 3. We can then use the table at the bottom of the page to build an indexing configuration to filter out our Legal documents. Click Add indexing rule, set the Policy to Exclude, the Rule to Path and the Value to /Legal/**/*. Click Save in the table to add the rule to the indexing configuration.

Note that the new indexing rule configuration won’t be saved until you click the Save button at the top of the page. You can do so now.

Step 4. Modifying a content source’s indexing configuration will trigger a full synchronization of the source’s data. When complete, we should see that no documents from the top-level Legal folder were indexed into Workplace Search.

Using the above steps, here are some additional examples of indexing configurations for other use cases.

Revert indexing configuration to default

To revert a source’s indexing configuration back to the default state, you can remove all rules. The default rule will always stay in place, as all files that don’t fit a rule are automatically included in indexing.

Include only specific sub-directories

For file system based sources, rather than excluding content, perhaps you only want to index content within a specific set of directories. Note that rules are evaluated in order, which allows a complex mixture of include and exclude rules. This example will index all content in the Engineering and Design top-level folders, however the Templates folder within Design will also be excluded:

Rules can easily be reordered by dragging them within the table. You will need to click Save at the top of the page to store your changes and trigger a full synchronization.

Exclude specific file extensions

This configuration will exclude some common executable file types:

Exclude specific object types

This configuration will exclude some specific object types. Which object types you can exclude or include will depend on your content source. This is particularly useful for content sources that sync in a variety of document types, like Salesforce:

The schedule that determines when data is synchronized to Workplace Search can be configured for each content source. You can configure a schedule that synchronizes less frequently, to lower the burden on a third-party server, or you can synchronize more frequently to ensure your data is up-to-date. You can also configure windows of time when synchronization should not be performed. This can allow you to perform operations such as scheduled maintenance in a third-party system that Workplace Search connects to.

curl \
--request "GET" \
--url "<ENTERPRISE_SEARCH_BASE_URL>/api/ws/v1/sources" \
--header "Authorization: Bearer ${TOKEN}"

{
  "meta": { ... },
  "results":
  [
    {
      "id": "60de02d9a1c4934b6efe24dc",
      "service_type": "dropbox",
      "name": "Dropbox",
      ...
    }
  ]
}

curl \
--request "GET" \
--url "<ENTERPRISE_SEARCH_BASE_URL>/api/ws/v1/sources/60de02d9a1c4934b6efe24dc" \
--header "Authorization: Bearer ${TOKEN}"

{
  "id": "60de02d9a1c4934b6efe24dc",
  "service_type": "dropbox",
  "name": "Dropbox",
  ...
  "indexing": {
    ...
    "schedule": {
      "full": "P3D",
      "incremental": "PT2H",
      "delete": "PT6H",
      "estimates": {
        "full": {
          "next_start": "2021-08-09T04:45:48Z",
          "duration": "PT20S"
        },
        "incremental": {
          "next_start": "2021-08-06T06:45:48Z",
          "duration": "PT3S"
        },
        "delete": {
          "next_start": "2021-08-06T10:45:48Z",
          "duration": "PT11S"
        }
      }
    }
  },
  ...
}

curl \
--request "PUT" \
--url "<ENTERPRISE_SEARCH_BASE_URL>/api/ws/v1/sources/60de02d9a1c4934b6efe24dc" \
--header "Authorization: Bearer ${TOKEN}" \
--header "Content-Type: application/json" \
--data '
{
  "id": "60de02d9a1c4934b6efe24dc",
  "service_type": "dropbox",
  "name": "Dropbox",
  "is_searchable": true,
  ...
  "indexing": {
    ...
    "schedule": {
      "full": "P3D",
      "incremental": "PT1H",
      "delete": "PT6H"
    }
  },
  ...
}
'

workplace_search.content_source.sync.refresh_interval:
  full: PT72H
  incremental: PT1H
  delete: PT6H
  permissions: PT6M