Content extraction

The 3rd party services you sync with Workplace Search, such as Dropbox or Google Drive, usually contain a wide variety of documents and file types. Workplace Search will try to extract the content of these files, to transform the source document into a searchable document.

To make the document searchable, the Workplace Search connector tries to extract text content into fields, and images into thumbnail previews. Full text content extraction is available for many types of documents, including PDFs and most Office365 and GSuite formats. Thumbnail extraction is available for certain image formats.

Nevertheless, you might be surprised that some of your documents are not having their content extracted, or that the extraction is not perfect. The following documentation covers the file extensions and media types supported by Workplace Search, as well as how to troubleshoot surprising results.

There are some important facts and figures to note up front:

You have the option to toggle off extracting thumbnails and/or full-text from files, if you need to save RAM. However, these toggle options are only available after you create the content source. When you create a content source this triggers a full sync immediately, including thumbnail and full-text extraction.

If you want to avoid any thumbnail or full-text extraction you need to switch the toggle immediately after creating the content source. This ensures that the toggle is registered during the initial, metadata-only phase of the first full sync.

Media, or MIME types, are an internet standard for describing file formats.

Per this guide, the type represents the general category into which the data type falls, such as video or text. The subtype identifies the exact kind of data the type represents.

Workplace Search analyzes the file to determine its type, since file extensions are not reliable. Workplace Search leverages the industry standard open-source Apache Tika toolkit for detecting and extracting text and metadata. The Elasticsearch ingest attachment plugin also uses Apache Tika and can return a number of metadata fields.

The following file types are supported for full text extraction:

Formatted text files are normalized to decrease whitespace and minimize storage costs:

Workplace Search supports these MIME types for text files:

Content/text extraction from Google Docs, Sheets and Slides is also supported. Google docs do not have a native download format, and in order to extract their content, Workplace Search exports these files as PDFs.

However, extracting text from PDFs is not always a perfectly lossless process, and can lead to unexpected results in some cases.

Workplace Search provides document thumbnail previews for certain file types, to help you quickly find exactly what you need.

The Workplace Search thumbnail extractor supports specific media or MIME types:

Searching binary formats (e.g., images, audio, videos) is currently not supported. You can use a 3rd party content extractor and add the content via a custom source.

Since 8.3.0, Workplace Search has used an Elasticsearch ingest pipeline to power post-processing of all documents. This pipeline is named ent_search_connector, and is automatically created when Enterprise Search first starts. As a final step in the Full and Incremental sync jobs, every document extracted by these connectors will be sent through this ingest pipeline in order to clean up and compress the content of the body field. This primarily involves removing unicode replacement characters (�), and collapsing large sections of whitespace. This helps to ensure a more meaningful end-user display of your documents in your search experience. You can view and update this pipeline in Kibana or with Elasticsearch APIs.