Content sources API reference

edit

Content sources API reference

edit

This is a technical API reference. Refer to the Custom API sources for a conceptual walkthrough.

In this API reference

edit

Content source JSON representation

edit

Content sources represent the individual connections to 3rd party services, and the documents indexed from those connections. Before using the API to create, access, update, or delete content sources, it is important to be familiar with the JSON notation that will be present in many of these response bodies.

Content source fields

edit
id (required)

The string that uniquely identifies the content source. This ID is generated by Workplace Search.

service_type (required)

The string representing the content source’s service type.

created_at (required)

The string in RFC-3339 format representing the date/time at which this content source was originally created.

last_updated_at (required)

The string in RFC-3339 format representing the date/time at which this content source was last updated.

is_remote (required)

The boolean for whether or not this content source is a "remote" content source. See Standard content sources vs remote content sources.

details (required)

The list of objects that provide metadata for the content source and the account which authenticated/connected it. These details will be displayed in the source details Overview page.

While this key is required, the list may be empty.

Any objects present in this list will contain the following keys:

title (required)

The string for the human readable name of this piece of metadata.

description (required)

The string value to be displayed under the corresponding title.

groups (required)

The list of objects which represent Workplace Search groups that have access to this content source. Each object contains two keypairs, one for id and one for name.

name (required)

The string for the human readable display name of this content source.

schema (required)

The object representing the schema that each document in this content source must adhere to. The keys of this object can be anything constructed of lower-case letters, numbers, and the underscore (_) character. These keys define the field names that may be present on indexed documents. The values of these keypairs must be one of text, number, date, or geolocation.

display (required)

The object representing the display details which govern how documents are displayed in search results.

The display object contains the following keys:

title_field (required)

The string field name that will be displayed as a search result’s title.

subtitle_field (required)

The string field name that will be displayed as a search result’s subtitle. If null, the search result will not display a subtitle.

description_field (required)

The string field name that will be displayed as a search result’s long-form description or body. In the UI, this content will be truncated to fit. If null, the search result will not have a long-form description/body.

url_field (required)

The string field name that will be used to link a given search result to its source of truth.

detail_fields (required)

The list of objects representing how the detailed view of a search result item is displayed, when clicked on.

The detail_fields objects each contain the following keys:

field_name (required)

The string field name that this detail pertains to.

label (required)

The string for the human readable display of this field.

color (optional)

A string hex-color code. The color determines what color to fill the custom source icon with in the search UI.

color is only present for custom content sources.

context (required)

The string specifying whether this content source is available to groups of users, or a single user. The value can be either "organization" or "private."

is_searchable (required)

The boolean for whether or not this content source can currently be searched over on the search page.

document_count (conditional)

The integer of how many documents are currently indexed in this content source. This field is not applicable to Remote content sources.

last_indexed_at (conditional)

The string in RFC-3339 format representing the date/time when documents were last indexed into this content source. This may be null if documents have not yet been indexed. This field is not applicable to Remote content sources.

Example

edit

Below is an example content source’s JSON representation:

{
  "id": "603ebf0440b06a31115f4bad",
  "service_type": "custom",
  "created_at": "2021-03-02T22:41:08+00:00",
  "last_updated_at": "2021-03-02T22:41:16+00:00",
  "is_remote": false,
  "details": [
    {
      "title": "Cited Source",
      "description": "http://nps.gov"
    },
    {
      "title": "Usage",
      "description": "For Enterprise Search Examples"
    }
  ],
  "groups": [
    {
      "id": "603ea1a440b06a9c11fd832d",
      "name": "Default"
    }
  ],
  "name": "National Parks",
  "schema": {
    "description": "text",
    "title": "text",
    "nps_link": "text",
    "states": "text",
    "visitors": "number",
    "square_km": "number",
    "world_heritage_site": "text",
    "date_established": "date",
    "location": "geolocation",
    "acres": "number"
  },
  "display": {
    "title_field": "title",
    "subtitle_field": "states",
    "description_field": "description",
    "url_field": "nps_link",
    "detail_fields": [
      {
        "field_name": "states",
        "label": "States"
      },
      {
        "field_name": "description",
        "label": "Description"
      },
      {
        "field_name": "date_established",
        "label": "Established"
      },
      {
        "field_name": "visitors",
        "label": "Number of Visitors"
      },
      {
        "field_name": "location",
        "label": "Location"
      },
      {
        "field_name": "world_heritage_site",
        "label": "World Heritage Site?"
      },
      {
        "field_name": "square_km",
        "label": "Square Kilometers"
      },
      {
        "field_name": "acres",
        "label": "Acres"
      }
    ],
    "color": "#000000"
  },
  "context": "organization",
  "is_searchable": true,
  "document_count": 59,
  "last_indexed_at": "2021-03-02T22:41:15+00:00"
}

List All content sources

edit

Retrieve a paginated list of all content sources.

GET /api/ws/v1/sources

Request headers

edit
Authorization

Bearer token permitting API access.

For more authentication methods see API Authentication Reference

Request parameters

edit
page[current] (optional)

The integer page of results to retrieve. Default is 1.

page[size] (optional)

The integer size of results to retrieve in a single page. Default is 25

Request examples

edit
curl \
--request 'GET' \
--url 'https://f3fae9ab3e4f423d9d345979331ef3a1.ent-search.us-east-1.aws.cloud.es.io/api/ws/v1/sources' \
--header "Authorization: Bearer $ACCESS_TOKEN"
curl \
--request 'GET' \
--url 'http://localhost:3002/api/ws/v1/sources' \
--header "Authorization: Bearer $ACCESS_TOKEN"

With pagination:

curl \
--request 'GET' \
--url 'http://localhost:3002/api/ws/v1/sources?page[current]=2&page[size]=1' \
--header "Authorization: Bearer $ACCESS_TOKEN"

Responses

edit
200 OK

The paginated list of content sources is returned.

{
  "meta": {
    "page": {
      "current": 1,
      "total_pages": 1,
      "total_results": 4,
      "size": 25
    }
  },
  "results": [
    { <ContentSourceOne> },
    { <ContentSourceTwo> },
    { <ContentSourceThree> },
    { <ContentSourceFour> }
  ]
}

See the above content source JSON representation.

400 Bad Request

The request body was incomplete or invalid. The response body provides additional information.

401 Unauthorized

The authorization header was missing from the request or authentication failed.

404 Not Found

The authenticated user is not an Admin user, and therefore does not have permission.

Get a content source

edit

Retrieve a single content source by its ID.

GET /api/ws/v1/sources/<content-source-id>

Request headers

edit
Authorization

Bearer token permitting API access.

For more authentication methods see API Authentication Reference

Request parameters

edit
content-source-id (required)

The string ID for the content source to be retrieved.

Request examples

edit
curl \
--request 'GET' \
--url "https://f3fae9ab3e4f423d9d345979331ef3a1.ent-search.us-east-1.aws.cloud.es.io/api/ws/v1/sources/$CONTENT_SOURCE_ID" \
--header "Authorization: Bearer $ACCESS_TOKEN"
curl \
--request 'GET' \
--url "http://localhost:3002/api/ws/v1/sources/$CONTENT_SOURCE_ID" \
--header "Authorization: Bearer $ACCESS_TOKEN"

Responses

edit
200 OK

The single content sources is returned.

{ <ContentSource> }

See the above content source JSON representation.

401 Unauthorized

The authorization header was missing from the request or authentication failed.

404 Not Found

The authenticated user is not an Admin user, and therefore does not have permission.

OR

The provided content_source_id does not exist.

Create a content source

edit

Create a single custom content source.

Only custom content sources can be created with this endpoint. Other service types cannot be created programmatically.

POST /api/ws/v1/sources

Request headers

edit
Authorization

Bearer token permitting API access.

For more authentication methods see API Authentication Reference

Content-Type

application/json. This endpoint accepts only JSON POST data.

Request body

edit
Content source definition (required)

The JSON object defining the content source to be created.

name (required)

The string for the human readable display name of this content source.

The name may not start or end with special characters.

The name may not include special characters other than underscore (_).

The name may not be longer than 64 characters.

schema (optional)

The object representing the schema that each document in this content source must adhere to. The keys of this object can be anything constructed of lower-case letters, numbers, and the underscore (_) character. These keys define the field names that may be present on indexed documents. The values of these keypairs must be one of text, number, date, or geolocation.

If omitted, you cannot specify a display at creation time.

You can update schema later by indexing documents (the schema will be inferred) or by using the update endpoint.

display (optional)

The object representing the display details which govern how documents will be displayed in search results.

The display object contains the following keys:

title_field (required)

The string field name that will be displayed as a search result’s title.

This field must exist in the schema.

url_field (required)

The string field name that will be used to link a given search result to its source of truth.

This field must exist in the schema.

subtitle_field (optional)

The string field name that will be displayed as a search result’s subtitle.

This field must exist in the schema.

If null, no field will be assigned as a subtitle, and subtitles will not be displayed for any search result in this content source.

description_field (optional)

The string field name that will be displayed as a search result’s long-form description or body. In the UI, this content will be truncated to fit.

This field must exist in the schema.

If null, no field will be assigned as a description, and descriptions will not be displayed for any search result in this content source.

detail_fields (optional)

The list of objects representing how the detailed view of a search result item should be displayed when clicked on.

The detail_fields objects must each contain the following keys:

field_name (required)

The string field name that this detail pertains to.

This field must exist in the schema.

label (required)

The string for the human readable display of this field.

color (optional)

A string hex-color code. The color determines what color to fill the custom source icon with in the search UI.

color must be a 7-character string, beginning with a hash (#) followed by 6 hexadecimal digits (0-9, A-F). Other color strings or code formats will not be accepted.

is_searchable (optional)

The boolean for whether or not this content source should be searchable.

Example
edit

Below is an example content source definition’s JSON representation:

{
  "name": "National Parks",
  "schema": {
    "description": "text",
    "title": "text",
    "nps_link": "text",
    "states": "text",
    "visitors": "number",
    "square_km": "number",
    "world_heritage_site": "text",
    "date_established": "date",
    "location": "geolocation",
    "acres": "number"
  },
  "display": {
    "title_field": "title",
    "subtitle_field": "states",
    "description_field": "description",
    "url_field": "nps_link",
    "detail_fields": [
      {
        "field_name": "states",
        "label": "States"
      },
      {
        "field_name": "description",
        "label": "Description"
      },
      {
        "field_name": "date_established",
        "label": "Established"
      },
      {
        "field_name": "visitors",
        "label": "Number of Visitors"
      },
      {
        "field_name": "location",
        "label": "Location"
      },
      {
        "field_name": "world_heritage_site",
        "label": "World Heritage Site?"
      },
      {
        "field_name": "square_km",
        "label": "Square Kilometers"
      },
      {
        "field_name": "acres",
        "label": "Acres"
      }
    ],
    "color": "#000000"
  },
  "is_searchable": true
}

Request examples

edit
curl \
--request 'POST' \
--url 'https://f3fae9ab3e4f423d9d345979331ef3a1.ent-search.us-east-1.aws.cloud.es.io/api/ws/v1/sources/' \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer $ACCESS_TOKEN" \
--data '
{
  "name": "Test",
  "schema": {
    "title": "text",
    "url": "text",
    "my_field": "text"
  },
  "display": {
    "title_field": "title",
    "url_field": "url",
    "detail_fields": [
      {
        "field_name": "my_field",
        "label": "My Field"
      }
    ],
    "color": "#111111"
  },
  "is_searchable": true
}
'
curl \
--request 'POST' \
--url 'http://localhost:3002/api/ws/v1/sources/' \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer $ACCESS_TOKEN" \
--data '
{
  "name": "Test",
  "schema": {
    "title": "text",
    "url": "text",
    "my_field": "text"
  },
  "display": {
    "title_field": "title",
    "url_field": "url",
    "detail_fields": [
      {
        "field_name": "my_field",
        "label": "My Field"
      }
    ],
    "color": "#111111"
  },
  "is_searchable": true
}
'

Responses

edit
200 OK

The single created content source is returned.

{ <ContentSource> }

See the above content source JSON representation.

400 Bad Request

The request body did not meet requirements. See the errors array to determine what may be wrong.

{
  "errors": [
    "The property '#/schema/my_field' value \"integer\" did not match one of the following values: text, geolocation, number, date",
    "The property '#/display' did not contain a required property of 'title_field'",
    "The property '#/is_searchable' did not match any of the following types: boolean",
    "The property '#/name' value \"&special_chars*\" did not match the regex '(?-mix:\\A[\\w\\s.()'\\-\\/]*\\z)'"
  ]
}
400 Invalid Display

The request body included a display configuration that, while syntactically correct, did not align with the content source’s schema. This was not detected until after the content source was created, but did prevent the display settings from being applied as expected. The response includes both the current state of the new content source, and the errors in configuring the display.

{
  "created" : { <ContentSource> },
  "errors" : [
    "Invalid display setting: title_field: \"title\" is not a field on the content source",
    "Invalid display setting: subtitle_field: \"states\" is not a field on the content source",
    "Invalid display setting: description_field: \"description\" is not a field on the content source",
    "Invalid display setting: url_field: \"nps_link\" is not a field on the content source",
    "Invalid display setting: detail_fields: \"states\" is not a field on the content source, \"description\" is not a field on the content source, \"date_established\" is not a field on the content source, \"visitors\" is not a field on the content source, \"location\" is not a field on the content source, \"world_heritage_site\" is not a field on the content source, \"square_km\" is not a field on the content source, \"acres\" is not a field on the content source"

  ]
}

See the above content source JSON representation.

401 Unauthorized

The authorization header was missing from the request or authentication failed.

404 Not Found

The authenticated user is not an Admin user, and therefore does not have permission.

Update a content source

edit

Update a single custom content source.

Only custom content sources can be updated with this endpoint. Other service types cannot be updated programmatically.

PUT /api/ws/v1/sources/<content-source-id>

Request headers

edit
Authorization

Bearer token permitting API access.

For more authentication methods see API Authentication Reference

Content-Type

application/json. This endpoint accepts only JSON PUT data.

Request parameters

edit
content-source-id (required)

The string ID for the content source to be updated.

Request body

edit
Content source definition (required)

The JSON object defining the new state of the content source.

Unlike the create request body, all parameters of this body are required. You may find it useful to first do a get request, to ensure you send the complete current values to avoid unwanted changes.

name (required)

The string for the human readable display name of this content source.

The name may not start or end with special characters.

The name may not include special characters other than underscore (_).

The name may not be longer than 64 characters.

schema (required)

The object representing the schema that each document in this content source must adhere to. The keys of this object can be anything constructed of lower-case letters, numbers, and the underscore (_) character. These keys define the field names that may be present on indexed documents. The values of these keypairs must be one of text, number, date, or geolocation.

If the content source already contains documents, this schema must not attempt to change the types of fields for document that do not have an implicit type conversion available to them. For example, a document of

{
  "field_one": 42,
  "field_two": "the answer"
}

a schema update of

{
  "field_one": "text",
  "field_two": "text"
}

is fine, because 42 (numeric) can implicitly be converted to "42" (string) without issue. However, a schema update of

{
  "field_one": "number",
  "field_two": "number"
}

will fail, because there is no implicit numeric value for "the answer".

display (required)

The object representing the display details which govern how documents will be displayed in search results.

The display object contains the following keys:

title_field (required)

The string field name that will be displayed as a search result’s title.

This field must exist in the schema.

url_field (required)

The string field name that will be used to link a given search result to its source of truth.

This field must exist in the schema.

subtitle_field (optional)

The string field name that will be displayed as a search result’s subtitle.

This field must exist in the schema.

If null, no field will be assigned as a subtitle, and subtitles will not be displayed for any search result in this content source.

description_field (optional)

The string field name that will be displayed as a search result’s long-form description or body. In the UI, this content will be truncated to fit.

This field must exist in the schema.

If null, no field will be assigned as a description, and descriptions will not be displayed for any search result in this content source.

detail_fields (optional)

The list of objects representing how the detailed view of a search result item should be displayed when clicked on.

The detail_fields objects must each contain the following keys:

field_name (required)

The string field name that this detail pertains to.

This field must exist in the schema.

label (required)

The string for the human readable display of this field.

color (required)

A string hex-color code. The color determines what color to fill the custom source icon with in the search UI.

color must be a 7-character string, beginning with a hash (#) followed by 6 hexadecimal digits (0-9, A-F). Other color strings or code formats will not be accepted.

is_searchable (required)

The boolean for whether or not this content source should be searchable.

Example
edit

Below is an example content source definition’s JSON representation:

{
  "name": "National Parks",
  "schema": {
    "description": "text",
    "title": "text",
    "nps_link": "text",
    "states": "text",
    "visitors": "number",
    "square_km": "number",
    "world_heritage_site": "text",
    "date_established": "date",
    "location": "geolocation",
    "acres": "number"
  },
  "display": {
    "title_field": "title",
    "subtitle_field": "states",
    "description_field": "description",
    "url_field": "nps_link",
    "detail_fields": [
      {
        "field_name": "states",
        "label": "States"
      },
      {
        "field_name": "description",
        "label": "Description"
      },
      {
        "field_name": "date_established",
        "label": "Established"
      },
      {
        "field_name": "visitors",
        "label": "Number of Visitors"
      },
      {
        "field_name": "location",
        "label": "Location"
      },
      {
        "field_name": "world_heritage_site",
        "label": "World Heritage Site?"
      },
      {
        "field_name": "square_km",
        "label": "Square Kilometers"
      },
      {
        "field_name": "acres",
        "label": "Acres"
      }
    ],
    "color": "#000000"
  },
  "is_searchable": true
}

Request examples

edit
curl \
--request 'PUT' \
--url "https://f3fae9ab3e4f423d9d345979331ef3a1.ent-search.us-east-1.aws.cloud.es.io/api/ws/v1/sources/$CONTENT_SOURCE_ID" \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer $ACCESS_TOKEN" \
--data '
{
  "name": "Test Update",
  "schema": {
    "title": "text",
    "url": "text",
    "my_field": "text"
  },
  "display": {
    "title_field": "title",
    "url_field": "url",
    "detail_fields": [
      {
        "field_name": "my_field",
        "label": "My Updated Field Display"
      }
    ],
    "color": "#2222"
  },
  "is_searchable": true
}
'
curl \
--request 'PUT' \
--url "http://localhost:3002/api/ws/v1/sources/$CONTENT_SOURCE_ID" \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer $ACCESS_TOKEN" \
--data '
{
  "name": "Test Update",
  "schema": {
    "title": "text",
    "url": "text",
    "my_field": "text"
  },
  "display": {
    "title_field": "title",
    "url_field": "url",
    "detail_fields": [
      {
        "field_name": "my_field",
        "label": "My Updated Field Display"
      }
    ],
    "color": "#2222"
  },
  "is_searchable": true
}
'

Responses

edit
200 OK

The single updated content source is returned.

{ <ContentSource> }

See the above content source JSON representation.

400 Bad Request

The request body did not meet requirements. See the errors array to determine what may be wrong.

{
  "errors": [
    "The property '#/schema/my_field' value \"integer\" did not match one of the following values: text, geolocation, number, date",
    "The property '#/display' did not contain a required property of 'title_field'",
    "The property '#/is_searchable' did not match any of the following types: boolean",
    "The property '#/name' value \"&special_chars*\" did not match the regex '(?-mix:\\A[\\w\\s.()'\\-\\/]*\\z)'"
  ]
}
400 Illegal Schema Change

The request body included a schema that, while syntactically correct, would require field type changes for document values that cannot match the new type. This was not detected until after some updates may have been applied, but did prevent the schema update from being applied as expected. The response includes both the current state of the updated content source, and the errors in updating the schema.

{
  "updated" : { <ContentSource> },
  "errors": [
    "Field 'description' cannot be changed to type 'float'. Example IDs that would fail: [\"park_acadia\", \"park_badlands\", \"park_big-bend\", \"park_bryce-canyon\", \"park_canyonlands\", \"park_capitol-reef\", \"park_carlsbad-caverns\", \"park_congaree\", \"park_cuyahoga-valley\", \"park_denali\"]"
  ]
}

See the above content source JSON representation.

400 Invalid Display

The request body included a display configuration that, while syntactically correct, did not align with the content source’s schema. This was not detected until after some updates may have been applied, but did prevent the display settings from being applied as expected. The response includes both the current state of the updated content source, and the errors in configuring the display.

{
  "updated" : { <ContentSource> },
  "errors" : [
    "Invalid display setting: title_field: \"title\" is not a field on the content source",
    "Invalid display setting: subtitle_field: \"states\" is not a field on the content source",
    "Invalid display setting: description_field: \"description\" is not a field on the content source",
    "Invalid display setting: url_field: \"nps_link\" is not a field on the content source",
    "Invalid display setting: detail_fields: \"states\" is not a field on the content source, \"description\" is not a field on the content source, \"date_established\" is not a field on the content source, \"visitors\" is not a field on the content source, \"location\" is not a field on the content source, \"world_heritage_site\" is not a field on the content source, \"square_km\" is not a field on the content source, \"acres\" is not a field on the content source"

  ]
}

See the above content source JSON representation.

401 Unauthorized

The authorization header was missing from the request or authentication failed.

404 Not Found

The specified content source ID does not exist

OR

The authenticated user is not an Admin user, and therefore does not have permission.

Remove a content source

edit

Remove a single content source by its ID.

Only organization content sources can be deleted with this endpoint. Private content sources cannot be deleted with this endpoint.

DELETE /api/ws/v1/sources/<content-source-id>

Request headers

edit
Authorization

Bearer token permitting API access.

For more authentication methods see API Authentication Reference

Request parameters

edit
content-source-id (required)

The string ID for the content source to be removed.

Request examples

edit
curl \
--request 'DELETE' \
--url "https://f3fae9ab3e4f423d9d345979331ef3a1.ent-search.us-east-1.aws.cloud.es.io/api/ws/v1/sources/$CONTENT_SOURCE_ID" \
--header "Authorization: Bearer $ACCESS_TOKEN"
curl \
--request 'DELETE' \
--url "http://localhost:3002/api/ws/v1/sources/$CONTENT_SOURCE_ID" \
--header "Authorization: Bearer $ACCESS_TOKEN"

Responses

edit
200 OK

An acknowledgement that the content source was deleted

{ "deleted" : true }
400 Bad Request

The provided content_source_id is for a private content source, which is not allowed.

{ "errors": ["Invalid parameter(s): Private content sources cannot be deleted by this API"] }
401 Unauthorized

The authorization header was missing from the request or authentication failed.

404 Not Found

The authenticated user is not an Admin user, and therefore does not have permission.

OR

The provided content_source_id does not exist.

Get a document

edit

Retrieve a single content source document, given the IDs of both the content source and the document.

GET /api/ws/v1/sources/<content-source-id>/documents/<document-id>

Request headers

edit
Authorization

Bearer token permitting API access.

For more authentication methods see API Authentication Reference

Request parameters

edit
content-source-id (required)

The string ID for the content source that contains the desired document.

document-id (required)

The string ID for the document to be retrieved.

Request examples

edit
curl \
--request 'GET' \
--url "https://f3fae9ab3e4f423d9d345979331ef3a1.ent-search.us-east-1.aws.cloud.es.io/api/ws/v1/sources/$CONTENT_SOURCE_ID/documents/$DOCUMENT_ID" \
--header "Authorization: Bearer $ACCESS_TOKEN"
curl \
--request 'GET' \
--url "http://localhost:3002/api/ws/v1/sources/$CONTENT_SOURCE_ID/documents/$DOCUMENT_ID" \
--header "Authorization: Bearer $ACCESS_TOKEN"

Responses

edit
200 OK

The single content source document is returned.

The only guaranteed response fields are id, source, content_source_id, and last_updated. The others are content source specific, and must be part of the content source’s schema. You can view a content source’s schema with the get content source API.

{
  "id":  "some_string_id",
  "source": "custom",
  "content_source_id": "603d2c8319cbdbc907675c02",
  "last_updated": "2021-03-02T22:41:16+00:00",
  "some_text_field": "some text value",
  "some_number_field": 42,
  "some_date_field": "2021-03-02T22:41:16+00:00",
  "some_geolocation_field": "44.35,-68.21",
  "_allow_permissions": ["perm1", "perm2"],
  "_deny_permissions": ["perm3", "perm4"]
}
401 Unauthorized

The authorization header was missing from the request or authentication failed.

404 Not Found

The authenticated user is not an Admin user, and therefore does not have permission.

OR

The authenticated user has specified a private content source that is not theirs, and they therefore do not have permission.

OR

The specified document has document-level-permissions which prohibit access to the authenticated user.

OR

The provided content_source_id does not exist.

OR

The provided document_id does not exist.