Custom sources indexing API reference

Custom Source fields can be one of four different types:

Text fields are at the heart of search. They are analyzed fields and are used for full-text matching in information retrieval. Any group of characters or text that you want to search over should be text.

Example: A description of an object, the name of a product, the content of a review.

number fields represent a single-precision, floating-point value (32 bits): 3.14 or 42.

Number fields enable fine grained sorting, filtering, faceting, and boosting.

(If you need to represent a larger number, consider a text field as a workaround.)

Example: A price, a review score, the number of visitors, or a size.

Dates must be in ISO 8601 format, i.e. "2013-02-27T18:09:19Z" or "2013-02-27T17:09:19+01:00".

Example: A product release or publish date, birth date, an air date.

Geolocation fields are latitude-longitude pairs, representing locations.

Examples: A store where a product is located; the location of a venue.

Specify a geolocation using any of the following formats:

"location": "41.12,-71.34" 

"location": "drm3btev3e86" 

"location": [ -71.34, 41.12 ] 

"location" : "POINT (-71.34 41.12)" 

For more details, see Geo-point field type in the Elasticsearch documentation. However, be aware Enterprise Search supports fewer formats than Elasticsearch. Enterprise Search supports only the formats shown above.

Index new objects into a Custom Source or update existing documents.

Request limits: Maximum 100 documents per request

POST /api/ws/v1/sources/[ID]/documents/bulk_create

curl -X POST http://localhost:3002/api/ws/v1/sources/[ID]/documents/bulk_create \
-H "Authorization: Bearer [AUTH_TOKEN]" \
-H "Content-Type: application/json" \
-d '[
  {
    "_allow_permissions": ["permission1"],
    "_deny_permissions": [],
    "id" : 1234,
    "title" : "The Meaning of Time",
    "body" : "Not much. It is a made up thing.",
    "url" : "https://example.com/meaning/of/time",
    "created_at": "2019-06-01T12:00:00+00:00",
    "type": "list"
  },
  {
    "_allow_permissions": [],
    "_deny_permissions": ["permission2"],
    "id" : 1235,
    "title" : "The Meaning of Sleep",
    "body" : "Rest, recharge, and connect to the Ether.",
    "url" : "https://example.com/meaning/of/sleep",
    "created_at": "2019-06-01T12:00:00+00:00",
    "type": "list"
  },
  {
    "_allow_permissions": ["permission1"],
    "_deny_permissions": ["permission2"],
    "id" : 1236,
    "title" : "The Meaning of Life",
    "body" : "Be excellent to each other.",
    "url" : "https://example.com/meaning/of/life",
    "created_at": "2019-06-01T12:00:00+00:00",
    "type": "list"
  }
]'

{
  "results": [
    {
      "id": "1234",
      "errors": []
    },
    {
      "id": "1235",
      "errors": []
    },
    {
      "id": "1236",
      "errors": []
    }
  ]
}

List documents in a Custom Source.

POST /api/ws/v1/sources/[ID]/documents

curl -X POST http://localhost:3002/api/ws/v1/sources/[ID]/documents \
-H "Authorization: Bearer [AUTH_TOKEN]" \
-H "Content-Type: application/json" \
-d '{
  "page": {
    "size": 2
  },
  "sort": {
    "id": "asc"
  },
  "filters": {
    "any": [
      { "states": "Texas" },
      { "world_heritage_site": "true" }
    ]
  }
}'

{
  "meta": {
    "page": {
      "current": 1,
      "total_pages": 8,
      "total_results": 16,
      "size": 2
    },
    "warnings": [],
    "cursor": {
      "current": null,
      "next": "eyJzb3J0Ijp7ImlkIjoiYXNjIn0sInNlYXJjaF9hZnRlciI6WyJwYXJrX2NhcmxzYmFkLWNhdmVybnMiLDEuMCwicGFya19jYXJsc2JhZC1jYXZlcm5zIl19"
    }
  },
  "results": [
    {
      "last_updated": "2021-10-14T23:00:42+00:00",
      "source": "custom",
      "title": "Big Bend",
      "states": [
        "Texas"
      ],
      "world_heritage_site": "false",
      "updated_at": "2021-10-14T23:00:42+00:00",
      "content_source_id": "6168b6991b09597d71076700",
      "id": "park_big-bend"
    },
    {
      "last_updated": "2021-10-14T23:00:42+00:00",
      "source": "custom",
      "title": "Carlsbad Caverns",
      "states": [
        "New Mexico"
      ],
      "world_heritage_site": "true",
      "updated_at": "2021-10-14T23:00:42+00:00",
      "content_source_id": "6168b6991b09597d71076700",
      "id": "park_carlsbad-caverns"
    }
  ]
}

curl -X POST http://localhost:3002/api/ws/v1/sources/[ID]/documents \
-H "Authorization: Bearer [AUTH_TOKEN]" \
-H "Content-Type: application/json" \
-d '{
  "cursor": "eyJzb3J0Ijp7ImlkIjoiYXNjIn0sInNlYXJjaF9hZnRlciI6WyJwYXJrX2NhcmxzYmFkLWNhdmVybnMiLDEuMCwicGFya19jYXJsc2JhZC1jYXZlcm5zIl19",
  "page": {
    "size": 1
  },
  "filters": {
    "any": [
      { "states": "Texas" },
      { "world_heritage_site": "true" }
    ]
  }
}'

{
  "meta": {
    "page": {
      "current": null,
      "total_pages": 16,
      "total_results": 16,
      "size": 1
    },
    "warnings": [],
    "cursor": {
      "current": "eyJzb3J0Ijp7ImlkIjoiYXNjIn0sInNlYXJjaF9hZnRlciI6WyJwYXJrX2NhcmxzYmFkLWNhdmVybnMiLDEuMCwicGFya19jYXJsc2JhZC1jYXZlcm5zIl19",
      "next": "eyJzb3J0Ijp7ImlkIjoiYXNjIn0sInNlYXJjaF9hZnRlciI6WyJwYXJrX2V2ZXJnbGFkZXMiLDEuMCwicGFya19ldmVyZ2xhZGVzIl19"
    }
  },
  "results": [
    {
      "last_updated": "2021-10-14T23:00:42+00:00",
      "source": "custom",
      "title": "Everglades",
      "states": [
        "Florida"
      ],
      "world_heritage_site": "true",
      "updated_at": "2021-10-14T23:00:42+00:00",
      "content_source_id": "6168b6991b09597d71076700",
      "id": "park_everglades"
    }
  ]
}

Remove documents by ID from a Custom Source.

POST /api/ws/v1/sources/[ID]/documents/bulk_destroy

curl -X POST http://localhost:3002/api/ws/v1/sources/[ID]/documents/bulk_destroy \
-H "Authorization: Bearer [AUTH_TOKEN]" \
-H "Content-Type: application/json" \
-d '[
  [DOCUMENT_ID_1], [DOCUMENT_ID_2]
]'

{
  results: [
    {
      "id":1234,
      "success":true
    },
    {
      "id":1235,
      "success":true
    }
  ]
}

Remove documents by query from a Custom Source.

DELETE /api/ws/v1/sources/[ID]/documents

curl -X DELETE http://localhost:3002/api/ws/v1/sources/[ID]/documents \
-H "Authorization: Bearer [AUTH_TOKEN]"
-d '
{
  "filters" : {
    "last_updated" : {
      "from": "2020-06-01T12:00:00+00:00"
    }
  }
}
'

{
  "total" : 234,
  "deleted" : 234,
  "failures" : []
}

Each document within a content source must have a unique id. If you do not provide an id, a BSON id will be created for you. Two documents in two separate content sources may have the same id.

You can update existing documents by issuing a POST request to an existing id.

If the id does not exist, a new document is created. It is up to you to maintain the integrity of your id for each document within each Custom API Source.

We recommend that you avoid SHAs or any identifier derived from the content of a document. Any modification of the original data will alter the value, making it difficult to identify the document in the search index. This can lead to record duplication.

Custom sources allow you to define at the document-level which user may or may not access the result as part of the search experience. Two reserved fields (_allow_permissions and _deny_permissions) accept array-type values. Using proper user mapping, you can generate sophisticated document access controls.

Deny permissions take precedence.

Read more in the Document permissions for custom sources guide.