Web crawler (beta) API reference

Most endpoints within the crawler API are scoped to a particular App Search engine. The engine is identified by the engine name value provided in the URL of the request. If an engine could not be found for any API request, an empty HTTP 404 response will be returned.

Unless security is disabled, you must provide credentials to access API operations.

See Authentication.

Returns a crawl request object for an active crawl or returns an HTTP 404 response if there is no active crawl for a given App Search engine.

GET /api/as/v0/engines/{ENGINE_NAME}/crawler/crawl_requests/active

For successful calls, the response is going to look like this:

# 200 OK
{
  "id": "601b21adbeae67679b3b760a",
  "status": "running",
  "created_at": "Wed, 03 Feb 2021 22:20:29 +0000",
  "begun_at": "Wed, 03 Feb 2021 22:20:31 +0000",
  "completed_at": null
}

For cases when there is no active crawl for a given engine, the API responds with a 404 error:

# 404 Not Found
{
  "error": "There are no active crawl requests for this engine"
}

Cancels an active crawl for a given App Search engine or returns an HTTP 404 response if there is no active crawl for a given App Search engine.

POST /api/as/v0/engines/{ENGINE_NAME}/crawler/crawl_requests/active/cancel

In case of success, the response contains a single crawl request object with a canceling state:

# 200 OK
{
  "id": "601b21adbeae67679b3b760a",
  "status": "canceling",
  "created_at": "Wed, 03 Feb 2021 22:20:29 +0000",
  "begun_at": "Wed, 03 Feb 2021 22:20:31 +0000",
  "completed_at": null
}

For cases when there is no active crawl for a given engine, the API responds with a 404 error:

# 404 Not Found
{
  "error": "There are no active crawl requests for this engine"
}

Returns a list of the most recent crawl requests for a given engine. The number of items returned (default: 10) can be changed by using the limit argument.

GET /api/as/v0/engines/{ENGINE_NAME}/crawler/crawl_requests
GET /api/as/v0/engines/{ENGINE_NAME}/crawler/crawl_requests?limit=25

# 200 OK
[
  {
    "id": "601b21adbeae67679b3b760a",
    "status": "running",
    "created_at": "Wed, 03 Feb 2021 22:20:29 +0000",
    "begun_at": "Wed, 03 Feb 2021 22:20:31 +0000",
    "completed_at": null
  },
  {
    "id": "60147e93beae67bf7ef72e86",
    "status": "success",
    "created_at": "Fri, 29 Jan 2021 21:30:59 +0000",
    "begun_at": "Fri, 29 Jan 2021 21:31:00 +0000",
    "completed_at": "Fri, 29 Jan 2021 21:35:20 +0000"
  },
  {
    "id": "60146c07beae67f397300128",
    "status": "canceled",
    "created_at": "Fri, 29 Jan 2021 20:11:51 +0000",
    "begun_at": "Fri, 29 Jan 2021 20:11:52 +0000",
    "completed_at": "Fri, 29 Jan 2021 20:12:51 +0000"
  }
]

Requests a new crawl for a given App Search engine. If there is already an active crawl, the request returns an HTTP 400 response with an error message.

POST /api/as/v0/engines/{ENGINE_NAME}/crawler/crawl_requests

In case of success, the response contains a single crawl request object with a pending state:

# 200 OK
{
  "id": "601b21adbeae67679b3b760a",
  "status": "pending",
  "created_at": "Wed, 03 Feb 2021 22:20:29 +0000",
  "begun_at": null,
  "completed_at": null
}

When there is already an active crawl, the API returns an HTTP 400 response:

# 400 Bad Request
{
  "error": "There is an active crawl for the engine \"your-engine\", please wait for it to finish or abort it before requesting another one"
}

Returns details of a given crawl request. The crawl request is identified with a unique Crawl Request ID value.

GET /api/as/v0/engines/{ENGINE_NAME}/crawler/crawl_requests/{CRAWL_REQUEST_ID}

# 200 OK
{
  "id": "60147e93beae67bf7ef72e86",
  "status": "success",
  "created_at": "Fri, 29 Jan 2021 21:30:59 +0000",
  "begun_at": "Fri, 29 Jan 2021 21:31:00 +0000",
  "completed_at": "Fri, 29 Jan 2021 21:35:20 +0000"
}

Returns a crawl schedule object or returns an HTTP 404 response if there is no crawl schedule object for a given App Search engine.

GET /api/as/v0/engines/{ENGINE_NAME}/crawler/crawl_schedule

For successful calls, the response is going to look like this:

# 200 OK
{
  "engine": {ENGINE_NAME},
  "frequency": 2,
  "unit": "week"
}

For cases when there is no crawl schedule for a given engine, the API responds with a 404 error:

# 404 Not Found
{
  "errors": ["No crawl schedule found"]
}

Upserts a crawl schedule for a given App Search engine.

PUT /api/as/v0/engines/{ENGINE_NAME}/crawler/crawl_schedule
{
  "frequency": {INTEGER},
  "unit": {ENUM}
}

In case of success, the response contains the crawl schedule object:

# 200 OK
{
  "engine": {ENGINE_NAME},
  "frequency": 2,
  "unit": "week"
}

When the parameters are invalid, the API returns an HTTP 400 response:

# 400 Bad Request
{
  "errors": [
    "Crawl schedule frequency must be an integer",
    "Crawl schedule unit must be one of hour, day, week, month"
  ]
}

Deletes a crawl schedule for a given App Search engine.

DELETE /api/as/v0/engines/{ENGINE_NAME}/crawler/crawl_schedule

In case of success, the response contains an object showing the crawl schedule has been deleted:

# 200 OK
{
  "deleted": true
}

For cases when there is no crawl schedule for a given engine, the API responds with a 404 error:

# 404 Not Found
{
  "errors": ["No crawl schedule found"]
}