« View web crawler events logs Web crawler schema »

› ›

Web crawler events logs reference

See View web crawler events logs to learn how to view web crawler events logs in Kibana.

The web crawler logs many events while discovering, extracting, and indexing web content.

Enterprise Search records these events using Elastic Common Schema (ECS), including a custom field set called crawler.* for crawler-specific data (like crawl_id).

This document provides a reference to these events and their fields.

This reference describes the fields common to all web crawler events, including:

The remainder of the document describes different types of web crawler events:

Fields common to all web crawler events

edit

All web crawler events include the following common fields.

Crawler-specific fields

edit

crawler.crawl.id: A unique ID of a specific crawl.

Base fields

edit

@timestamp: A UTC timestamp of the event.
event.id: A unique identifier of the event.
event.action: The type of event. See the sections that follow.
message: A textual description of the event (useful for displaying in a UI for human consumption).

Service fields

edit

service.ephemeral_id: A unique identifier of the crawler process generating the ID (changes every time a process is restarted).
service.type: All events will have this set to crawler.
service.version: Current version of the Enterprise Search product.

Process fields

edit

process.pid: The PID of the crawler instance.
process.thread.id: The id of the thread logging the event.

Host fields

edit

host.name: The host name where the crawler instance is deployed.

Crawl lifecycle events

edit

Each crawl lifecycle event records important checkpoints within the lifecycle of a specific crawl, for example: start, seed, end. Most of the event information is captured in the message field, along with the other common fields described above. The fields below provide additional details.

Each crawl lifecycle event has one of the following values for event.action:

crawl-start: Emitted when a crawl is started. Includes crawl configuration.
crawl-seed: Emitted every time a crawl is seeded with a set of URLs from the outside. Includes the list of URLs submitted to the crawler.
crawl-end: Emitted when a crawl is ended for any reason (finished, canceled, etc).
crawl-status: Periodic events with a snapshot of crawler status metrics used for monitoring an active crawl over time.

Crawl start events

edit

event.kind: Set to event.
event.type: Set to start.
event.action: Set to crawl-start.
crawler.crawl.config: A serialized version of the crawl config.

Crawl seed events

edit

event.kind

Set to event.

event.type

Set to change.

event.action

Set to crawl-seed.

crawler.crawl.seed_urls

A list of URLs used to seed a crawl.

crawler.url.type

A type of the URLs being added:

content for generic content URLs.
sitemap for sitemap and sitemap-index URLs.
feed for RSS/ATOM feeds.

Crawl end events

edit

event.kind: Set to event.
event.type: Set to end.
event.action: Set to crawl-end.
event.outcome: Set to success or failure depending on how a crawl ended (canceled crawls will be considered failed, etc).

Crawl status events

edit

event.kind: Set to metric.
event.type: Set to info.
event.action: Set to crawl-status.
crawler.status.*: A set of metrics describing the global state of a crawl and crawl-specific stats that may be useful to understand the state of a crawl over time.

URL lifecycle events

edit

Each URL lifecycle event is scoped to a particular URL within a specific crawl. Each event describes what happened to the URL during the crawl, for example: how and when did the crawler discover it?, why did the crawler skip it? These events have enough details to allow a human operator to understand exactly how the system discovered a specific URL, what decisions have been made about it, and what was the result of processing the URL.

Each URL lifecycle event has one of the following values for event.action:

url-seed: URL submitted to the crawl backlog for processing (from a seed list, from within the crawl, via an API, etc).
url-fetch: URL fetch attempt including timing information, server response headers, HTTP code, etc.
url-discover: URL discovery events. Each time the crawler discovers a URL on a page and makes a descision about it, the URL and the decision are logged.
url-extracted: Events logged when we finish content extraction from a URL (maybe with some basic metadata extracted from the page).
url-output: An event marking the end of URL processing.

Fields common to all URL lifecycle events

edit

All URL lifecycle events include the following common fields:

Identification fields:

crawler.url.hash: A unique identifier (hash) for the URL as it is handled by the crawler. All events for the same URL within a single crawl share the same hash.
crawler.url.source_hash: A unique identifier of the URL that was used to discover this URL (only used for cases when a URL was discovered during a crawl and not submitted as a seed URL).

URL details:

url.full: The full URL string.
url.scheme: Scheme portion of the URL.
url.domain: Domain portion of the URL.
url.port: Port of the URL.
url.path: Path of the URL.
url.query: URL query string. Included when available.
url.fragment: URL fragment. Included when available.
url.username: Username portion of the URL. Included when available.
url.password: Password portion of the URL. Included when available.

URL seed events

edit

These are small events used to track the flow of URLs into the crawler system and are primarily focused on tracking how a specific URL got into the backlog.

event.kind

Set to event.

event.type

Set to start.

event.action

Set to url-seed.

crawler.url.type

A type of the URL being added:

content for generic content URLs.
sitemap for sitemap and sitemap-index URLs.
feed for RSS/ATOM feeds.

crawler.url.source_type

A name of the source used for seeding the crawl:

seed-list for seed-list URLs submitted as a part of the crawl configuration.
organic for URLs discovered during a crawl by following organic links.
redirect for pages discovered by following a redirect.
canonical-url for pages discovered via the canonical URL meta tag.

crawler.url.source_url.hash

Set to the hash of the URL the crawler used to discover this page (only for URLs discovered during a crawl and not for entry points).

crawler.url.crawl_depth

A positive number, indicating the number of steps the crawler had to take from our seed URLs set to reach this specific page.

URL fetch events

edit

These are the primary events that will be used for troubleshooting networking layer issues with a crawl. They therefore aim to provide enough insight into what happened during a fetch attempt and what were the results.

These events represent a single HTTP request. If the crawler followed redirects, it logs a separate record for each event including information about the redirect response to help with redirect chain troubleshooting.

event.kind: Set to event.
event.type: Set to access.
event.action: Set to url-fetch.

Event timing and outcome details:

event.start

The start of the HTTP request.

event.end

The end of the HTTP request.

event.duration

Response timing for the HTTP request (total time it took to get the full response).

event.outcome

An ECS categorization field. Denotes whether the event represents a success or a failure from the perspective of the crawler:

failure - for all 3xx, 4xx and 5xx responses.
success - for all 2xx responses.
unknown - for network timeouts.

HTTP request details:

http.request.method: The method of the request.

HTTP response details:

http.response.body.bytes: The size of the response body in bytes (for successful responses only).
http.response.status_code: A string status code.

HTTP redirect details:

crawler.url.redirect.location: A Location header content for redirect responses.
crawler.url.redirect.count: Number of redirects followed so far in a redirect chain (starts with 1 on the first redirect and is increased on each subsequent redirect until a non-redirect response is received or the maximum number of redirects is reached).

URL discover events

edit

These are small events used to troubleshoot URL discovery within the crawler. Each time the crawler sees a new URL (extracted from a page, from a sitemap or from following a redirect), it logs information about it along with the decision on what will happen to the newly discovered link.

event.kind

Set to event.

event.action

Set to url-discover.

event.type

Depending on the decision regarding the URL, set to one of:

allowed if the URL will be added to the backlog for future crawling.
denied if the URL will not be followed (the message field will have a human-readable explanation of why the crawler decided not to follow it).

crawler.url.source_type

A type of the source used for discovering the link:

organic for URLs discovered during a crawl by following organic links.
redirect for pages discovered by following a redirect.

crawler.url.source_url.hash

Set to the hash of the URL the crawler used to discover this page (for URLs discovered during the crawl and not for entry points).

crawler.url.crawl_depth

A positive number, indicating the number of steps the crawler had to take from our seed URLs set to reach this specific page.

crawler.url.deny_reason

A field with a code explaining the reason for skipping a URL during a crawl:

already_seen when this exact URL/page has already been processed in this crawl.
link_too_deep when we hit a crawl depth limit.
link_too_long when we hit a URL length limit.
link_with_too_many_params when we hit a limit on the number of URL parameters allowed.
link_with_too_many_segments when we hit a limit on the number of URL segments allowed.
queue_full when we hit a backlog size limit.
sitemap_denied when a URL is prohibited from crawling by a sitemap rule.
domain_filter_denied for prohibited cross-domain links.
page_already_visited for crawl-scoped URL de-duplication events.
incorrect_protocol for non-HTTP links and non-HTTPS links in HTTPS-enforced mode.

URL extracted events

edit

These events are focused on the extraction portion of the crawler process and are logged to help an operator troubleshoot the process of content extraction for the pages on their domains. The primary focus here is capturing the details of the extraction process.

Each event represents a single extractor handling a single piece of content.

event.kind

Set to event.

event.action

Set to url-extracted.

event.module

The name of the extractor generating the event (e.g. html).

event.type

Depending on the decision regarding the URL, set to one of:

allowed if the URL has been allowed to be indexed.
denied if the URL has not been indexed because of a crawl rule, a robots.txt rule, etc (the message field will have a human-readable explanation of what happened).

Event timing and outcome details:

event.start

The start of the extraction process.

event.end

The end of the extraction process.

event.duration

End-to-end timing for the extraction process (total time it took to get the data extracted).

event.outcome

An ECS categorization field. Denotes whether the event represents a success or a failure from the perspective of the crawler:

failure if extraction process failed and we are going to drop the content.
success if extraction process succeeded (or failed in a graceful manner).

Extraction result details:

crawler.extraction.content_type: Content type for the page.
crawler.extraction.content_size.bytes: The size of the page.
crawler.extraction.fields_extracted: The list of fields extracted.

URL output events

edit

These events are designed to capture the results of ingestion of a single piece of content into an external system (file, App Search, etc). The main goal here is to capture any data needed to tie a URL fetched and processed by the crawler to the changes performed in the external system as a result of the crawl.

Each event represents a single output module handling a single piece of content.

event.kind: Set to event.
event.type: Set to end.
event.action: Set to url-output.
event.module: The name of the output module generating the event (e.g. file, app-search).

Event timing and outcome details:

event.start

The start of the output ingestion process.

event.end

The end of the output ingestion process.

event.duration

End-to-end timing for the output ingestion process (total time it took to get the data processed by the module).

event.outcome

An ECS categorization field. Denotes whether the event represents a success or a failure from the perspective of the crawler:

failure if output ingestion process failed and we are going to drop the content.
success if output ingestion process succeeded (or failed in a graceful manner).
unknown for cases specific to an output module.

Output ingestion results (file module):

crawler.output.file.directory: The directory where the event has been logged.
crawler.output.file.name: The name of the file where the event has been logged (base name without the directory).

Output ingestion results (app-search module):

crawler.output.app-search.engine.id: The id of the engine used to ingest the content.
crawler.output.app-search.engine.name: The name of the engine used to ingest the content.
crawler.output.app-search.document_id: The id of the document within the engine.
crawler.output.app-search.content_hash: The content hash used for de-duplication purposes.

Content ingestion events

edit

A special kind of event used to troubleshoot the ingestion process. These events are used only by complex output modules and, potentially, only enabled in debug mode or by using a special crawl config option. The goal of these events is to explain the ingestion process results in more details than could be captured by a URL output event.

event.kind

Set to event.

event.type

Set to info.

event.action

Set to ingest-progress.

event.module

The name of the output module generating the event (e.g. file, app-search).

message

Details on what is happening with the extraction process.

App Search logs URL-scoped events that explain how a specific piece of content from the crawler got ingested into the external system. These are important for troubleshooting cases when the crawler discovers and crawls a URL, but due to App Search de-duplication logic the content does not get ingested, etc.

ingest-progress

An event logged by an output module to help an operator troubleshoot the ingestion process. These are pretty generic events using the message field to explain what is happening.

URL identification fields:

These are used to correlate an ingestion event to the rest of the events generated by the crawler for a specific page:

crawler.url.hash: A unique identifier for the URL as it is handled by the crawler, all events for the same URL within a single crawl share the same hash (since it is calculated as SHA1 hash of the URL itself).
url.full: The full URL string.
url.scheme: Scheme portion of the URL.
url.domain: Domain portion of the URL.
url.port: Port of the URL.
url.path: Path of the URL.
url.query: URL query string. Included when available.
url.fragment: URL fragment. Included when available.
url.username: Username portion of the URL. Included when available.
url.password: Password portion of the URL. Included when available.

« View web crawler events logs Web crawler schema »