- App Search Guide: other versions:
- Installation
- Getting started
- Authentication
- Limits
- Users and access
- Guides
- Adaptive relevance events logs reference
- Analytics Tags
- Crawl web content
- Crawl a private network using a web crawler on Elastic Cloud
- Crawl custom fields using proxy
- Curations
- Elasticsearch search
- Elasticsearch index engines
- Create Elasticsearch index engines
- Configure dynamic field mappings and analyzers in an Elasticsearch index engine
- Elasticsearch engines text field conventions
- Facets
- Hierarchical Facets
- Indexing Documents
- Language Optimization
- Log settings
- Meta Engines
- Precision tuning (beta)
- Query Suggestions
- Search UI
- Relevance Tuning
- Result Settings
- Result Suggestions
- Role based access control (RBAC)
- Sanitization, Raw or Snippet
- Search
- Synonyms
- View web crawler events logs
- App Search web crawler
- Web crawler FAQ
- Web crawler reference
- Web crawler events logs reference
- API Reference
- Adaptive relevance API reference (beta)
- Analytics APIs
- Analytics clicks API
- Analytics counts API
- Analytics queries API
- API logs API
- Click API
- Credentials API
- Curations API reference
- Documents API
- Elasticsearch search API
- Engines API
- Log settings API
- Multi search API
- Query suggestion API
- Schema API
- Search API
- Search API boosts
- Search API facets
- Search API filters
- Search API group
- Search API precision (beta)
- Search API result fields
- Search API search fields
- Search API sort
- Search API analytics tags
- Search settings API
- Search Explain API
- Source engines API
- Synonyms API
- Web crawler API reference
- API Clients
- Configuration
- Known issues
- Troubleshooting
View web crawler events logs
editView web crawler events logs
editApp Search stores its web crawler logs in Elasticsearch. Kibana provides two user interfaces to view these logs: Discover and Logs.
For a complete reference of all events, see Web crawler events logs reference.
View web crawler events logs in Discover
editTo view crawler logs in Discover, you must first configure a Kibana data view for the logs-crawler-default data stream.
- Navigate to Stack Management → Data Views via the left sidebar.
- Select Create data view.
-
On the Create data view page, set the data view name to
logs-crawler-defaultand create the data view. - Navigate to Discover via the left sidebar.
- Select your newly created data view, and you are ready to begin exploring your crawl events in detail.
You will likely want to set up some custom columns in Discover to more easily discern crawl events at a glance.
To add a new column, select an entry from the list of available fields on the left sidebar.
Handy columns for crawler events include crawler.crawl.id, url.domain, url.path, event.action, and http.response.status_code.
View web crawler events logs in Logs
editTo view crawler logs in Logs, you must first configure a source configuration.
- Navigate to Logs UI via the left sidebar, under Observability.
-
Under Settings, specify
logs-crawler-defaultas the Log indices data view. -
Configure any desired custom columns. A nice place to start is by removing the preset
event.datasetandmessageoptions and addingcrawler.crawl.id,url.domain,url.path,event.action, andhttp.response.status_code.Autocomplete will be available on these columns after the initial source configuration has been applied. You can edit these settings at any time.
- Apply the configuration, navigate to Stream, and you are ready to begin exploring your crawl events in detail!