Data Quality dashboard
editData Quality dashboard
editThe Data Quality dashboard shows you whether your data is correctly mapped to the Elastic Common Schema (ECS). Successful mapping enables you to search, visualize, and interact with your data throughout Elastic Security.
Use the Data Quality dashboard to:
- Check one or multiple indices for unsuccessful mappings, to help you identify problems (the indices used by Elastic Security appear by default).
- View the number of documents stored in each of your indices.
- View detailed information about the fields in checked indices.
- Track unsuccessful mappings by creating a case or Markdown report based on data quality results.
Requirements
To use the Data Quality dashboard, you need the appropriate user role with the following privileges for each index you want to check:
-
monitorormanage -
view_index_metadataormanage(required for the Get mapping API) -
read(required for the Search API)
Check indices
editWhen you open the dashboard, data does not appear until you select indices to check.
- Check multiple indices: To check all indices in the current data view, click Check all at the top of the dashboard. A progress indicator will appear.
To customize which indices are checked when you click Check all, change the current data view.
- Check a single index: To check a single index, click the Check now button under Actions. Checking a single index is faster than checking all indices.
Visualize checked indices
editThe treemap that appears at the top of the dashboard shows the relative document count of your indices. The color of each index’s node refers to its status:
- Blue: Not yet checked.
- Green: Checked, no incompatible fields found.
- Red: Checked, one or more incompatible fields found.
Click a node in the treemap to expand the corresponding index.
Learn more about checked index fields
editAfter an index is checked, a Pass or Fail status appears. Fail indicates mapping problems in an index. To view index check details, including which fields weren’t successfully mapped, click the Check now button under Actions.
The index check flyout provides more information about the status of fields in that index. Each of its tabs describe fields grouped by mapping status.
Fields in the Same family category have the correct search behavior, but might have different storage or performance characteristics (for example, you can index strings to both text and keyword fields). To learn more, refer to Field data types.
View historical data quality results
editYou can review an index’s data quality history by clicking View history under Actions, or by clicking the History tab in the details flyout. You can filter the results by time and Pass / Fail status. Click a historical check to expand it and view more details.
Recent historical data includes the Incompatible fields and Same family views. Legacy historical data only includes the Incompatible fields view.
Export data quality results
editYou can share data quality results to help track your team’s remediation efforts. First, follow the instructions under Check indices to generate results, then either:
Export results for all indices in the current data view:
- At the top of the dashboard, under the Check all button, are two buttons that allow you to share results. Exported results include all the data which appears in the dashboard.
- Click Add to new case to open a new case.
- Click Copy to clipboard to copy a Markdown report to your clipboard.
Export results for one index:
- View details for a checked index by clicking the Check now button under Actions.
- From the Incompatible fields tab, select Add to new case to open a new case, or click Copy to clipboard to copy a Markdown report to your clipboard.
For more information about how to fix mapping problems, refer to Mapping.