What’s new in Kibana 7.9.0: A new architecture delivers instant page loads
Today we are excited to announce the availability of Kibana 7.9. With this release, we are rolling out an updated underlying architecture for Kibana itself resulting in a faster overall experience as well as productivity enhancements for the Kibana developer community. Kibana 7.9 also brings with it new dashboarding capabilities to help drive discovery and insight, Kibana Lens enhancements designed to supercharge your data visualization efforts, and a host of new mapping, and machine learning capabilities.
Want to experience all of these new Kibana features right now? Starting a free trial of Elasticsearch and Kibana on Elastic Cloud takes less than 3 minutes and is the only managed Elasticsearch offering to include these new capabilities. You can also easily download the latest versions of Kibana and the Elastic Stack.
For a full list of bug fixes and other changes, don’t miss the Kibana 7.9 release notes.
A new architecture for Kibana brings you snappier page loads and so much more
If the 7.9 Kibana experience feels faster, it’s not your imagination. We have been working for more than 18 months to re-think Kibana’s entire architecture from first principles in order to build a new system designed for scalability and efficiency. We are thrilled in 7.9 to report that work has been completed and that all of Kibana has been moved to this new architecture.
The most immediate difference you’ll notice with this migration is simply how fast Kibana pages load. Whether you are diagnosing a site outage, investigating suspicious user activity, or putting together a dashboard in time for a meeting, moving between interfaces is immediate and helps keep you in the flow of finishing your task. For the Kibana development community, this new architecture brings a number of powerful enhancements designed to help you build features faster and with greater efficiency. For more details on this effort, check out Elastic principal software engineer Josh Dover’s blog post, “Introducing a new architecture for Kibana.”
Rolling out the welcome mat for Elastic Enterprise Search in Kibana
At Elastic we talk a lot about how Kibana is the window into the Elastic Stack and how that stack supports Elastic’s three solutions: Elastic Observability, Elastic Security, and Elastic Enterprise Search. In 7.8 we reorganized the Kibana side navigation to help make observability and security applications easier to find and in 7.9 we’re completing the solution trifecta by adding enterprise search to the fold. For existing users of App Search and Workplace Search, you’ll now be able to quickly see your existing engines from within Kibana as well as launch directly into those environments. Not already using Elastic Enterprise Search, but curious to learn more? Check out the What's New in Elastic Enterprise 7.9 release blog to get all the greatest updates and learn more about how to bring modern, powerful search experiences to your company’s internal content, website, or mobile application.
Go from Dashboard to document level data in seconds with the new explore underlying data feature
Dashboards in Kibana provide an easy, yet powerful way to centralize visualizations built on top of Elasticsearch data. Viewers of Kibana dashboards enjoy not just detailed charts, but also the ability to easily interact and filter analyses in order to discover additional insights. In 7.8 we released the Drilldowns capability for Kibana which lets you create custom drill paths between one or more dashboards. In 7.9 we’re going even further by introducing the explore underlying data feature that allows you to simply use a chart’s action panel to go directly to the Discover app so you can investigate the document level details of the chart data you were just looking at — complete with the previously selected filter, query, and time range.
This new capability works from the context menu of any standard Kibana visualization (including Kibana Lens visuals) based on a single Elasticsearch index. In can also be enabled as an in-chart context menu by editing your kibana.yaml settings file to add: xpack.discoverEnhanced.actions.exploreDataInChart.enabled: true
Doing more with Anomaly Explorer visuals in dashboards
In 7.8 we made it possible to add visuals from Anomaly Explorer right into a dashboard just like any other chart to help you build more robust views of your data. In 7.9 we’ve added three new capabilities to that experience designed to give you and viewers of your dashboards even greater interactivity. The first is the ability to select cells within an embedded Anomaly Explorer visual and use that selection as a time filter for the rest of the dashboard. The second new capability is simply the option to use selected cells as an overall dashboard filter. And lastly, you can now navigate from an embedded Anomaly Explorer view in a dashboard directly to that view in the Anomaly Detection tab within Elastic Machine Learning. All of these new features help you easily integrate visualized machine learning results directly within your dashboards, making it simple to create analyses that offer deeper insights into your data.
Embedding Kibana: New iframe configurations
Taking Kibana dashboards and embedding them in your internal wiki or even your external website is a powerful way to put visualized insights from Elasticsearch data in front of hundreds if not thousands or even tens of people. In 7.9, thanks to an outstanding community contribution from Alex Wild, you have greater flexibility to show exactly which elements appear when you embed Kibana via iframe. Specifically, you can now choose to include or exclude the filter bar, query, time bar, and top menu.
Kibana Lens
Visually comparing data using multiple y axes
Comparing two metrics at the same time is a powerful and quick way to spot outliers. For example, you might be monitoring for potential intrusion threats against your website by tracking something like port scans in one analysis and unique IP addresses in another. Both of these are the right places to be looking, but separately these metrics might not tell the whole story. The ability to place these two indicators visually on top of each other might be the only way to flag that a single IP is actually conducting a suspiciously high number of the overall port scans against your site for a given time period.
In 7.9 we want to make this kind of data comparison even easier in Kibana Lens, which is why we’re so excited to be rolling out the ability to have two metrics on a y axes at the same time. Lens has always been able to stack multiple metrics on top of one another (either from the same index or different ones), but with the new multiple y axes capability you no longer have to have those fields share the same left-hand axis. In 7.9 you can control which y axis is measured on the right and which is on the left, giving you the power to compare and contrast data with vastly different ranges. Using this feature will be especially helpful when looking to compare an aggregated metric (e.g., total pageviews) against a percentage (e.g., webpage bounce rate) since the potentially wide ranged metric whose results could tally in the millions won’t visually drown the percentage whose results are most often between 1 and 100.
Custom color selector
Crafting the look of your Kibana Lens visualization just got easier in 7.9 with the new ability to pick a specific color for a field on the y axis. The same format & style tab used to shift an axis to the right or left to achieve multiple y axes also contains a series style input that lets you determine the exact color you’d like a metric to be. You can choose between using the visual color picker or entering in an exact hexadecimal color code if you already know the precise color you’d like to use. If you are segmenting your y axis with another field in the “Breakdown by” area to build a stacked bar chart or draw multiple lines, etc., Lens will still automatically pick which colors are used. However, where you are using multiple independent metrics on the y axis for comparison analytics, using this new custom color selector is highly recommended to help you visually create stark color contrast.
Handling data gaps with fitting functions
It is not uncommon when analyzing data to come across gaps where there is simply no data available. Your website was down for maintenance and so there just aren’t any pageviews for a one-hour period or maybe the time range you are examining when investigating suspicious log activity is granular enough that there are large visual gaps between timestamps. Whatever the situation, it’s important to be able to visually represent these scenarios accurately or you run the risk of accidentally conveying the wrong message.
In 7.9, Lens helps you do this quickly and easily by introducing the chart settings dropdown which includes data fitting functions designed to let you choose exactly how to represent sparse data. Lens’s default is to not fill in data gaps (hide function), however you can now also choose to fill gaps with zeros (zero function), fill gaps with a line (linear function), or fill gaps according to the last or next data point (last and next functions). Going back to the examples above, a website maintenance outage means there were zero pageviews happening and so representing those breaks by showing a dip to zero might be appropriate. In the hunting suspicious log activity example though, being so zoomed in on a time range that the visual is showing the gaps between single minute timestamp updates doesn’t mean there is a dip in data, just that what’s being shown is literally between normal intervals. In this scenario it might be more appropriate to use the linear or last options to convey that there is in fact no downward trend or missing data. Regardless of the scenario, Kibana Lens in 7.9 is ready to help you paint the right picture when it comes to filling in the gaps (or not).
Canvas
Variables: Edit once, update everything
It’s no secret that Canvas gives you the power to create stunning visual dashboards and presentations from your data stored in Elasticsearch. In 7.9 we are making the process for building and iterating beautiful Canvas workpads more efficient by introducing variables when writing Canvas expressions. Where previously you might have hardcoded something in the expression editor like the color of a shape or even the name of an index used by a chart or metric, you now have the ability to use the “var” function. This allows you to reference a string, boolean, or number that acts as an alias. That alias is tied to a specific value (e.g., hexidecimal color code, index name, etc.) and is accessible in the new variables interface outside of the expression editor. Ultimately this makes changing a single variable all that is potentially needed to update multiple Canvas elements simultaneously.
The new variables feature is incredibly useful especially if you use your Canvas workpads as templates. If you clone or update workpads often and want to efficiently swap style colors, fonts, etc., or have a lightning fast way to simply point some or all of the elements at a different Elasticsearch index, then you are going to love using variables.
Elastic Maps
More out-of-the-box map layers: Network traffic for security practitioners
In 7.8 we introduced our first prebuilt solution specific layer in Elastic Maps for observability professionals intended to automatically take APM data and build real user monitoring (RUM) traffic and performance maps in just a few clicks. With 7.9 we're adding a new solution layer for security practitioners, designed to take network data and rapidly build geography-driven network traffic analyses. If it feels like you’ve seen something like this before, you have. This visual is affectionately known as a pew-pew map and already lives as an embedded item in the Network interface for Elastic Security. Our goal with adding it as a prebuilt layer inside of Elastic Maps is to extend its potential impact for threat hunters with the ability to further customize the analysis using additional data layers as well as make it easier to add to dashboards and Canvas workpads.
Building filled maps faster with the new choropleth wizard
Heatmap? Filled map? Shaded map? If you’ve ever wondered how exactly to refer to a map where the areas inside the map shapes are colored (e.g., a world map comparing various countries, colored according to a metric like GDP, literacy, military budget, etc.), you are in luck because Elastic Maps 7.9 can tell you. It’s officially called a choropleth map and it’s never been easier to build one in the Elastic Maps app with the new choropleth step-by-step wizard. The new Choropleth option in the “Add layer” menu lets you quickly select the administrative boundaries for your geoanalysis or geo_shapes as defined in one of your Elasticsearch indices. From there it is just a few more clicks to define what metric governs the coloring of your map shapes and then the standard easy to use Elastic Maps formatting options. A key time saver with this new feature is the way the wizard automatically handles the term joins required to connect the data you want to measure with the data needed to create the map shapes.
Leveraging custom vector data sources
While Elastic Maps provides you with an incredible number of options for building and designing maps built on top of your Elasticsearch data, our goal is also to make it simple to bring in your own customizations. That’s why we’re excited in 7.9 to roll out the ability for you to connect to your own vector data sources in order to control both the context and style of your map. With the new Vector Tile option in the “Add Layer” menu, you can quickly add a URL to your vector data, add the name of the layer you would like to use, then apply stylings to match your desired aesthetic.
Another reason to zoom-in: New global administrative regions added
The ability to see provinces, cantons, states, and counties globally just got a lot easier in 7.9 with the release of a new layer for administrative regions. Thanks to this enhancement you can use your global Elasticsearch data to do things like compare website traffic coming from French departments and German states without having to add each country’s administrative levels individually. Or from a security perspective, in just a couple clicks you can have a detailed view of network activity as specific as something like Chilean administrative districts and Argentine provinces. We’ve added nearly 60 countries worth of administrative data in this release with the goal of letting you create the exact geoanalysis you need.
Alerting
Trigger ServiceNow incidents with alerts from the Elastic Stack
The release of the new alerting framework in 7.7 introduced an easy way to create alerts in the Elastic Stack and connect to them actionable outcomes like triggering emails, sending messages, or initiating escalations in third-party platforms. With 7.9 we’ve added ServiceNow as a native alerting connector, allowing you to have Elastic Stack alerts create incidents in the ServiceNow platform.
Machine Learning
Correcting polluted models: Managing snapshots
Ever been alerted by Anomaly Detection only to discover after digging in that the culprit is a benign blip in the data causing false alarms? Maybe something went down for maintenance or a project triggered an unusual, but very explainable, set of activity. Maybe it’s a one-time event so big it is warping baseline results (e.g., Black Friday traffic spike, etc.). Regardless, the result is still a chunk of data feeding into your model and skewing the results.
In 7.9, fixing this becomes incredibly simple with the new model snapshot management capability accessible directly in the Anomaly Detection user interface that lets you quickly revert back to an earlier snapshot or even just skip the problem events. Simply navigate to the model snapshots tab in job management and use your mouse to make the updates you need.
While reverting a model to a previous snapshot has been possible via our API, bringing this capability directly into the UI and also adding the ability to exclude events in a time range supports our ongoing efforts to make machine learning not just powerful, but also easy enough for everyone to use. With model snapshot management, anyone can quickly get an anomaly detection job back to an effective state. And for that reason, we also want to make it even easier to spot when and how models change (especially when those changes are big) which is why we’re adding some great new features to annotations.
Automatically generated Anomaly Detection annotations
To help give you more insight into Anomaly Detection jobs, we’re introducing a new feature with 7.9 that automatically creates an annotation when a significant change in the model calculation happens. Examples of things that might trigger this include periodicity or a trend or step change. You’ll still be able to add your own annotations and to help make sure you can tell the difference between a system-generated annotation and a user created one; we’re also updating the annotation table itself by adding search and filter capabilities.
Seeing your data: Preview charts in the Transforms wizard and data frame analytics
Whether you're transforming your data to get it ready for analysis or evaluating your data after running it through a data frame analytics job, understanding the distribution and range of your fields is important. With 7.9 we’ve added the ability to toggle-on histogram charts in the Transform and data frame analytics wizards so that you can quickly understand the shape of your data columns. All of this is designed to give you richer visibility into your data so you can spot outliers faster as well as more easily compare prediction results vs. actual values.
More machine learning integrations with Elastic Observability and Elastic Security
On top of the great new standalone machine learning capabilities added in 7.9, Elastic solutions continue to weave machine learning capabilities directly into their experiences to better support monitoring and security professionals. Elastic Security now has five new advanced anomaly detection jobs in 7.9 for AWS CloudTrail designed to spot when a malicious actor may have created, updated, or even deleted a log. If you are using the APM application within Elastic Observability you’ll notice in 7.9 that anomaly data is being surfaced in charts as well as in the service maps.
Get hands-on with Kibana 7.9
Ready to try out some of these new features? Your own personal Kibana 7.9 experience is just minutes away using the free trial option on Elastic Cloud. Downloading the latest builds is also a great way to get started checking out these new capabilities. Find us on Twitter (@elastic) or in our forum and let us know what you think. And as always, never hesitate to report any problems on the GitHub issues page.
Related blogs: