​Building a Better Search Experience in Kibana

By
Matt Bargar
Editor's Note (November 9, 2020): This blog has been updated due to missing images.

Search is important. It’s kind of the reason why the Elastic Stack exists. It’s why a query bar sits atop almost every app in Kibana. Search is an integral part of the experience. For such a core part of Kibana, the current query bar is somewhat lacking. It gets the job done. Some power users can really make it sing. We’re missing some basics though. Autocomplete would be nice. Scripted field support would be good. Some dynamic, in page help wouldn’t hurt either, instead of a static link to the docs.

Improvements to the Kibana search experience have been a long-standing ask, but there was a common issue holding us back. The (shocking) truth is Kibana knows almost nothing about your query. Whatever you type into that text input is stuffed into an Elasticsearch Query String Query with a minimum of fuss. This lack of introspection is the source of a whole slew of annoying issues involving the query bar. It’s why we don’t provide any meaningful autocompletion or error detection. It’s why scripted fields don’t work. It’s why we don’t provide better in-page documentation. The list goes on.

In order to make search a more pleasant experience, Kibana needed to get smarter. It needed to understand the queries users type in. We thought about parsing the existing syntax, but the Query String Query wasn’t built specifically for Kibana. It aims to solve a different set of problems. Kibana deserves its own query language, one built from the ground up with Kibana users in mind.

Introducing Kuery

Today we’d like you to meet Kuery, Kibana’s experimental new query language. Kuery was built to simplify Kibana’s search experience and provide a foundation for powerful new features. This initial version is fairly basic but it already solves some of the issues we discussed above. 

  • Simple queries will look familiar to the experienced Kibana user. If you just type in some search terms, we’ll look for those across all of your fields.
  • A search can also be restricted to a single field in the usual way.
  • If you’ve used range queries, boolean operators (AND/OR), negations, or grouping, these all work like they did in the old query language with some minor syntactic tweaks.

You may be thinking: “Ok… what gives? You built this great new query language and… it looks just like the old one”. Ok ok, hold on. Here’s where it gets interesting.

Every query deserves a name

Every query type is actually a named function. In the examples above we were using a shorthand operator syntax provided for the most common query types. We wanted simple queries to remain simple and concise. But every query type also has a name. We can rewrite the second example (response:200) like so:

The colon operator is an alias for the is function. We can rewrite the range query in the third example in a similar manner: range(bytes, gte=0, lte=8000). When we say every query is a function we really mean it, even the boolean operators can be written as functions. Hence why we give you the shorthand operators. The important thing to note is that every query has a canonical name. That gives us an easy way to extend the language to support any arbitrary number of new query types. Want a geo_bounding_box query? It’s yours!

Check out the full list of currently supported query types in the docs. In the future we’d like to make these functions pluggable so users could even define their own custom query types.

Unifying the bars

In addition to the query bar, Kibana also has a filter bar. The difference between the two can be confusing since they essentially serve the same purpose. Each is used to filter your result set, but they have slightly different sets of capabilities.

Having multiple ways to achieve the exact same result tends to lead to a confusing experience. One of our goals with Kuery was to eliminate this confusion by removing the filter bar and powering up the query bar.

Interactions in the UI now update your query instead of adding filters in a separate bar. Not only does this reduce confusion, it also teaches new users how to use the query language as they play around with Kibana. Most of the filter actions like negate, delete, and edit (which were represented by obscure icons on mouse hover) can now be achieved intuitively by simply editing the query string.

If you liked the interactive experience provided by the filter UI, don’t fret. We have plans to integrate the filter editors directly into the query bar. Designs are still in the works but the goal is to make the query bar useful for beginners and advanced users alike.

Scripted fields

Ever tried to use a scripted field in Kibana’s query bar? Disappointed by the lack of results?

Scripted fields are a Kibana concept built on top of Elasticsearch’s scripting capabilities. If you think back to what we said at the beginning of this post, our old queries were simply passed through unmodified to the Query String Query. Elasticsearch doesn’t know anything about Kibana’s scripted fields, so it just looks like a query for a non-existent field. Since Kibana knows how to parse the new query language we can detect scripted fields and build a proper script query that Elasticsearch will understand. Now scripted fields will just work.

The future is what you make of it

So that’s what Kuery looks like today. Hopefully we’ve whet your appetite. This is only the beginning, a foundation we can build all kinds of cool stuff on top of. We’ve got tons of ideas for the future but more than anything we’d like to hear from YOU. What did you like or hate about the old query language? What features are you dying to have? This is an opportunity to build the Kibana query language you always wanted.

Here’s a smattering of ideas to give you some inspiration:

  • Autocomplete for query syntax and field values
  • Seamless migrations of saved searches
  • Dynamic help text
  • Go beyond search, define visualizations directly in the query string

Download the 6.0 beta, give Kuery a spin, and let us know what you think. Become an Elastic Pioneer and you’ll also have a chance to win some cool swag. Check out the docs for a full explanation of the new language’s current capabilities and details on how to enable it in Kibana’s Advanced Settings.