Developing new Kibana visualizations

By
Peter Pišljar

Creating new visualization types

In 6.0 we made some significant changes to the visualize API and how visualizations are implemented. We also released documentation with it which should provide basic description of the interface. This tutorial should help you get started quickly.

Let us discuss some terms first

We will build a visualization similar to Kibana's built-in Metric visualization, except we will try to keep it simple without any advanced options.

When you start with creating new metric visualization in Kibana you are presented with a screen looking something like this:

image_0.png

On the left side we have the default editor and on the right side there is the actual visualization.

If you look at the side editor you notice it consists of few parts, which are numbered in the image above.

  1. There is the data definition tab. This tab will help us select the right data from elasticsearch.
  2. The first part of data selection tab lets us define the metrics we are interested in. This could be a Count, Min, Max and so on. It allows us to add multiple metrics, and each metric will have some specific configuration, for example, like the field name being used.
  3. The second part of data selection tab allows us to define something we call buckets. It describes how elasticsearch should split data before calculating the actual metrics. Here we can add bucket aggregations like Terms or Date Histogram. Some visualizations allow as to define multiple buckets.
  4. The options tab allows us to configure additional options of our visualization like font size for example.

Bootstrapping the plugin

New visualizations are just Kibana plugins with the right uiExports. We won't go over all the details. Instead, let's start with creating a folder inside Kibana's plugins directory. We will use a super smart name of 'test_vis'.

  • Create a new directory named 'test_vis' under kibanas plugin directory.
  • Create package.json file inside that folder to describe our plugin
{
  "name": "test_vis",
  "version": "kibana"
}
  • And finally create the index.js file. This pretty much just points to the main file of our visualization.
export default function (kibana) {
  return new kibana.Plugin({
    uiExports: {
      visTypes: [
        'plugins/test_vis/test_vis'
      ]
    }
  });
}

Visualization definition

The test_vis.js file referenced in above index.js file will define a new visualization type and register it. As it is described in the documentation there are multiple factories depending on the rendering technology you are using. You could even extend it with your own. But in this tutorial we will use the base visualization factory, which is the one we recommend to use. Base visualization type does not make any assumptions about the rendering technology.

Let's look at our example test_vis.js.

First we import our styles and option template (we will talk about it later) as well as our visualization controller, which will take care of rendering the visualization. 

import './test_vis.less';
import optionsTemplate from './options_template.html';
import { VisController } from './vis_controller';
import { CATEGORY } from 'ui/vis/vis_category';
import { VisFactoryProvider } from 'ui/vis/vis_factory';
import { VisTypesRegistryProvider } from 'ui/registry/vis_types';
import { VisSchemasProvider } from 'ui/vis/editors/default/schemas';

function TestVisProvider(Private) {
   const VisFactory = Private(VisFactoryProvider);
   const Schemas = Private(VisSchemasProvider);
}

The createBaseVisualization method of the VisFactory accepts the definition object. Take a look at the documentation to get a better idea of what these properties do. The important parts are that we define the name for our visualization, the controller class which will take care of rendering, the default configuration of the visualization itself and the configuration for the editor. 

  return VisFactory.createBaseVisualization({
    name: 'test_vis',
    title: 'Test Vis',
    icon: 'fa fa-gear',
    description: 'test vuis',
    category: CATEGORY.OTHER,
    visualization: VisController,
    visConfig: {
      defaults: {
        // add default parameters
        fontSize: '30'
      },
    },

We will be using the default Kibana editor in this tutorial. This is the side editor you see in many Kibana visualizations. We need to provide the optionsTemplate, which should be the angular template for the options tab. We also need to provide schemas definition, which tells which aggregations can be configured.

In the below example our schema definition contains a single object of group metrics. The minimum is set to one (so users will have to configure at least one metric), some aggregations are excluded from the list and the default configuration is provided. 

    editorConfig: {
      optionsTemplate: optionsTemplate,
      schemas: new Schemas([
        {
          group: 'metrics',
          name: 'metric',
          title: 'Metric',
          min: 1,
          aggFilter: ['!derivative', '!geo_centroid'],
          defaults: [
            { type: 'count', schema: 'metric' }
          ]
        }
      ]),
    }
  });
}

At the end we need to register our provider function with the VisTypesRegistryProvider. 

// register the provider with the visTypes registry
VisTypesRegistryProvider.register(TestVisProvider);

Visualization Options

In the visualization definition visConfig we set the default options for the visualization. In this case, fontSize. We could provide more configuration options there and nest them as we like. We also need to provide the UI for changing them. The optionsTemplate property on the editorConfig allows us to provide an angular template to do just that. We could also provide multiple option tabs.

Our options_template.html looks something like this. Note how we reference the fontSize parameter with the vis.params.fontSize variable. 

<div class="form-group">
  <label>Font Size - {{ vis.params.fontSize }}pt</label>
  <input type="range" ng-model="vis.params.fontSize" class="form-control" min="12" max="120" />
</div>

But what if we need more control over it ? Instead of providing HTML we could also provide an angular directive: 

import './options_directive';
…
   optionsTemplate: '<myvis_options_directive></myvis_options_directive>'
...

Visualization Controller

The last missing part is the visualization controller. This is the actual code that will render the visualization to the screen. We need to create a class with render and destroy functions. The constructor will get the DOM element to which visualization should be rendered and the Vis object. It should prepare everything it can before the data is available. 

class VisController {
  constructor(el, vis) {
    this.vis = vis;
    this.el = el;
    this.container = document.createElement('div');
    this.container.className = 'myvis-container-div';
    this.el.appendChild(this.container);
  }

The destroy function needs to clean up. Here we just remove all the DOM. we should also remove any hanging listeners or pending timers. 

  destroy() {
    this.el.innerHTML = '';
  }

The render method will receive the data object along with status object. It will be called every time a change happens which requires an update of visualization like changing time range, filters, query, uiState, aggregation, container size or visualization configuration.

Here we re-render whole visualization every time, but this is not the most optimal behaviour. Your code could inspect the status object to find out what exactly triggered the render call (was it a change in time range for example?) and update accordingly to that. For example a change in container size should probably not require to redraw the whole thing. 

  render(visData, status) {
    this.container.innerHTML = '';
    return new Promise(resolve => {.
      resolve('when done rendering');
    });
  }
};
export { VisController };

Now we need to provide our visualization implementation. First, we’ll extract the data. Note how we can’t rely on aggConfig to be present, but we should always check if it is and use fieldFormatter to correctly format the value in such case. As we didn’t provide a responseHandler to our visualization it will use the default response handler, which returns data in tabular format. 

  const table = visData.tables[0];
  const metrics = [];
  table.columns.forEach((column, i) => {
    const value = table.rows[0][i];
    metrics.push({
      title: column.title,
      value: value,
      formattedValue: column.aggConfig ? column.aggConfig.fieldFormatter('text')(value) : value,
      aggConfig: column.aggConfig
    });
  });

And at last we add the elements to the DOM: 

  metrics.forEach(metric => {
    const metricDiv = document.createElement('div');
    metricDiv.className = 'myvis-metric-div';
    metricDiv.innerHTML = `<b>${metric.title}:</b> ${metric.formattedValue}`;
    this.container.appendChild(metricDiv);
  });

Using parameters

In our visualization definition we defined the fontSize parameter. We can access it inside the visualization thru vis.params: 

metrics.forEach(metric => {
    const metricDiv = document.createElement('div');
    metricDiv.className = 'myvis-metric-div';
    metricDiv.innerHTML = `<b>${metric.title}:</b> ${metric.formattedValue}`;
    metricDiv.setAttribute('style', `font-size: ${this.vis.params.fontSize}pt`);
    this.container.appendChild(metricDiv);
  });

Adding bucket configuration

Many kibana visualizations allow you to define a bucket aggregation to then show you a metric for every bucket. For example, this is especially useful with date histograms where each date could be one bucket.

To add bucket support to our aggregation we first need to tell the editor that we support buckets: 

    editorConfig: {
      optionsTemplate: optionsTemplate,
      schemas: new Schemas([
        {
          group: 'metrics',
          name: 'metric',
          title: 'Metric',
          min: 1,
          aggFilter: ['!derivative', '!geo_centroid'],
          defaults: [
            { type: 'count', schema: 'metric' }
          ]
        }, {
          group: 'buckets',
          name: 'segment',
          title: 'Bucket Split',
          min: 0,
          max: 1,
          aggFilter: ['!geohash_grid', '!filter']
        }
      ]),
    }
  });
}

And we need to handle them in our visualization controller's render method: 

  const table = visData.tables[0];
  const metrics = [];
  let bucketAgg;
  table.columns.forEach((column, i) => {
    // we have multiple rows … first column is a bucket agg
    if (table.rows.length > 1 && i == 0) {
      bucketAgg = column.aggConfig;
      return;
    }

    table.rows.forEach(row => {      
      const value = row[i];
      metrics.push({
        title: bucketAgg ? `${row[0]} ${column.title}` : column.title,
        value: row[i],
        formattedValue: column.aggConfig ? column.aggConfig.fieldFormatter('text')(value) : value,
        bucketValue: bucketAgg ? row[0] : null,
        aggConfig: column.aggConfig
      });
    });
  });

Adding events

What about handling click events? Easy! We can add a click handler to our DOM elements: 

  metrics.forEach(metric => {
    const metricDiv = document.createElement('div');
    metricDiv.className = 'myvis-metric-div';
    metricDiv.innerHTML = `<b>${metric.title}:</b> ${metric.formattedValue}`;
    metricDiv.setAttribute('style', `font-size: ${this.vis.params.fontSize}pt`);

    metricDiv.addEventListener('click', () => {
     if (!bucketAgg) return;
     const filter = bucketAgg.createFilter(metric.bucketValue);
     this.vis.API.queryFilter.addFilters(filter);
    }); 
    this.container.appendChild(metricDiv);
  });

When the click event fires, we create a filter for the selected value. We then add this filter to the filter-bar using the this.vis.API.queryFilter.addFitlters method.

Using kibana visualizations in your plugin

In 6.0 using existing kibana visualizations in your own plugins has become much easier. We also added documentation around it.

So let’s create a new plugin. There is already a community resource available on how to do that, so we are not going in depth on that. Let’s quickly review the steps.

  • Create a folder inside the kibana/plugins/ for your new plugin. We will call it test_visualize_app.
  • Define package.json in this folder describing your plugin
  • Create index.js in this folder with your app definition

Index.js file will create a new plugin object and define the uiExports for the plugin. In uiExports we will define our app. If we want to be able to use existing Kibana visualizations we need to tell Kibana which modules we will use. Also we need to inject some variables from Kibana.

Here is the example index.js file for my plugin: 

export default function (kibana) {
  return new kibana.Plugin({
    uiExports: {
      app: {
        title: 'Test Visualize',
        description: 'This is a sample plugin',
        main: 'plugins/test_visualize_app/test_vis_app',
        uses: [ 
          'visTypes',
          'visResponseHandlers',
          'visRequestHandlers',
          'visEditorTypes',
          'savedObjectTypes',
          'spyModes',
          'fieldFormats',
        ],
        injectVars: (server) => {
           return server.plugins.kibana.injectVars(server);
        }
      }
    }
  });
}
  • Create the public folder inside your plugin folder
  • Create the main js file (we named it test_vis_app.js as you can see in the index.js file referenced above) and define the routes and controller for the plugin.

Here is a minimal example of test_vis_app.js: 

require('ui/autoload/all');
require('ui/routes').enable();
require('ui/chrome');
import './test_vis_app.less';
import './test_vis_app_controller.js';
const app = require('ui/modules').get('apps/test_app', []);
require('ui/routes').when('/', {
  template: require('./test_vis_app.html'),
  reloadOnSearch: false,
});

We will come back to the template HTML file, the .less file for the styles and the controller file later.

Embedding saved visualizations in your plugin

Our first application will be really simple. It will have a select control with all the saved visualizations on top and it will render the selected visualization in the main area.image_1.png

Getting a list of saved visualizations

Let's create the test_vis_app_controller.js. This will define a simple angular controller holds the main logic of our application.

First we need to load the visualizeLoader which will help us get the list of saved visualizations as well as embedding those visualizations to our DOM.

To load saved visualizations we will use getVisualizationList method, which will return a list of all saved visualizations. Each object in the list will contain:

  • title <string>: title of the saved visualization
  • id <string>: unique id of the saved visualization
  • type <object>: vis type of the saved visualization
import { getVisualizeLoader } from 'ui/visualize/loader';
const app = require('ui/modules').get('apps/kibana_sample_plugin', []);
app.controller('TestVisApp', function ($scope) {
  $scope.visualizationList = null;
  $scope.selectedVisualization = null;
  let visualizeLoader = null;
  getVisualizeLoader().then(loader => {
    visualizeLoader = loader;
    loader.getVisualizationList().then(list => {
      $scope.visualizationList = list;
    });
  })
});

We also need to add a template file test_vis_app.html to show our dropdown and prepare a placeholder where we will render visualizations. This is the file we referenced in the test_vis_app.js above. Note how we set ng-controller to use the controller we just defined. 

<div class="test-vis-app app-container" ng-controller="TestVisApp">
  <div class="test-vis-app-selector">
    <select 
       ng-options="item.id as item.title for item in visualizationList"       
       ng-model="selectedVisualization"
     ></select>
  </div>
  <div class="test-vis-app-visualize"></div>
</div>

If there are any saved visualizations in this Kibana instance you should get the dropdown filled with them. The div with class test-vis-app-visualize will be used as a container where we'll load our visualizations.

Its very important that we set this div to use the flex display, else visualization will not render correctly, as well to set the flex display on visualization and visualize directives. Let's update our test_vis_app.less: 

.test-vis-app, .test-vis-app-visualize, visualize, visualization {
  display: flex;
  flex: 1 1 100%;
}

Embedding the visualization

To embed the visualization we will also use the VisualizeLoader, more specifically its embedVisualizationWithId method, to which we need to provide the DOM element to which it should render, the visualization id as well as all the other parameters that you can pass to visualize directive. As we don’t have time picker in our plugin we will need to provide the time range.

The timeRange accepts regular time stamps as well as the date math expressions. 

  const visContainer = $('.test-vis-app-visualize');
  const timeRange = {
    min: 'now-7d/d',
    max: 'now'
  };
  $scope.$watch('selectedVisualization', (visualizationId) => {
    if (!visualizationId) return;
    visualizeLoader.embedVisualizationWithId(visContainer, visualizationId, {
      timeRange: timeRange
    });
  });

And this is it. Reload Kibana and test it out.

Passing additional options to the visualization

In many scenarios you will have a requirement to pass additional options to visualization like the timeRange we mentioned above or a showSpyPanel option. 

visualizeLoader.embedVisualizationWithId(visContainer, visualizationId, {
      timeRange: timeRange,
      showSpyPanel: false
    });

Take a look at visualize directive documentation to find out about additional options.

How can I know when visualization is done rendering?

The embedVisualizationWithId function will return a promise (WARNING: in 6.2 this behaviour will change and the method will return the handler directly). Once the promise is resolved the visualization is done rendering. The promise gets resolved with a handler object which has a destroy method which you should call when you want to clean up: 

visualizeLoader.embedVisualizationWithId(visContainer, visualizationId, {
      timeRange: timeRange
    }).then(handler => {
      console.log('done rendering')); 
      ….
      handler.destroy(); // call to clean up
    });

Embedding a saved object

Sometimes you will need to have more control over the saved visualization (maybe you want to modify it slightly prior to rendering, add additional filters or apply a query) or you might not have a saved visualization at all but have used a different way to obtain a saved object.

Let's assume you want to load a saved visualization but modify its search source prior to rendering it. We will use a savedVisualizations service to load the visualization and then embedVisualizationWithSavedObject method to embed it into our DOM. 

import { FilterManagerProvider } from 'ui/filter_manager';
app.controller('TestVisApp', function ($scope, Private, savedVisualizations) {
   const filterManager = Private(FilterManagerProvider);
   $scope.$watch('selectedVisualization', (visualizationId) => {

    if (!visualizationId) return;
    savedVisualizations.get(savedVisualizationId).then(savedObj => {
          const filters = filterManager.generate('response.raw', '200');
          savedObj.searchSource.get('filter').push(filters[0]);
          visualizeLoader.embedVisualizationWithSavedObject(visContainer, savedObj, {
            timeRange: timeRange
          });
      });
  });
});

Using kibana visualizations

Above we looked into how we can render a saved kibana visualization inside our plugin. But what about using kibana visualization types with our own data and configuration, without actually saving anything?

For this purpose we are going to use <visualization> directive. We will need to import Vis and visualize: 

import 'ui/visualize';
import { VisProvider } from 'ui/vis';

In this example we will render a simple tag cloud. Tag cloud uses the tabify response handler, which means we need to provide it data in such format. Lets create our data structure: 

$scope.myVisData = {
    tables: [{
      columns: [
        { title: 'Tag' },
        { title: 'Count' }
      ],
      rows: [
        [ 'test', 100 ],
        [ 'tag', 150 ],
        [ 'for', 200 ],
        [ 'tagcloud', 10 ],
      ]
    }]
  };

As you can see the data structure is very simple. On top level there is an object with tables property. There can be multiple tables listed but for this example we use just one. Each table has columns property which is an array of columns. Each column object has a title property.

Each table also has rows property, which is an array of rows, where each row is an array of cell values.

We will also need to provide the configuration for visualization. In this example we are gonna keep it to the minimum: 

  const visConfig = {
    type: 'tagcloud'
  };

Now we can create our visualization object: 

  const Vis = Private(VisProvider);
  $scope.myVis = new Vis('logstash-*', visConfig);

All we are missing is the template to render this: 

<visualization vis="myVis" vis-data="myVisData"></visualization>

image_2.png

Or a similar example with region maps? We first need to add the serviceSettings dependency to our controller: 

app.controller('TestVisApp', function ($scope, Private, savedVisualizations, serviceSettings) {

The serviceSettings service is used to load the layers that the map can use.

Now we can load the available region map layers: 

  serviceSettings.getFileLayers()
    .then(function (layersFromService) {

And prepare our Vis and data: 

    $scope.myVisData2 = {
      tables: [{
        columns: [
          { title: 'Tag' },
          { title: 'Count' }
        ],
        rows: [
          [ 'GB', 100 ],
          [ 'FR', 150 ],
          [ 'DE', 200 ],
          [ 'ES', 10 ],
        ]
      }]
    };
  const visConfig2 = {
    type: 'region_map',
    params: {
      selectedLayer: layersFromService[1],
      selectedJoinField:layersFromService[1].fields[0]
    }
  };
  $scope.myVis2 = new Vis('.logstash*', visConfig2);
});

<visualization ng-if="myVis2" vis="myVis2" vis-data="myVisData2"></visualization>

Where to go from here?

You can get all the code used above on github. Here is the first part, creating your own visualization. And here is the second part, using Kibana visualizations in your own plugin.

We linked to the documentation in quite a few places above. And if you need additional help don’t hesitate to contact us on discuss. However to go more in depth you will probably need to dive into Kibana source code.