Using Kibana for Business Intelligence

Create Your First Elasticsearch Cluster

Go to found.elastic.co and create an account. When you have created the account, log in to the Found admin console and create an Elasticsearch cluster by choosing New Cluster. With the 14 day free trial, you get an Elasticsearch cluster with 256MB reserved memory and 2GB reserved disk space. This should be sufficient for the example in this article.

Choose the region in which you want your cluster to run, then choose the Elasticsearch version. I recommend using the latest stable version. At the time of writing this is version 1.0.1. Enable the Kibana-plugin, and give your cluster a descriptive name, for example Twitter BI PoC.

Once your Elasticsearch cluster is created, note the endpoints given on the next page. You can later find the endpoints by going to the cluster's Overview page. See the figure below. You use these endpoints whenever you need to communicate with you Elasticsearch cluster. We always recommend to use the HTTPS endpoint, as traffic to and from your cluster will be encrypted.

To check that your cluster has been successfully created, go to the Elasticsearch Head dashboard. Now you should see the cluster with the name “instance-0000000000X”. Look for the text "cluster health: green" (or "yellow") in the header. Then everything should be OK.

Create the Index

The next thing we need to do is to create the index where we store our Twitter data. Run the following script from a terminal window to create an index with the appropriate mapping.

Note: Remember to swap the endpoint URL in this script with the endpoint URL of your cluster. Also note that the index is called tweets. We will use this index name later.

#!/bin/bash

curl -XPUT "https://4ae884a4b3f9cef8988b4fae82274701-eu-west-1.foundcluster.com:9243/tweets" -d '{
  "settings": {
	  "analysis": {},
	  "mapping.ignore_malformed": true
  },
  "mappings": {
	  "elasticsearch": {
		  "dynamic_templates": [
			  {
				  "created_at": {
					  "match": "created_at",
					  "mapping": {
						  "type": "date",
						  "format": "EEE MMM dd HH:mm:ss Z yyyy"
					  }
				  }
			  },
			  {
				  "text": {
					  "match": "text",
					  "mapping": {
						  "type": "string",
						  "analyzer": "simple"
					  }
				  }
			  },
			  {
				  "strings": {
					  "match_mapping_type": "string",
					  "mapping": {
						  "type": "string",
						  "index": "not_analyzed"
					  }
				  }
			  },
			  {
				  "counts": {
					  "match": "*count",
					  "mapping": {
						  "type": "long"
					  }
				  }
			  },
			  {
					"geo": {
						"match": "geo",
						"mapping": {
							"type": "object",
							"enabled": false
						}
					}
			  },
			  {
					"coordinates": {
						"match": "coordinates",
						"mapping": {
							"type": "object",
							"enabled": false 
						}
					}
			  },
			  {
					"place": {
						"match": "place",
						"mapping": {
							"type": "object",
							"enabled": false
						}
					}
			  }
		  ]
	  }
  }
}'

For the sake of this demo, you don't need to understand how this script works. It is, however, important to specify these mappings in order to populate your Elasticsearch cluster with Twitter data. If you need an introduction to mappings, see Njål Karevoll's article An Introduction to Elasticsearch Mapping.

After you have run the script, go to the Head-dashboard again and verify that an index with the name tweets has been created. The number of docs should be 0.

Setting up Your Zap

After creating a Zapier account, Make a Zap! by clicking on the link on the top of the Zapier webpage.

Twitter is already registered as an app in Zapier, and is listed in the App Directory. However, Elasticsearch is not listed as an app in Zapier, but because Elasticsearch supports the HTTP protocol, we can communicate with Elasticsearch by using the Web Hook feature. We can send and retrieve data to and from Elasticsearch using HTTP/HTTPS commands. The most common commands are GET, POST, PUT and DELETE. We will use the POST command to feed data to our Elasticsearch cluster.

"mappings": {
	  "elasticsearch": {
	  		  "dynamic_templates": [

The Basics of a Kibana Dashboard

A Kibana dashboard is made up of one or more rows, each containing one or more panels. Note that a row has a span of 12.

The header of the dashboard contains the name of the dashboard and some global menu items:

Create a Histogram of All Tweets

In the empty row we created above, click the Add panel to empty row link. Select the histogram panel type and choose a Span of 12. A span of 12 will fill the entire width of the dashboard. Type in the text created_at in the Time Field box. Click save, and voilà! You have a nice histogram showing tweets over time.

Note: The default interval on the histogram is 1 year, so when you create the histogram and when you load the dashboard you will only see a solid bar in the histogram. To change this, click the View link in the histogram and choose an interval of 1 hour.

Explore the histogram by choosing a different Time filter from the Kibana dashboard's header. You can also narrow the time selection by marking the desired time period with your mouse cursor.

Filters

Every time you narrow the selection of data, either using the histogram or by selecting a subset of data using other panels, a filter is created. You can see all filters by clicking on the Filtering button, just below the Query bar.

Often you need to remove some or all of these filters to display the desired data in Kibana. Personally, I like to leave the Filtering section open while working with Kibana.

Filters can be toggled on or off using the checkbox, and some filters can even be edited directly in the Filtering section. Time filters can not be edited.

Remove all filters.

Twitter Users

Now we are going to create a panel that shows some statistics on twitter users that have tweeted about Elasticsearch lately.

Inspecting Tweets

Take a look at the panel in the bottom row. This panel contains all the tweets that match your filters at any given time. If you have removed all filters, this panels will show every tweet in your Elasticsearch cluster. In the left column, you see all the Fields that a tweet may contain. Not all tweets contain all Fields. In the right column, you see the content of the tweets. If you click on a tweet you get a tabular overview of all Fields and corresponding Values for that tweet.

By checking off fields in the left column, you can toggle which values that will be displayed in the right column. Now check off the following fields: created_at, lang, retweet_count, text, user.name and user.screen_name. If you wish, you can change the order of the fields at the top of the right column.

Now go to the Twitter Users panel and click on the user imarshut's bar, then use the mouse cursor and highlight a short period where he tweeted about Elasticsearch. Now you can go to the bottom panel and look at imarshut's tweets during this period.

Retweet Characteristics

We'll finish off by creating one last panel that will enable us to look at the retweet characteristics of Elasticsearch related tweets.

In row two, add a panel of type terms, with a span of 6. Call this panel Retweet Count. Type retweet_count in the Field-box, set the Length to 20 and set Order to reverse_term. Then select table as the style and remove the checkmarks from Missing and Other.

You will now get a panel that looks something like this:

The Term here is the same as the retweet_count field in a tweet. Accordingly, the Count column shows the number of tweets that has been retweeted the number of times displayed in the Term column. For instance, we see that 1 tweet has been retweetet 346 times and 3 tweets have been retweeted 41 times. Click on the magnifying glass on the first line and see which tweet this is.

Remove the filter. Then click on a specific user in the Twitter Users panel and see how much this user has been retweeted. For the user¹ the retweet statistics would look something like this:

One tweet has been retweetet 20 times, and 19 tweets have not been retweeted at all in the given period.