Practicing The Art of Zen(desk) With The Elastic Stack

By
Pius Fung

Zendesk is a popular lightweight helpdesk solution. In this blog post, I will demonstrate how to use the Elastic Stack to provide discovery and visualization capabilities on data that is extracted from Zendesk via a custom Logstash plugin.

Logstash: Gather What Matters (Not Just For Logs)

Logstash is often used for log collection. However, it is highly extensible and has an intuitive plugin framework developers can use to build their own plugins to fetch data from other sources. To give a perspective on what it will take to build a custom Logstash plugin: I am a software support engineer (not a seasoned developer) with some experience writing internal tools and integrations. With no prior Ruby knowledge, I was able to follow the Logstash plugin development guide and write a custom Logstash Zendesk plugin that fetches objects from Zendesk using the official Zendesk Ruby API Client. The majority of the time (~60%) was actually spent on learning the basics of Ruby and interaction with the Zendesk REST API and Zendesk Ruby API Client. The rest was writing code, testing and tweaking the implementation.

The custom/community Logstash Zendesk plugin mentioned above supports fetching various Zendesk objects like organization, user, ticket, comment and topic. The following is an example input block (refer to the Github repository linked above for additional details). You can control the types of objects fetched, fetch mode (full or incremental based on last updated time) and whether to append comments to the ticket in chronological order, etc. 

input      
{      
  zendesk       
  {         
    domain => "company.zendesk.com"         
    user => "user@company.com"         
    # password => "your_password"         
    api_token => "your_api_token"         
    organizations => true         
    users => true         
    tickets => true         
    tickets_last_updated_n_days_ago => 1         
    comments => true         
    append_comments_to_tickets => true         
    topics => true         
    sleep_between_runs => 60       
  }     
}

And the logging output of the plugin at --verbose level:

verbose level

Kibana: Open Up Your Mind

With the Zendesk data ingested into Elasticsearch via Logstash, you can use the popular Kibana tool to search, discover and visualize the data. For example, here’s a date histogram aggregated by the severity levels of tickets over time. Imagine the other possibilities such as returning the top N customers mentioning keywords like ldap, hadoop, etc.. in their comments.

Kibana tool to search

Kibana dashboard

Watcher: Help Others Be Mindful

With the commercial plugin (Watcher trial), you have endless opportunities to create watches not just to help streamline support operations (eg. tracking high severity and aging tickets, etc..). With the power of text search and aggregations, you can also monitor and alert on product quality trends.

For example, the following watch’s search input action identifies tickets with comments mentioning out of memory conditions in the past 30 days and aggregates the result by week showing the top 10 tickets sorted by version and date. This will produce a payload that can be transformed to your desire and then sent to an output action. To whom do you send this to? Imagine setting up an output action for the payload to alert the product management team using their email alias or HipChat/Slack room so they can determine if there are areas in the product that can be improved (eg. circuit breakers).

circuit breakers

Shield: Block Out Disturbances

You don’t want your customer data to be wide open. Use the commercial plugin (Shield trial) to require authentication and access control, and set up encryption to further secure your data. Classified customers? No problem, Shield provides more granular role based access control via document and/or field level security.

Found: Find Peace Of Mind

Deploying the Elastic Stack on Found is quick and simple. It literally takes just minutes to spin up a Found cluster provisioned with the latest and greatest Elastic releases. Since it is important to prevent unauthorized access to the data set, you can choose Found Premium which includes the commercial plugins mentioned above (Shield and Watcher). Once the Found cluster has started, you can simply configure the Logstash Elasticsearch output in your Logstash pipeline to reference the Found cluster’s host(s), Shield user and password, and start indexing! The data will begin to appear in the Kibana app.

Kibana app

The best part is that Found is a managed service by the creators of the Elastic Stack so you will not have to worry about the uptime of the cluster or spend hours explaining to your infrastructure team what you have deployed or waiting for them to perform operational tasks. It also takes automatic/periodic backups of your indices.

periodic backups

This means you will have more time to build impressive dashboards, work on your regular job, or walk the dog :)

As you can see, the Elastic Stack is applicable to many different use cases, not limited to logging. Its flexibility is in its name. With Found, you can also get up and running quickly!