Being Exceptionless with Elasticsearch

Exceptionless is an open source technology company that deals in real-time event reporting and logging, focusing specifically on error reporting. Our goal is to help the world's developers improve their applications and user experience by being… well... exceptionless.

To be Without Exception

Exceptionless provides real-time error reporting for JavaScript, Node, ASP.NET, Web API, WebForms, WPF, Console, and MVC apps. Everything is pulled into a dashboard that organizes each event into a simple yet robust and fully-featured view that gives you quick access to actionable data that will help your app become exceptionless!

We’ve learned quite a bit since the first version of Exceptionless, and we’d love to share our story on why we ended up migrating to Elasticsearch with great success.

In the first version of Exceptionless, we were using MongoDB for our primary document data store. We chose it for it’s ease of use, ability to scale out instead of up, and the cross platform story. It worked pretty well until we started to grow, then we began having issues scaling several features, such as real-time dashboards with automatic time zone shifting, backups, restores, and storage.

Real-time Dashboards

When we first built our real-time based dashboards with MongoDB, we had to do a ton of research on timezones, which is a monumental task in and of itself. We found that we would need to pre-aggregate data into 15 minute buckets of time (to account for different time zone offsets) into separate collections, in addition to our event collections. This would allow us to quickly return statistics data for any 15 minute time frame, but it also added the expense of additional storage and processing time needed to shift the buckets of stats into the desired time zone with each request.

We were really excited to learn that, by moving to Elasticsearch, we would not only solve this difficult problem using the date_histogram, but that it would also allow us to:

• Get live statistics for any timezone offset for any time range
• Index much faster, since we don’t need to pre-aggregate data
• Add new visualizations without worrying about having the pre-aggregated data
• Maintain a lot less code
• Remove pre-aggregated statistic collections

Fig. 1: Exceptionless Dashboard

Fig. 2: Exceptionless Kibana 3 Dashboard

Backup and Restoring

One of the huge pain points prior to moving to Elasticsearch was backing up MongoDB. There isn’t a lot of documentation for doing so in production, which leads users to bring down the cluster to do a quick backup via MongoDump, zip the backup, and upload it to some destination. The other option is to pay monthly fees and run an extra node for the MMS service, which does oplog backup and restore.

One of the most underrated features of using Elasticsearch is backups and restores! It has been the most pleasant experience to configure and use. We setup a Cron job on one of our nodes to run curator for cleanup, and to run a backup to azure blob storage via the cloud-azure plugin. The nicest thing about this is you can quickly choose exactly what index to restore to a development machine if you wish to debug production data, while keeping your data secure and backed up in the cloud.

Disk Space

After migrating our data we found that it used about one-fifth the disk space as MongoDB (please note that WiredTiger wasn’t released until sometime after we migrated). This allows us to cut our storage costs across the board.

Scaling Out

There is nothing worse than waking up to find that a VM reboot of one of the nodes or adding a new node caused your MongoDB cluster configuration to become corrupted. This is a very scary situation we once had to deal with and resolve by restoring the cluster configuration. We always found that dealing with the MongoDB cluster configuration is something you don’t touch if it’s working.

We love how easy it is to setup and manage an Elasticsearch cluster. It’s a dream come true! No matter how you want it to run, it’s just a simple configuration setting or REST call away! Best of all, you can share your configuration with others to get feedback!

Conclusion

We are very happy with the move to Elasticsearch and continue to receive additional value from it with every release. We really like that it’s easy to setup and is open source friendly!

Blake Niemyjski is a full time open source software architect and private pilot. Since the late 90’s, Blake's curiosity has been focused around developing software that helps the masses. As a hobbyist developer turned student and professional, Blake's passion is to further his knowledge and create technologies for the future.