Powering a BI application with Elasticsearch at Yodle

I’ve attended two Elastic{ON} Tours at this point, both in NY in 2015 and 2016.  In 2015, my team and I were about to start work on a large project that had Elasticsearch at its core, and in 2016 I had the opportunity to give a talk at the conference about that very project.

I really enjoy the Elastic{ON}Tour conferences as I’m able to hear about what’s coming next from the team at Elastic, learn how other attendees have leveraged Elasticsearch, and have conversations that spark ideas for what we could be building next. For example, I had the opportunity to speak with members of the Prelert team about ways we may be able to leverage Prelert to augment our existing alerting infrastructure or perhaps even use it in future versions of our own product.

The folks at RedOwl gave a great talk about how they leveraged Elasticsearch to detect unwanted behavior in an enterprise.  I enjoyed this talk as the history of their architecture in some ways mirrored the path we took when building out our own product.  We both started with a Hadoop based solution and eventually made our way over to using Elasticsearch.

Another great talk was given by Elastic about the 5.0 release and their product roadmap.  It’s great to hear about all the new features that we’ll be able to incorporate into our product going forward.  Given that our use-case is heavily numeric we’re looking forward to the potential speed increase made possible by the latest changes in Lucene.  Additionally, while we’ve experimented with groovy scripts in the past, enabling the painless scripting language by default will provide a new option for us when deciding how to structure our documents and queries.

When Elastic approached me about speaking at Elastic{ON} Tour I jumped at the chance.  It was a great opportunity to discuss our use case of Elastic and tell the story of Centermark.  Centermark is a distributed marketing automation platform that Yodle provides to franchise and multi-location businesses.  It facilitates the management of the marketing efforts for the hundreds of geographically distributed businesses in a franchise network providing them with actionable insights about the marketing and operations of their franchise.

In the talk I walk through the evolution of the Centermark product as well as the history of its technical architecture that led us to Elasticsearch, particularly appreciating the speed with which it tears through the data. It’s that speed that allows us to deliver business insights to our customers. We’ve been working on Centermark for a while, and as tends to be the case, we have looked back on some of our architectural decisions more fondly than others. In particular, back in 2013 we made a decision to use Hadoop to pre-aggregate metrics which, while helping us achieve our goals around pageload speed, made it difficult to provide our customers with the flexibility that they needed. Publicly walking through the shortcomings of the early versions of our product was a bit exposing, but also offered a sense of atonement. Ultimately I believe this is what makes our journey interesting as it’s an honest story about how we found the right architecture for our product using the Elastic Stack, but not on the first try.

For more information about Yodle or the Centermark product, check out these links: yodle.com, centermark.yodle.com, yodlecareers.com, yodletechblog.com.

Mark Drago joined Yodle in 2010 and is now a Director of Engineering. He has been leading the part of the Yodle engineering team dedicated to the franchise market for the past 4 years.