Lexer: Powering the Search for Customer Insight

Lexer has built a powerful Customer Data Platform (CDP) which unifies data from a range of sources to provide clients with actionable insight into customers and prospects. These data sources enrich Lexer's client data with other data such as social media posts and consumer financial transaction data. This provides Lexer's clients with a powerful cross-channel view of each customer and prospect so they can better serve and win new customers.

Lexer's Customer Engagement Hub enables clients to take advantage of this data to engage with stakeholders in a more human way. For example, they can provide faster, more contextualized customer care. They can also personalize the customer experience of websites and apps, and target the right audiences with the right information.

As Lexer has evolved, the sources of data it brings together have multiplied, as have the tools it provides its clients. The Elastic Stack has helped power this evolution and is a core component in how Lexer and its clients search, analyze, and visualize data.

The What

Lexer was founded in 2010 and began using Elasticsearch that same year. Its Customer Data Platform is built on Elasticsearch, using the search and analytics engine to power actionable insights from all its data sources.

Initially, the Customer Data Platform managed around millions of pieces of social content per day, capturing mentions of its clients that were publicly posted to channels such as Facebook or Twitter. At that time, the Customer Data Platform was predominantly used for social listening. However, Lexer's clients soon wanted the ability to do more with their data such as engage customers on social media and understand their purchasing habits. Lexer responded by offering new tools to help clients with customer engagement and discovery.

Aaron Wallis of Lexer said that by connecting data from a number of 1st, 2nd and 3rd party sources the Customer Data Platform essentially created human profiles.

"Our data machine has grown bigger and bigger over the last couple of years. We are bringing in Wi-Fi, demographic, movement, and other unstructured datasets from a variety of partners and then layering it with our clients' data, including transaction information and customer identifiers like emails and customer IDs. So we can start to build unique profiles of each customer which can be used to personalize offers and communication," said Aaron.

These insights helps Lexer's clients engage with customers more personally and in real-time, enabling them to develop smarter strategies for activating new prospects. For example, Lexer's clients can better segment their audiences and understand the lifetime value of different customer groups. They can also identify those most likely to make a purchase and contact them at the right time with the right offer.

These capabilities have increased the value that Lexer delivers to clients and opened the door to new opportunities in Australia and the US. However, powering these capabilities and scaling the platform has come with some challenges. As Lexer ingested more and more data into its Customer Data Platform, the speed and robustness of the platform came into question.

The Why

As its data handling needs grew, Lexer made the decision to upgrade from Elasticsearch 2.3.4 to Elasticsearch 5.4.1. With Elasticsearch already underpinning its successful Customer Data Platform, no other solution made sense, and upgrading to version 5.4.1 would provide the robustness and performance that Lexer needed as it continued to scale.

"Elasticsearch has been evolutionary for our business. It's saved us hundreds of hours in development time and helped us to build a platform that can easily mine massive volumes of data in real-time," said Aaron. "There was no question that we still needed Elasticsearch, but it was time for us to upgrade."

The How

Before Lexer could upgrade, it needed to move its data into smaller indexes. It had a single index of 2.8 billion social posts, comments, messages, articles and blogs which was resulting in a recognizable lag in query performance for their customers. To combat this, they transitioned to smaller indices tailored to the event type. This paved the way for a smooth migration and increased the speed and efficiency of client search requests. It also allowed for increased scalability.

Next, Lexer updated its client side libraries to take advantage of native Elasticsearch features such as geolocation searches.

Once testing was complete, it moved on to the upgrade using Elasticsearch's snapshot and restore facility. This facility allows users to copy index data and then restore that data very quickly into a different cluster. In Lexer's case, it copied the index data to Amazon's Simple Storage Service (S3) and completed a full disaster recovery scenario, restoring its entire 2.8 billion object data set within just 40 minutes.

"We not only upgraded our cluster, but created a robust system that backs up our entire data set every 15 minutes," said Aaron. "If something goes wrong, we can restore our entire cluster in 30 to 40 minutes whereas in the past, it could have taken us up to a week."

The benefits of the upgrade were immediate. General query performance improved 13% while performance of features in the Customer Engagement Hub improved up to 40%.

These performance enhancements translate to a time savings of 6 hours of waiting time per day across all of Lexer's clients.

The upgrade has also reduced the cost and burden of maintenance. Aaron said if a node were to fail in the middle of the night, the team can simply create a new one and get back to sleep. They don't worry about these types of issues like they did in the past. Also, the clusters have been configured in a way that future upgrades can be done with ease. This will lower the investment of time and costs and give Lexer more flexibility to take advantage of future versions.

The process of the upgrade was made easier with the support provided to Lexer as part of its Platinum subscription. During planning, Lexer was able to call on Elastic for support in building out its proof of concept for the upgrade and to confirm architecture requirements. This not only saved Lexer time but helped ensure a seamless upgrade.

Lexer continues to benefit from Platinum support, learning and adopting best practices that have improved the reliability and performance of its Customer Data Platform. Additionally, they've adopted more components of the Elastic Stack, including Logstash, Kibana, and Monitoring from the X-Pack. Logstash and Monitoring are particularly useful for monitoring the health and security of Lexer's cluster and infrastructure.