The Challenge
The Solution
Case Study Highlights
Enable fast search for content
- Provide powerful search to quickly find the right content amongst 4 billion terms
- Reduce search response time from 1-4 seconds to less than half a second
- Deliver real-time indexing so new content is available immediately
Streamline creation of content
- Auto-suggest next term(s) in a list based on existing user input
- Automatically recognize languages and provide translated terms
- Recommend related content from users with similar learning goals
The Elasticsearch team has been phenomenal in helping us. They are always available. And it is not just a random support person who is available. The experts are available. We have been extremely happy with the service we have been getting from Elasticsearch.
Searching through 4 Billion Terms
Quizlet has quickly grown to be one of the world's most popular online studying tool and, with more than 20 million unique visitors per month, they've also landed a place in the list of the top 100 US websites. Quizlet's users have already generated millions of online "study sets" – content in flashcard form that can be studied through various modes - covering more than 4 billion terms. The ability for users to search through these sets to find the right information is critical.
"Prior to Elasticsearch, we were using Sphinx for search," recounts Robert Deaton, Software Engineer at Quizlet. "We started to hit a wall in a few ways. First, we could not keep Sphinx live with the data, which is a big deal when a teacher creates a set that students need to find immediately."
When a user created a set, it was only available after Sphinx completed a full re-index, a process that could take up to 6 hours.
"The performance degraded to the point where it was not acceptable to us any longer," Deaton recalls. "Average search time per page was more than a second, and often 2 to 4 seconds, and that was just far too slow for what we wanted."
Finding and Creating Great Content
Elasticsearch replaced Sphinx as the main site search on the Quizlet site, and is also used to provide functionality – such as language detection and auto completion – to help users create new sets.
"Helping students and teachers get more use out of Quizlet requires that our users are able to find great user-generated content on our site quickly and efficiently," explains Thompson Paine, General Manager of Quizlet. "Elasticsearch is integral to this part of Quizlet."
"In addition, users need to be able to easily create high-quality content," he continues. "Elasticsearch is becoming integral to this aspect of Quizlet by enabling some exciting new features for creating content. I think this will be a key driver for our growth, and a key driver for educating kids in the US."
For example, when a user creates a set, Quizlet will be able to automatically recommend the next few terms, a whole new set of terms, or related existing sets, based on the user's input. For language sets, such as a French-to-English set, Quizlet will be able to automatically finish the French word and provide the English definition. This feature will be powered entirely by Elasticsearch's completion suggester.
Quizlet is providing the infrastructure for studying across the US right now. Elasticsearch is helping empower education in the US through Quizlet.
The more great content users put on our site, the greater service we can provide. The easier it is for users to find that great content, the better service we provide. In both of these ways, Elasticsearch is an integral part of our business.
Reducing Page Load Times from Seconds to Milliseconds
"Speed is especially important when you are dealing with high schools from across the US that have substandard Internet access to begin with," says Paine.
Quizlet's baseline goal is to keep page load time under 100 milliseconds, with search pages under 500 milliseconds. Elasticsearch enables Quizlet to maintain these service levels, noticeably faster than the site's previous search tool.
"When we deployed Elasticsearch, we saw page load times significantly reduced across the site, and the load on our database dropped 30% ," says Deaton. "Now, changes to our database are updated immediately in Elasticsearch."
"The Elasticsearch cluster handled everything well and we did not have any problems scaling this past semester, which was amazing since traffic increased during December exams alone by 40%," Deaton confirms. "This was something that we were able to set aside and not worry about, when everything else needed attention."
Keeping Services Up and Running for 20 Million Users per Month
Quizlet also uses Elasticsearch, Logstash and Kibana internally for maintaining system performance, monitoring web servers, SQL servers, PHP and other services.
"Performance and uptime is a big deal for us," Paine explains. "Being able to constantly maintain a high level of performance on the site is crucial to providing our service to students and teachers."
"Elasticsearch lets us look back at any relevant events, to pinpoint when and why any performance problems started happening," says Deaton. "Elasticsearch has been an invaluable tool in watching the performance of the site, and diagnosing problems."
"Prior to using Logstash and Kibana, we didn't have any kind of centralized logging solution," Deaton adds. "So if we wanted to look something up on a log viewer, we had to write custom scripts or grep the logs manually. Now anyone can go search the logs. It is definitely helping our development team and made it easier for more people to be involved."
