4x Performance Increase
Drastically Reduced IT Spend
Less Time Spent on Maintenance
After switching to Elastic Cloud from FAST ESP, Guidestar by Candid went from managing their clusters on a daily basis to checking cluster health once a month.
Meeting Customer SLAs with Elastic Support
Whether it's architecture guidance or validating ideas, Elastic Support helps Guidestar by Candid's Elastic Cloud deployment stay in tune.
Company Overview: Better Data. Better Decisions. Better World.
Nonprofit employees make up the third-largest workforce in the United States. They are a force that touches millions of lives every day – one that is working hard to make the world a better place.
Guidestar by Candid is a 501(c)(3) public charity whose mission is to provide better data to help people make better decisions that will lead to a better world. They aggregate information about over 2 million nonprofits – things like their mission, legitimacy, impact, reputation, finances, programs, transparency, and governance – and make it available at no cost as part of their pledge to public service. They also offer a variety of paid products and services for users and organizations that want to further leverage their rich nonprofit data set.
Powering the Search for a More Philanthropic World
Guidestar by Candid’s goal is to make the world a better place. They are the world’s largest source of nonprofit information, aggregating data from more than 2 million nonprofits to help more than 7 million donors and grantmakers make informed decisions around which nonprofits they support.
Guidestar by Candid initially relied on FAST ESP as its search engine, but once FAST became part of SharePoint in 2014, they knew they needed to make a change. They began by switching their APIs over to Elasticsearch and saw positive results, so when they decided to start moving their infrastructure to the cloud, they chose Elastic Cloud on AWS to continue the migration of their other search offerings off of FAST.
Once they made the switch to Elastic Cloud, not only did they see faster load times, an increase in user engagement, and experience major cost savings, but their engineers are also happier and more productive.
2008: Implementing FAST ESP
With data serving as the center of Guidestar by Candid's business, they need it to be readily available and easily accessible to their users. Early on, Guidestar by Candid concluded that fast and relevant search results were the gateway to connect users, donors, grantmakers, and nonprofit managers with the information they need to make confident decisions. It then became one of their top organizational priorities.
Data is core to our business, and making that data more readily available to our users is what we're all about.
In 2008, they implemented FAST ESP as their enterprise search solution, which worked well until . . .
May 2014: SharePoint or ?
Microsoft, who acquired FAST ESP, announced they were going to stop supporting FAST as a standalone product and only offer it through SharePoint. FAST ESP also only supported XML, which Guidestar by Candid found to be too computationally-heavy for them to achieve the performance benchmarks they were looking for. And ultimately, after six years, it was time for Guidestar by Candid to refresh its software and data architecture.
Software and data architecture isn't a fixed asset that you buy and then depreciate over time. You need to keep reinvesting in it. It was time to reinvest in our architecture whether FAST was going away or not.
Guidestar by Candid began researching alternatives that were open source, had native JSON support, were horizontally scalable, had user community engagement, and RESTful API endpoints for administrative tasks.
They started looking into Lucene. After playing around with it, they realized they needed more functionality, as well as support to ensure they were able to meet customer SLAs. They next looked at technologies that were built upon Lucene, and came across Elasticsearch.
September 2014: Elasticsearch's First Test
Around the time Guidestar by Candid discovered Lucene and Elasticsearch, they were about to release the next generation of their APIs, which are used by more than 200 philanthropic websites and applications including American Express, BlackBaud, CrowdRise, JustGive, and NetworkForGood. Instead of ripping FAST out all at once, they thought this would be a good testbed to try a new search solution and see how things went.
Using Lucene required GuideStar's API users to construct complex search queries in order to produce relevant results. Guidestar by Candid believed that Elasticsearch could do a much better job without as much time being spent constructing search queries. In order to properly test this assumption, Guidestar by Candid decided to roll out a beta version of their Search API and test "one box" search functionality.
At this time, they also decided to purchase an Elastic subscription to help them architect the software the right way from the onset, as well as make sure they were able to adhere to their customer SLAs should anything go awry.
You can't be on your own when you go to production and have SLAs to meet. That, and it's a lonely feeling when you have an unsupported platform and there's no one to ask. If you're the one who knows the most about it, and you don't know the answer, and there's no one to talk to – that's an island to be on, and nobody likes that.
In January 2015, Guidestar by Candid released the beta of its Elasticsearch-powered Search API. The results were so overwhelmingly positive that GuideStar decided to pivot from their APIs and focus on the search capabilities of Guidestar.org.
Mid 2015: Guidestar by Candid's Hardware Leases and Colocation Are Up for Renewal
By the middle of 2015, Guidestar by Candid was contemplating whether or not to renew its hardware leases and sign another contract with a co-location facility. At that time, the strategic decision was made to renew both contracts for just two years and start transitioning workloads to public cloud offerings, particularly AWS.
Shane Ward, Senior Director of Technology at Guidestar by Candid, attended Elastic{ON} 2015, Elastic's first user conference, and remembered hearing about Elastic Cloud, a hosted version of Elasticsearch that automates critical processes such as installation, configuration, maintenance, backup, and high-availability of Elasticsearch clusters. He contacted his Elastic account manager to see if they could give Elastic Cloud on AWS a spin. Elastic gladly provided Shane and his team with a free full production instance to test, and within a week, they were seeing fantastic results.
The speed of our queries and index rates we tested were three times faster on Elastic Cloud than when we ran FAST on our own hardware.
These results gave Shane and his team the confidence they needed to fully migrate from FAST to Elastic Cloud. Additionally, keeping their applications hosting consistent with their data sources and cloud platform meant Elastic Cloud on AWS was a great fit.
October 2015: Migrating to Elastic Cloud
Shane and his team decided to start the migration to Elastic Cloud with two of their most popular search products: organizational search, which lets people find nonprofits by cause, and people search, which allows users to get information on nonprofit leaders and board members.
The transition itself was relatively easy due to the search abstraction layer they had in place with FAST. One of the first things they had to figure out was their cluster size – which they worked with their Elastic support engineer on: one master node, two data nodes, with two replicas for each index.
Elastic Support looked at our document size and gave us a good estimation as to what number of master nodes and data nodes to go with. That number has been right on the money since day one and has been working perfectly for us.
They also implemented X-Pack, which is part of their Elastic subscription, to secure their cluster with SSL encryption. This lets them prevent their data from being intercepted when communicating with the cluster in the cloud.
Their database administrator, Rich, who Guidestar by Candid refers to as the "grand poobah" of Elasticsearch, leveraged Elasticsearch's search templates to work on the product in isolation without impacting the live search application.
The product team at Guidestar by Candid used this opportunity as a chance to enhance the search UI. One of the big improvements they made was to preview how many results fit under certain filters, such as cause area, location, and financial information, by utilizing Elasticsearch's aggregations – which update in real-time. They didn't have this capability with FAST, and a user's search would often result in no matches as they couldn't get a feel for the results to expect without fully executing the search.
Before they went into production, user testing of the new search experience showed search results were a lot more relevant – and quick. Guidestar by Candid also did some automated load testing and found their Elastic Cloud cluster could handle 4x the load compared to when they were running FAST on-premise.
In August 2016, Guidestar by Candid went into production with Elastic Cloud on AWS and launched their new search products to the world.
The Results: Lower Costs, Higher Engagement
Here's what Guidestar by Candid has experienced so far as a result of their move to Elastic Cloud:
4x Faster Load Times
Guidestar by Candid is seeing 2.5-4x faster load times with Elastic Cloud compared to FAST ESP.
Drastically Reduced IT Spend
When Guidestar by Candid used FAST ESP, they had 19 virtual machines dedicated to search, which was 18% of their entire co-located infrastructure. With Elastic Cloud, they are able to meet the same demand with a two-node cluster, which ultimately saves them hundreds of thousands of dollars in infrastructure, licensing, support, and labor costs. They're also no longer fighting for resources with other systems across the organization, like their internal CRM or Exchange email server.
34% Increase in User Engagement
Thanks to the real-time nature of Elasticsearch's aggregations, Guidestar by Candid enhanced the interactivity of their search filters and saw a 34% increase in user engagement. User engagement is a leading indicator of churn. If your customers aren't engaging with your product, they won't renew their subscriptions.
Leveraging Assistance From Elastic Support
Whenever they have a question, Guidestar by Candid doesn't hesitate to check in with Elastic Support – engaging with them on architecture guidance, validating ideas, and consulting on various other decisions they were making – all of which is possible due to Elastic's dedicated support model. Dedicated support means everyone at Guidestar by Candid interacts with the same Elastic support engineer each time they have a question or need help. After working together over a period of time, this engineer essentially becomes an extended part of a customer's team.
We definitely appreciate the dedicated support engineer that's assigned to us – it cuts out a lot of wasted time. Even just knowing the name of the person on the other side who's helping you is fantastic. That's a major improvement over different support contracts where you've got to get the person on the other side up to speed every single time you open a ticket.
Upgrading is a Cinch
Elastic does fairly frequent product releases to get the latest features into users' hands as quickly as possible. In order to upgrade to the latest version of FAST, Guidestar by Candid had to stand up another cluster, test everything, and then schedule a night or weekend to switch everything over. With Elastic Cloud, upgrading is done with the single click of a button through the Cloud user console. This takes care of automatically spinning up a new cluster with the latest version of Elasticsearch, populating and configuring everything against it, replicating the data, and switching traffic over. No nights, no weekends. It's as simple as that.
Less Maintenance, More Time for Core Business (and Sleep!)
When using FAST, Guidestar by Candid had to engage with their clusters for management and performance monitoring on a daily basis. With Elastic Cloud, they check-in just once a month to make sure the cluster is healthy – they no longer have to spend time on hardware maintenance, or deal with alarms in the middle of the night to fix outages in their production environment.
We are not a search engine shop. We are nonprofit software. Now we can concentrate more on core business and adding features to our applications rather than worrying about our search engine acting up in the middle of the night.
Increased Agility
Guidestar by Candid is no longer scared to experiment as Elastic Cloud is less cumbersome than FAST and they know that nothing is going to fall over. When the product team wants to test a new set of features, Guidestar by Candid can stand up a brand new cluster and have it fully populated with the 16 million documents they have in 60-90 minutes. This would have taken a day and a half with FAST.
At the end of the day, Guidestar by Candid wants to work with Elastic because of shared values: we too want to make data more insightful. We too want to support our customers in the best way possible. We too embody the same passion for what we do, day in and day out. And now, we get to work together on achieving a shared goal, which feels just grand.
Learn more about Elastic Observability and our partnership with AWS. Get started today and try Elastic Observability on AWS for free.
Guidestar by Candid's Cluster
- Clusters1
- Indexes2
- Nodes3
- Query Rate8 per second
- Hosting EnvironmentElastic Cloud on AWS
- Replicas1
- Documents16 million
- Time-based IndicesReal time indexing
- Total Data Size35 GB
- Node Specifications64GB RAM, 2 data centers, 1.5TB SSD Storage
- Daily Ingest Rate250,000