Betting on Elasticsearch for Enterprise-wide Search with BA Insight

Personally, I began working with Elasticsearch two years ago. The company I work for, BA Insight, a partner of Elastic, bet on Elasticsearch one year ago and incorporated it into all of our products. We’ve been wonderfully pleased with the power of what you can do with it, and in this blog, I’ll discuss how we’ve combined Elasticsearch with SharePoint for the enterprise search use case.

Combining Elasticsearch with SharePoint

Elasticsearch developers may not be too familiar with SharePoint, but it’s ubiquitous. More than 200,000 organizations use SharePoint today, often as their UI of choice for employee-facing applications, because it is easy for administrators and end users to tailor. BA Insight has been working in the SharePoint ecosystem for 10 years, and has a very tight relationship with Microsoft, so it’s home base for us. When we started working with Elasticsearch we focused first on making a great solution for enterprise search with Elasticsearch as the underlying search engine and SharePoint as the UI.

Example application: Fund Management Portal

This screenshot is an example of a typical application, designed for investment managers at financial services companies. We’re told by these users that it has become the first thing people check at the beginning of the day, and something they use constantly throughout the day. We’ve set up specific search screens for these financial analysts that encapsulates areas of interest and includes search tabs and embedded Kibana visualizations to drill down into fund performance. It includes:

• Personalized dashboard for each user with current activity (my deals, my investments, my regional news), related things that could help me (knowledge in my area of expertise, knowledge related to my deals, potential experts for my deals), and related things I might help others with (deals in the same sector as my deals, deals in my area of expertise). Each of these is just a content-by-search web part, which fires a query when the page is loaded. We’ve made it contextual through logic based on the user profile that shapes the query.
• Content from a variety of different systems that surfaces in different views (search everything, search knowledge, search people, search news). This is made possible by connectors we’ve built into the Elasticsearch index, and each view results in a different SharePoint result source (you can add a new one in three clicks). We’re up to 55 pre-built connectors for Elasticsearch.
• Investment Analysis center that uses Kibana visualizations along with BA Insight’s Visual Refiners product for exploration of funds, trends, etc.

Elasticsearch vs. SharePoint’s built-in search

We’ve traditionally worked with the search engine included with SharePoint, and still do a lot with it. However, there are several things that we do now that we couldn’t do before we started using Elasticsearch. Most notably:

• Real-time indexing has been a huge benefit to our customers. In other enterprise search engines, the latency on the content side is measured in minutes, and therefore making financial transactions searchable has been a nightmare. Elasticsearch’s real-time indexing has provided immense value in a number of verticals, but it’s especially popular in financial services.
• GeoSearch — searching and presenting things in 2D and 3D spaces — is something that our oil and gas customers are very interested in. This is not available natively with SharePoint, but it’s a very strong capability in Elasticsearch.
• Complex Queries — such as regular expression queries — and pipeline aggregations are particularly valuable for research and logistics applications. Our customers here do very complex searches for specific patents, part numbers, chemicals, proteins, genomes, etc. Many of these just weren’t possible before using Elasticsearch.

What does the future look like?

This is an exciting time. Elastic is growing incredibly fast as a company, and SharePoint is also in a huge resurgence with lots of new releases, so we are sort of holding on to a rocket ship with each hand. I’m particularly excited about:

• X-Pack - We use X-Pack with our customers as a value-added capability. For example, our customers can set alerts with Watcher, get information about when reports are ready, and get periodically scheduled updates. We’ve also been able to make our software leverage X-Pack without always requiring it.
• Graph – Elastic’s new Graph API (which is part of X-Pack) is very powerful, and it’s a good match for our text analytics capabilities, like entity extraction and relationship extraction. We already work with Microsoft’s Office Graph, but that is in many ways a sealed ‘black box’. Elasticsearch is the opposite – completely open and transparent, so we can build cool things with it quickly.
• Community – we are a relative newcomer to the Elasticsearch community. SharePoint also has a vibrant community, but the intersection is small – which I think is an opportunity to cross-pollinate. As we get more involved with the Elasticsearch community, we’re looking for ways to contribute and are open to all suggestions.
• Multi-platform – we’re starting to see more folks who are really interested in pre-built connectors, document preview, and some of our other products, but who are not using SharePoint, SQL, or Windows at all. They’re asking us to remove dependencies so that they can use our products in different environments. New territory for us, but not rocket science, so we’ve started down that path. I’d love to hear from folks about their needs and experiences around this.

We have also got some cool applications, and interesting case studies in the pipeline. And of course in the near term, we are excited about helping customers adopt and succeed with our products, and with Elasticsearch.

  Jeff Fried (@jefffried) is a longstanding search nerd and CTO of BA Insight.