Crawl, Code, Search: Elastic Site Search and Elastic App Search

By
Kellen Evan

Elastic offers effective and valuable managed search services. In this article, we will take a look at two of them: Elastic Site Search and Elastic App Search. Both services share the same goal: provide the tools to create and fine-tune relevant and valuable search experiences with minimal implementation time. While similar, the services differ in how they accomplish this goal.

By the end of this article, you will have a solid idea of which solution is right for you.

Challenge: Deep Search

Both Elastic Site Search and Elastic App Search utilize Elasticsearch, an open source, distributed, RESTful search engine. And they need it! Search is a challenging problem and Elasticsearch is a tried and tested tool that can help remedy its deepest pain points.

We can distill search challenges into three key aspects:

Ingestion. You have data. You want to make your data searchable. The act of ingestion takes an object — a webpage,  a response from a backend API — and turns it into a searchable document. This process is known as indexing: to take data and translate it into an optimal form for a search engine to process. How will you index your data? Will you host the infrastructure? What if the data are on the public internet? What if they are within a private intranet?  

You want simplified ingestion, flexibility, and minimal development overhead.

Result Delivery. Your engine is filled with documents and you can search for them. What will you receive in return? The documents, yes, but which documents, how many, and how will they look? How will it _feel_? What about autocomplete, will you build a predictive experience, one that gets ahead of your users? Will the results generate value?

You want relevant results that help you accomplish your business goals, improve ROI, and provide an enjoyable user experience.

Management. Once you have designed and developed a search experience, how will you manage it? If it took developer time to build, how much more will be needed to adjust and improve it? Can non-technical stakeholders participate in ongoing optimization? How will you manage access? Will you capture analytics and integrate search into your analytics pipeline?

You want search that works well today, is useful and insightful to all stakeholders, and can be improved and altered with minimal friction.

Two Good Things: Apples and Oranges

Both Elastic App Search and Elastic Site Search can satisfy deep search requirements across many different use cases. Whether you are an ecommerce platform, a knowledge base, a media company, a gaming application, a SaaS offering, or a mixed-use platform, either solution will lead to a high quality search experience.

Both solutions are fully managed, well documented, have access to world class support, and feature:

  • Best-in-class relevance and fine-grain relevance tuning through Synonyms, Weights, Results Rankings, and more.
  • Language optimized Engines for improved search relevance in 14 different languages.
  • Deep analytics for valuable insight into how users are searching, what they are finding, and what they are not finding.
  • API based indexing and deep search APIs with query-time parameters like faceting, filtering, sorting, and boosting.

But which to choose! Ultimately it comes down to how you would best like to solve ingestion, result delivery, and management.

Do you want to have your web pages automatically scanned and indexed, use out-of-the-box solutions or plugins, leverage a clean and functional dashboard, and apply supportive API endpoints? Elastic Site Search is for you.

Or do you want to weave the deepest and most varied search and feature API endpoints into your application code and fine-tune relevance via an accessible and modernized dashboard? Elastic App Search is the chosen one.

Elastic Site Search: Crawl for All

At the heart of Elastic Site Search is the Site Search Crawler. The Site Search Crawler works like any other web crawler. Google, DuckDuckGo, Bing — every major search engine will dispatch a crawler, a sophisticated system, to all publicly available web pages. The crawlers will scan and index these web pages, collecting metrics, capturing contents, and building documents in the process.

The Site Search Crawler is hosted and managed by Elastic. It will take care of ingestion for you. It is automated, responds intelligently to errors, and requires no ongoing configuration. Add a domain, optionally curate your Meta tags, Robots.txt file, RSS/Atom Feeds, or Sitemap, then sit back as your pages are translated into searchable, indexed documents.

For example, consider that you have a large public knowledge base filled with helpful articles. By entering your website address, the Site Search Crawler will crawl and index each webpage. Once indexed, your pages are organized into fields against a schema. Now the relevance tuning capabilities of a feature like Weights can be applied.

You can select a set of specific fields and then tune the ”weight” of each field. For example, what if we want to match user searches on the title of a document and not have the body text influence our results? What if we want to favour results which are more popular, which have received more clicks?

We would select the title, body, and popularity fields then adjust the weight on a scale of 1-10.

While automated ingestion and slick features are alluring, what about creating the search experience? How does Elastic Site Search help me bring this fancy indexing power and fine-grained relevance control into my website? The Design and Customization options presented by Elastic Site Search range from simple out-of-the-box solutions to fully customizable and dynamic search experiences.

By installing the Site Search JavaScript snippet and creating a new search field or altering your existing search field, you can use the configurable out-of-the-box Site Search overlay to display your search results.

And for custom design? Elastic Site Search contains a set of APIs to help you do just that. There are first party clients that support Java, Node.js, Python, and Ruby. We offer popular JavaScript libraries for search and autocomplete which you can use as a starting point to help you build imaginative custom experiences:

Where Elastic Site Search shines brightest is when the crawler can perform the heavier lifting. It requires little time to get started, can be tuned to your liking, and can be trusted to keep your results up to date. Elastic Site Search APIs and plugins are useful for supplementation. However, some features are only available within the dashboard and do not have a corresponding API endpoint.

For full programmatic control over deeper API endpoints, we introduce Elastic App Search.

Elastic App Search: API-centric, Human Friendly

Elastic App Search is API-centric. There are first party clients available for Ruby, JavaScript, Java, Node.js, and Python. Elastic App Search does not utilize a crawler.

The developer using the Elastic App Search APIs will determine how to generate the object data that they will index and how they will apply the various API endpoints. Ingestion, result delivery, and the overall implementation are handled programmatically. Whether one is building search into an engaging dashboard, a complex web or mobile application, a game, or a store, if you can program it, you can make your objects searchable.

All dashboard functionality is translated into robust, fine-grained APIs that you can then write into your application code. For example, both Elastic Site Search and Elastic App Search allow you to create a synonym set. Synonym sets have excellent utility; searchers will often use very different vocabulary. A car to one person is a vehicle to another, or a jalopy, or an automobile, and on, and on...

Within Elastic Site Search you can apply synonyms via the dashboard:

You can do the same within the Elastic App Search dashboard...

… Or you can request against a well-documented API endpoint: 

curl -X POST 'https://host-xxxxxx.api.swiftype.com/api/as/v1/engines/rent-a-car/synonyms' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer private-xxxxxxxxxxxxxxxxxxxx' \
-d '{
  "synonyms": ["car", "vehicle", "jalopy"]
}'

The Elastic App Search APIs go much deeper. Consider how one building an ecommerce platform may consume the Analytics API suite. Search begins with a free expression and so the analytical insights gained from it can be profound. The Analytics Suite can return information on user queries and reveal which documents received clicks over an adjustable time frame. 

curl -X POST 'https://host-2376rb.api.swiftype.com/api/as/v1/engines/sample-engine/analytics/queries' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer private-xxxxxxxxxxxxxxxxxxxxxxxx' \
-d '{
  "filters": {
    "date": {
      "from": "2018-06-15T12:00:00+00:00",
      "to": "2018-06-19T00:00:00+00:00"
    }
  }
}'

For example, what if an item sold within your store became an overnight sensation? Suddenly, searches and clicks for a document receive an explosive increase. There is significant business potential! What do you do?

You can write a function that will consume the Analytics APIs to capture the most popular documents found through search. You can then extend that function to automatically publish these documents to a prominent, eye-catching view within your homepage. User search reveals demand and automated functions optimize for it in real time. While analytics are deep and useful, search capabilities are where Elastic App Search shines.

The schema fields within a document can have one of four value types: text, number, date, and geolocation. Elastic App Search has deep search functionality across all four types. Geolocation is gaining popularity for good reason. An individual is present somewhere or heading to a new place and they want to know what is nearby.

By consuming user coordinates, you can write their geographic location into search queries and boost result relevance based on proximity.

Say we have a mobile application that lists healthy restaurants around the world... 

curl -X GET 'https://host-xxxxxx.api.swiftype.com/api/as/v1/engines/food-paradise/search' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer search-xxxxxxxxxxxxxxxxxxxxxx' \
-d '{
  "boosts": {
    "current_location": {
      "type": "proximity",
      "function": "linear",
      "center": "37.6213, -122.3790",
      "factor": 8
    }
  },
  "query": "sushi"
}'

A query for Sushi would multiply the relevance score of a result by the provided factor, given its proximity to the centre. The centre is informed by the location data one receives from the searcher. This provides one with the opportunity to deliver spatially aware and therefore highly relevant results.

A third case possible only with the depth of the Elastic App Search APIs is advanced grouping. Picture a case where you serve documentation. As you change product versions, so too will you change and version your documentation. 

curl -X GET 'https://host-2376rb.api.swiftype.com/api/as/v1/engines/sample-engine/search' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer search-o5bk7qpaedd2xmcsavb1d8os' \
-d '{
  "query": "meta tags",
  "result_fields": {
    "url": {
      "raw": {}
    },
    "title": {
      "raw": {}
    },
    "description": {
      "raw": {}
    },
    "version": {
      "raw": {}
    }
  },
  "group": {
    "field": "url"
  }
}'

If a user were to search for a particular feature, how would you reconcile the different document versions wherein the feature appears? A group query would ensure that all disparate versioned pages for the same topic return as a single search result. You can allow the user to select their desired version from that singular result, instead of having them parse dozens of results that coincide with different versions.

Elastic App Search opens up search to be much more than a visitor seeking a relevant document within a search field. Search becomes a functional action -- you have documents and your code can search through them in clever ways. You can automate actions based on what you want to find without needing the overarching context of a search field.

At first glance it may seem as though Elastic App Search is a product for the “more technical” user. It is true that the APIs are sure to please any developer looking for flexible and relevant search. But while Elastic App Search does leave the interface creation and indexing up to you, the dashboard and all the goodies within it are friendly to all levels of technical expertise.

Summary

Elastic solutions help you address your business needs. If you are looking to bring a quality search experience into your websites or your applications, look no further than Elastic Site Search or Elastic App Search — whichever works best for you.

Explore the Elastic Site Search or Elastic App Search solution pages to learn more. Both solutions provide a free 14 day trial that does not require a credit card.