How to

Elastic Advent Calendar, 2020: the full recap!

By
David Pilato

Wow, it's finally here! After 25 fantastic articles we've reached the end of the 2020 Elastic Advent series.

We've covered ECS, Synthetics, Monitoring Tekton tasks and pipelines, OpenTelemetry, log correlation with APM, Index patterns and tradeoffs for observability, Rank features, New data types, Phonetic search, Korean language search, Elasticsearch Rust Client, Searchable snapshots, ESM, Workplace Search, App Search, Data import/export, Data visualisation, Kibana Lens, Kibana Maps, Free trainings, Preparing your certification, Preparing interviews, and so much more. In the topics we've spoken in Chinese, English, French, German, Korean, Russian and Spanish.

We hope you have enjoyed the posts, topics and information shared in them from our engineers. Here's a recap of the final few days, as well as a summary from all 25 days.

The 25 days of Elastic

Dec 1 [english] — Synthetics: proactive problem detection (Elasticsearch and Python - tools for a data scientist), by Ahil PonArul

Elastic is excited to introduce synthetic monitoring to our Uptime solution. This allows for more advanced uptime checks beyond basic pings. Combined with our new User Experience UI it also enables proactive problem detection.

This is a step by step example of how to set Synthetics up, as well as integrate it with APM.

Dec 2 [german] — Aufgemerkt & zugehört! Bessere Suchresultate mit phonetischer Suche, by Alexander Reelsen 

Phonetische Suche ist eine Möglichkeit, nach ähnlichen klingenden Termen zu suchen, indem ein phonetischer Hash im invertierten Index gespeichert wird. In diesem Artikel schauen wir uns einige unterschiedliche phonetische Algorithmen in Elasticsearch an und wie diese die eigene Suche verbessern können.

Dec 3 [english] — Cross version Elasticsearch data migration with ESM, by Medcl Zeng

I heard that you are using Elasticsearch, that's great as, you know for search, it is the best choice, and it is evolving very fast. There are so many new and nice features coming up or already that i guess you can't wait to upgrade to the latest version, right?

This blog post will explain how ESM could help to do your data migration.

Dec 4 [english] — Validate Elastic Common Schema (ECS) fields using Security Detection Rules, by Eric Beahan 

The Elastic Common Schema (ECS) provides an open, consistent model for structuring your data in the Elastic Stack. By normalizing data to a single common model, you can uniformly examine your data using interactive search, visualizations, and automated analysis.

Elastic provides hundreds of integrations that are ECS-compliant out-of-the-box, but ECS also allows you to normalize custom data sources. Normalizing a custom source can be an iterative and sometimes time-intensive process. However, we can use the Elastic Security Detection Engine to help quickly identify ECS non-compliance in our events.

Dec 5 [français/english] — Recherchez tous vos documents, n'importe où, avec Workplace Search | Searching anything, anywhere with Workplace Search, by David Pilato

You already know that Workplace Search comes with a lot of connectors which help you connect your enterprise document data sources and have a federated way to search across all that information. But what if a specific data source is not supported yet?

This post will cover how you can create a custom data source to send your own data. We'll also cover an example of how this was used in the community FSCrawler project.

Dec 6 [spanish/english] — Cargando datos de OSM en Elasticsearch | Uploading data from OSM into Elasticsearch, by Jorge Sanz 

One of the most common issues for Elastic stack users that deal with geospatial data is how to upload geospatial data users is how to ingest data in Elasticsearch. You can check Kibana 7.10 docs to learn about different ways to achieve this. Some time ago we wrote a blog post that introduces ogr2ogr, a tool from the GDAL library that helps on ingesting data from dozens of formats into Elasticsearch.

In this Advent Calendar post, we develop an example of this workflow using Docker to leverage the last version of the GDAL tool and OpenStreetMap as a popular source of Open Data Points of Interests.

Dec 7 [français/english] — Utilisez Rally comme outil d'import/export de données | Using Rally as a data import/export tool, by Laurent Huet 

This post will show you how you can use Rally to export data from one cluster to another. The idea is to extract the whole data from one or more indices in a (big) flat file and reuse it later to easily import in another cluster.

Rally helps you do that very easily.

Dec 8 [russian/english] — Rank features для поиска в e-commerce | Rank features for e-commerce search, by Mayya Sharipova

Modern e-commerce search is expected to be fast, relevant and provide an opportunity for promoting certain results. This article demonstrates how rank_feature and rank_features types of elasticsearch can help in this goal. We will use an example of a commercial search engine for a shoes shop.

Dec 9 [english] — Don't let your Christmas tree Rust in a corner, by Sylvain Wallez

Christmas trees are part of what makes this time of the year so unique and brings sparkles in the eyes of children and adults alike. But what do you do with the tree once the party’s over? The best is to make sure it’s recycled and used as compost or firewood. Now how do you find where to dispose of your tree so that it will be correctly taken care of?

I live in Toulouse in the south of France, and the local authorities have not only set up a lot of collection places, but also published their location as open data. It’s even listed on European open data. There’s a treasure trove of information there!

Let’s use this information to build a nice map in Kibana, and use the Rust client for Elasticsearch for that? Why Rust? Because it’s a great language that is growing in popularity, and this is an opportunity to experiment!

Dec 10 [spanish/english] — Las 12 mejores características de Mapas desde GA | Top 12 new features in Maps since GA, by Nathan Reese

Maps was released as generally available (GA) in Kibana 7.3. Reaching production-ready is not the final destination, but just another stop on a long journey. Each Kibana point release includes so many incredible features that it’s hard to keep up with them all. So, as 2020 comes to an end, let's look back and revisit some of the most impactful features since going to GA.

Dec 11 [english] — This holiday season, learn new Elastic skills, by Pablo Musa

Elastic has a lot of different products across multiple solutions. We make it as easy as possible for our users to benefit from our solutions, but it is still hard to master so many topics. To address that, Elastic invests a lot in creating the best possible content. And even though we create them in many formats, such as docs, blogs, webinars, ElasticON presentations, and training, it can still be daunting to learn so many different concepts. In this blog we will help you make the most of your holiday season by learning new skills.

Dec 12 [spanish/english] — Tu infraestructura en un mapa | Mapping your infrastructure, by Thomas Neirynck

This post shows how you can use the Elastic Stack to map IoT, Security or Observability data. The data that is collected in these use-cases often do not explicitly contain a latitude or longitude. It is still possible to map this data in Kibana by using a “Term Join”.

Dec 13 [english] — Making it personal: Tailoring content with signed search keys in App Search, by Orhan Toy

Signed search keys in Elastic App Search give you more control of a user's search experience. You can tailor the experience to show results you know are more relevant to the specific user while also letting you control what data the user can see and search over.

Dec 14 [chinese] — 如何成为一名 Elastic 认证工程师, by Xiaoguo Liu

对 Elastic 认证的类别，培训及认证流程做了详细的介绍。针对中国地区的工程师如何应对 Elastic 认证做了详细的描述。

Dec 15 [english] — Preparing for an Elasticsearch Interview, by Aravind Putrevu

Elasticsearch is the most popularly used data store for building a Search Engine, Centralized Logging, Observability, or Threat Hunting use cases.

That also means Elasticsearch is omnipresent in many organizations.

In this post, we'll see what are some important topics that you need to prepare for an Elasticsearch interview.

Dec 16 [french/english] — Monitorer les tâches et pipelines Tekton avec Elastic Observability | Monitoring Tekton Tasks and Pipelines with Elastic Observability, by Maxime Gréau

Do you know that Elastic performed 21 releases in 2020?

Each time a release is promoted, this is 500+ artifacts published to multiple public places (bucket, Docker registries, Maven Central, Rubygems, and so on) and available on Cloud at the same time. This complex process became a non-event thanks to our Unified Release workflow based on Tekton Tasks and Pipelines and monitored with Elastic Observability.

This blog post shows how to run your first Tekton Task, and then how to install and use the Elastic Observability Solution to monitor many Tasks and Pipelines deployed within a cluster.

Dec 17 [korean] — 한글 형태소 분석기 파헤치기, by Jongmin Kim

Elasticsearch 에서는 Elastic 에서 공식으로 제공하는 한글 형태소 분석기인 nori 를 사용할 수 있습니다. 한글은 띄어쓰기가 없는 복합어가 대다수이기 때문에 의도하지 않은 대로 분석이 되는 경우가 많아 nori 를 사용하기 위해서는 목적에 맞는 사용자 사전을 등록해야 할 때가 많습니다.

Dec 18 [english] — Set up searchable snapshots in ECK, by Idan Moyal

Searchable snapshots, recently released as BETA in Elasticsearch 7.10, let you reduce your operating costs by using snapshots for resiliency rather than maintaining replica shards within a cluster.

In this blog we’ll demonstrate how to create a hot-cold topology using Elastic Cloud on Kubernetes (ECK). For the cold tier we will mount a snapshot using the new searchable snapshots API. The demonstration is carried out on Google Kubernetes Engine (GKE) and can easily be adjusted to other Kubernetes environments.

Dec 19 [english] — OpenTelemetry in Go Applications using Elastic APM, by Ricardo Ferreira

Distributed tracing technologies allow developers to virtually glue together disparate services to build a cohesive transaction that can be observed by folks in the operations team. This is super important because the distributed nature of modern cloud-native applications makes it hard for teams responsible for maintaining these applications up-and-running to rapidly perform RCA (Root Cause Analysis) of issues when they happen.

Though tracing technologies are not necessarily new only in recent years it gained enough traction to become one of the three main pillars of an observability strategy—notably logs, metrics, and distributed tracing.

To speed up developer adoption, multiple standards such as OpenTracing and OpenCensus have been created throughout the years. However, it didn't make any sense to have multiple standards since this creates more harm than good. For this reason the standard OpenTelemetry was created out of the existing ones to be an observability framework for cloud-native software.

In this post, I will walk you through how to instrument applications written in Go to emit traces compatible with the OpenTelemetry specification, as well as how to send these traces to Elastic APM.

Dec 20 [russian/english] — Разведочный анализ данных с Kibana | Exploratory data visualization with Kibana, by Raya Fratkina

Practitioners in the field of data visualizations often talk about 2 types of visualizations: exploratory vs explanatory. To quote Google definitions (the most authoritative source, of course), "Exploratory data visualizations (EDVs) are the type of visualizations you assemble when you do not have a clue about what information lies within your data.”

Elastic stack is a great tool for such exploration since because of the flexible ways you can combine search, filtering, and aggregations to understand your data. In addition, you don't need to go through a costly process of defining a schema before you can start exploring.

Dec 21 [english] — When neither logging nor code tracing is enough: Log Correlation with APM, by Emanuil Tolev

Application Performance Monitoring and logging both provide critical insight into your ecosystem. When paired together for context, they can provide vital clues on how to resolve problems with your applications. This post assumes you’re familiar with what an APM (also known as “code tracing”) system does, what log monitoring is, and the benefits of both. Elastic offers free solutions for both as part of Elastic Observability.

Dec 22 [german] — Index-Patterns und ihre Tradeoffs für Logs, Metriken und Traces, by Philipp Krenn

Index patterns in the Elastic Stack for time-series data have been evolving. From single indices to daily ones, over rollovers, to the latest development, data streams. This post looks into each approach’s tradeoffs and how to use them with Beats, Elasticsearch, and Kibana.

Dec 23 [spanish/english] — Nuevas incorporaciones a la familia de tipos keyword: constant_keyword y wildcard | New additions to the keyword family: constant_keyword and wildcard, by Imma Valls

We’ve recently introduced two additional keyword types, wildcard and constant_keyword. In this post, we’ll try to briefly introduce them.

Dec 24 [english] — New Kind on the Block, Kibana Lens!, by Rashmi Kulkarni

Millions of people already use Kibana for a wide range of purposes, but it was still a challenge for the average business user to quickly learn. Visualizations often require quite a bit of experimentation and several iterations to get the results “just right”. Visualizations in Kibana paired with the speed of Elasticsearch is up to the challenge, but it still requires advance planning or you’ll end up having to redo it a few times.

The new kid on the block, Kibana Lens, was designed to change this and we’re here to learn how to take advantage of this capability. So let’s get started! We are excited to announce that with the 7.10 release, Kibana Lens has officially become generally available. Lens is the easiest and most intuitive way to visualize data in Elasticsearch with a simple drag-and-drop interface that lets anyone instantly begin exploring data for insights, regardless of their previous Kibana experience.

Dec 25 [english] — Santa’s Little Helper - Workplace Search, by Sean Story

You might have picked up on this from movies: Santa’s operation is pretty high tech.

It’s a big job, coordinating a massive workforce of Elves to ensure the timely delivery of billions of presents each year. This year, Santa is particularly excited to make use of Elastic’s Workplace Search for its first Christmas.

You see, Santa and his team are working with a lot of data:

    • Children all over the world are sending him letters and emails to let him know what their wish lists are.
    • He’s got R&D designing and perfecting thousands of new toys each year.
    • The assembly line teams require precise numbers to ensure that the right quantity of each toy gets produced.
    • His fulfillment team has a mad scramble at the end to ensure that every single child gets the specific toy they wanted.
    • The support desk has to field enormous request volumes on these new toys,
      • From parents before the holiday (“Does this toy come with batteries included?”)
      • From children afterwards (“Are there any cheat codes?”).

In years previous, this meant Santa had to hold a lot in his head - remembering how to log in to each of these separate data stores, how to search in them, and how to find the right subject-matter-experts for any given problem. But not this year!

Thank you!

All the of the topics will be kept on the Elastic Discuss Forums so you can refer back to them at any time. And, as these are Discuss topics, you can also continue the conversation with the authors and other community members.

Thanks for following this year's series, we hope it’s provided some useful inspiration for your use of the Elastic Stack. If you’d like us to repeat this, if you have ideas for next year or any other feedback, please let us know via Twitter (@elastic) or feel free to create a topic in our Meta category with your comments.

We look forward to building upon it for 2021!