Why Elasticsearch is an indispensable component of the Adyen stack

At Adyen, we use Elasticsearch to power various parts of our payments platform. This includes payment search, monitoring, and log search. Let’s take a look at how we use Elastic for these different use cases and see how we capitalize on the power of Elasticsearch. 

We recently did a talk about some of our Elasticsearch adventures at an Elastic meetup. You can find a recording here.

Before we look into our different use cases, it helps to get a high-level understanding of Adyen’s architecture.

At Adyen, we do many things, but the core of what we do is payment processing. Our payment platform is built for any business and every customer journey. We process payments across all sales channels. 

Let’s say you want to buy a beer at the local pub (a merchant) and you want to pay with your credit card, which you received from a bank, called the issuing bank. Now when you pay for the beer, the pub wants to know if your account is valid and if you can make the payment. For this, the pub needs to contact the issuing bank.

This is where Adyen comes in.

When you provide your payment details — a credit card in this example — to the pub, the pub contacts Adyen with the payment  details, and Adyen figures out the rest. This usually means sending the payment details to a third-party, or scheme, such as Mastercard or Visa, which forwards it to the issuer.

The Adyen flow looks like this:

But why doesn’t the merchant contact the scheme directly? 

Because the next shopper might not use a credit card but wants to pay by scanning a WeChat Pay QR code. The merchant does not want to build a separate integration for every way of paying, because it should also work for payment methods like credit/debit cards, SEPA, iDeal, Klarna, PayPal, and so on. This is especially true for international merchants, and this is where part of the power of Adyen lies.

Now imagine we do such a transaction not just once but hundreds of times per second. These are the principles that we follow that allow us to keep doing this efficiently and still be flexible:

• One platform worldwide
• A single application that receives payments
• Abstract away differences between payments to make it easier to handle the data later

This is, of course, a very simplified view and explanation. In reality, there are hundreds of components and moving parts that work seamlessly together to make everything work. 

Now that we have a basic understanding of the scale of the architecture, let’s look at how we use Elastic.

Payment search

Customer Area is our merchant-facing web application where merchants can gain insights into their transactions and perform operational actions. In the Customer Area, there is a feature called payment search. It is the most used functionality in the Customer Area, and it allows merchants to search their payments by using an order reference, email address, payment method, and so on.

We use Elasticsearch as a search engine in our application because it is built to provide simple REST APIs and has a distributed nature, speed, and scalability. 

Adyen processes millions of payments every day. We want to provide a near real-time payment search functionality for merchants. 

So, imagine that a shopper is purchasing a pair of shoes in a store. The credit-card payment for the shoes needs to be immediately visible to the merchant so they know the buyer’s purchase was successfully processed —  usually within seconds.

In order to make this possible, we have to index all this data into the Elasticsearch cluster when the payment is successfully processed. For these purposes, we have an in-house streaming/consumer framework that helps us to make this happen in a scalable way. It is implemented to take the burden from accounting clusters and create a summary of the data in the streaming clusters. This data is fed to the consumer framework, which indexes the data into the payment Elasticsearch cluster.

The consumer framework implements three parts of the indexing journey: 

• The first part is filtering, which decides if the current stream item is of any interest to this consumer. This is used to filter out the payments which have a creation date before a certain date. 
• The filtered payment goes to the consuming part, where we apply the business logic. In our use case, it converts a payment to a payment document and creates index/update requests depending on the status of the payment. 
• After the consumer framework reaches a certain amount of consumed payment or a certain time, it goes to a persisting level where it performs the bulk operation with the bulk request. The bulk request consists of index/update requests that we consumed so far. After it gets a successful response from the Elasticsearch cluster, it saves the in-memory state of the consumer to db. In case some requests fail, the consumer framework includes them into the next persist and then saves the state. This helps us make sure we index all the data. 

This cycle in the consumer framework runs continuously, as long as we have new payments processed. It allows us to give our customers a near real-time, reliable, and fast payment search experience.

Platform monitoring

The goal of the monitoring team at Adyen is to provide solutions that can monitor the whole platform for merchant-facing issues. This is already quite a challenging task, and on top of that, the solution must be flexible enough to be used by other teams, each with a very unique use case. 

Plus, the alerts must occur on a (near) real-time basis. To summarize, monitoring aims to provide:

• Real-time alerts
• End-to-end monitoring
• Integrated, scalable solutions to all teams at Adyen

To accomplish this, we leverage the Elastic Stack as the underlying basis to build the necessary tools. The monitoring solution looks upon custom-based analytical events. The analytic events are freely created by the programmers and are the main data source for all the configurable monitors. 

These events exist in parallel and complement the logging infrastructure — and the data in databases. Next, the monitors accommodate the analytic events into multiple use cases, such as spikes, drops, underperforming transactions, anomaly detection, forecasting, and more. These monitors are responsible for creating alerts that are, finally, delivered to the right individuals (or teams).

From a business perspective, the objectives were:

• Customizable monitors for predefined use cases.
  • Provide monitors for standard use cases (such as a drop in volume, or a spike in errors) that are easy to set up for anyone in the organization
• Out-of-the-box measurements.
  • Every new merchant, or host, is already capable of generating events from Day 1
• End-to-end alert lifecycle integration.
  • Use a single interface to create any monitor, tweak its settings, and receive the alerts

From a development perspective, the objectives of monitoring were:

• Real-time user-based analytical events
• Parallel and complementary with the logging structure
• Seamless ES template mapping to/from java objects
• Enabled/disable event materialization routes on top-level events.
  • Long-term storage

Keeping the lights on

Apart from payment search and monitoring, we also use Elasticsearch for storing and retrieving logs, and we use Kibana for log search, which is integral for troubleshooting issues internally. 

Our logging cluster was at 1 petabyte at one point. For payment search, we did a migration from Elasticsearch v1.7 to v7.6 with 12TB of data last year, and the flexibility of Elasticsearch made the migration smooth. So at Adyen, Elasticsearch will continue to be an integral part of keeping our lights on.