Rabobank: Enhancing the Online Banking Experience with Elasticsearch
With nearly 900 branches spanning 50 countries, Rabobank is a leading global financial provider. And like many other big banks, they’ve built a powerful financial transaction system on top of mainframes. But storing that data is only the beginning. Rabobank needs to be able to find and serve up years worth of financial data in real time to all their customers, all while maintaining PSD2 compliance. While great for transactional operations, mainframes aren’t ideal for handling complicated search requirements. And with over 10 millions customers — some with thousands of accounts — solving this problem with a mainframe can be very costly.
It was around this need for a powerful search solution that Team Fortress was created. Forming in late 2016, Team Fortress is a four person team, led by René Bouw. Their objective was to improve the delivery of payment and savings transactions for Rabobank’s customers. They did so by dramatically increasing the amount of transaction history data that customers could view and save. Before the project began, Rabobank customers were only able to retrieve transaction data from one account at a time, and only as far back as 16 months. For anything older than 16 months, customers had to go to their local branch to request an access to the data, which incurred additional charges. For the project to be considered a success, Rabobank needed customers to be able to search through all their accounts at once, as far back as 8 years. That’s a big ask for a mainframe, but not for Elasticsearch.
After running a POC using the open source Elastic Stack, Rabobank decided to purchase a commercial subscription. Indeed, while the powerful search functionality that Rabobank needed was available in open source, the security features available with a commercial subscription — like encryption (TLS) authentication and logging audit data — were essential to be compliant with the bank industry security requirements.
They developed an architecture that allowed them to pull transactional information from their mainframes into both a data store and Elasticsearch, with the Elasticsearch data being limited to 8 years of retention. This also allowed Rabobank to maintain the transactional speed of their mainframe system while also making dramatic search improvements for the users. Speed changes were noticed immediately.
“The speed is impressive according to all users we ask, including myself. Some technical people even double check the results, because 'that can't be true, that fast and accurate'. But it is.” — René Bouw, Product owner and solution architect of Team Fortress, Rabobank.
Not only is Rabobank searching faster than ever, they’re searching through more data than ever. With over 23 billion transactions spanning 80TB of data, Rabobank sees upwards of 200 events per second — over 10 million per day. And each query can span thousands of accounts, with corporate customers having over 5,000 accounts that they can now query at once. And being able to do all of this without adding any extra operations to their costly mainframes has helped save them millions of euros per year. Today, all front-end applications use Elasticsearch for search or for aggregating payment and saving transactions.
Knowing that the Elastic Stack is also great for logging and metrics, Team Fortress also developed a monitoring solution using Elasticsearch, Logstash, and Kibana to monitor all of the components of their data store solution. Using the Elastic Stack, they can monitor the CPU, memory usage, logs, errors messages, and more from all of the applications in their solution as well as the virtual machines running the applications. They use customised dashboards for the machines and drill-down dashboards for the applications. They also use the Elastic Stack alerting functionality to let them know about problems the moment they pop up. Said Bouw, “We have never missed an incident yet. We chose Elastic because of its resilience.”