BPCE-IT boosts productivity and service quality with Elastic Observability and AIOPS

The cooperative and mutual banking group, BPCE, has long been committed to the digital transformation of all its subsidiaries and services. This is a major project for the second-largest banking group in France. For BPCE, actionable data is essential, whether that’s in the quality of services offered to its customers and employees, or in the development of new uses for banking activities. As a result, the BPCE strategic plan has embraced the use of data as one of its top priorities.

BPCE-IT is an independent IT entity serving all subsidiaries of the BPCE Group. As the Head of BPCE-IT's Data for Ops, Antoine Chevalier leads the team responsible for supporting BPCE Group on the path to digital transformation by providing effective and innovative solutions for data processing and observability. To achieve its objectives of enhancing service quality, implementing AIOps, and providing an automated service offering that meets the diverse needs of BPCE subsidiaries, the BPCE-IT Data for Ops team undertook the construction of a data platform built on Elastic.

The need for automation, robot advisors, powerful analysis, dashboarding, and machine learning are all made possible using Elastic technology. With the centralized analysis of application logs from all servers, BPCE gains the capability to effectively address the complex challenges encountered by the company.

And this is how THEIA was born, an application developed and built on Elastic that meets the challenges of Data for Ops. Named after the Greek goddess who holds the gift of unlimited sight and the ability to identify situations clearly, THEIA aims to integrate and evolve the two previous uses of Elastic at BPCE-IT — managing large amounts of data and observability and making it possible to restructure the entire IT system and improve the quality of service through the use of machine learning.

The implementation of THEIA at BPCE-IT addresses a complex problem: how to improve application management in such a sizable information system? The banking applications managed by BPCE-IT are interconnected, involving many servers, making understanding and managing their overall activity challenging. As such, it was imperative to set up a large-scale log management system and introduce AIOps to manage application activity effectively.

However, with such a complex IT system, finding a solution that could meet this challenge while being easily adopted by users and ensuring the security and confidentiality of bank data was essential. Antoine explains: “When a customer uses his Caisse d'Epargne or Banque Populaire application, I call on a large number of application ecosystems on different servers, operated by different teams. Before THEIA, anomalies could be hard to spot. The power of Elastic Observability coupled with work on the quality of our data has allowed us to have a clear view of our complex operations.”

BPCE-IT and part of Natixis' IT joined forces at the start of 2021. Natixis, an investments and savings subsidiary of BPCE, had already explored Elastic's capabilities, utilizing it for dashboards and machine learning. Antoine remembers quickly recognizing the potential of AIOps: "Centralizing the data meant being able to use it for analytics and machine learning. Although initially not at scale, we felt that this approach made sense and would be decisive in the years to come."

However, BPCE-IT had adopted a more democratized approach to Elastic, leveraging its large data capacity, but in a more conservative way that primarily centered around log management.

Tasked with establishing a more ambitious and scaleable service offering for the company, Antoine and the BPCE-IT Data for Ops team sought to combine the high-performance platform and volumetric capacity of the former BPCE-IT with the more AIOps-orientated offering from Natixis, to enhance the daily management and production of applications. For this, they knew they had to choose Elastic, which had already proven itself at both entities thanks to its reliability and scalability.

Ultimately, the goal is to make AIOps accessible to all the teams in charge of applications through an automated and scalable service offering. By deploying machine learning across the information system, BPCE-IT aims to significantly enhance its ability to anticipate incidents, improve service quality, and ensure seamless operations with end-to-end transaction monitoring. The integration of machine learning within BPCE-IT's application production teams enables the implementation of supervision on previously challenging-to-monitor operations, leading to significant gains in resolution times and a noteworthy improvement in the quality of customer service.

The implementation of THEIA enabled end-to-end monitoring of the company's various applications. To ensure seamless communication between applications and the interpretability of logs, a common language was adopted by setting up a data dictionary. BPCE-IT utilized Elastic Common Schema (ECS) as a base, customizing it for their own use cases with unique terminology and keys.

Integrating a data catalog into the data dictionary serves as a central element for data exploitation. Antoine highlights: "Before, every IT department was using data independently. With the catalog, THEIA could collect and unify data all over the BPCE group so that everyone could benefit from it." This catalog empowers THEIA users with knowledge about data availability, access permissions, retention periods, and how it is organized. It enables the collection and unification of data from all BPCE-IT applications while providing essential governance around this data.

To streamline data ingestion and ensure scalability and reproducibility, BPCE-IT has developed a proprietary ETL (Extract, Transform, Load) tool called Logcraft, significantly simplifying integration with Logstash and Kafka. The ingestion pipeline is thoroughly automated, encompassing everything from message consumption in Kafka, creating and deploying Logstash configurations, to data integration in THEIA. This level of automation allows for the rapid processing of new requests and ensures strict adherence to the data dictionary and platform’s best practices.

BPCE-IT chose Elastic and its unified platform for real-time data search and analysis at scale. Elasticsearch allows BPCE to store, index, search, and analyze structured, semi-structured, and unstructured data.

The role of Elastic Observability is essential for BPCE for monitoring and analysis of all its applications. Antoine says, “100% of BPCE's applications operated by BPCE-IT are monitored with Elastic: whether they are in our data center, in the cloud, on OpenShift, or in our private cloud.”

BPCE-IT deployed with Elastic Cloud Enterprise (ECE) to meet its specific needs. This centralized approach enables BPCE-IT to build a service offering around Elastic, by quickly providing containerized Elastic instances to internal customers without restrictions, in addition to the THEIA offering.

By enabling the collection and analysis of up to 10TB of application data a day in an automated way, as well as the centralized and simplified management of instances, Elastic supports the availability and performance of all BPCE-IT applications.

Using Elastic to implement THEIA at BPCE has seen extremely positive results. The main objective was to make the data travel more fluidly to reduce log processing time and improve the overall data quality sent to the platform. The BPCE-IT teams have successfully scaled the data feedback process, enabling logs to be taken into account quickly and efficiently, going from what used to take several months to just a few minutes.

With Elastic Observability, BPCE-IT was able to implement a model that could detect anomalies more quickly. By making users more autonomous in managing their data, the IT team can spend more time on higher value-added aspects such as machine learning, automation of the platform and support.

By using Elastic to support its THEIA platform, BPCE-IT has significantly improved the quality of its data while reducing processing times, enabling better use of the data to extract added value.

For Antoine, Elastic brings considerable flexibility and comfort. Having control over the functioning of Elastic instances brings a lot in terms of operation.

BPCE-IT is confident in the product's ability to adapt and in its development prospects. Antoine and his team's vision an expansion of the THEIA platform's utilization within BPCE-IT, with a particular emphasis on the promising aspects of machine learning. To achieve this, the Data For Ops teams are strategically focusing on meeting the growing demand for their offering, continually providing users of THEIA As a Service with new possibilities.

A key component of the “as a service” approach has been the implementation of a training website. This intranet, called TEKlab (Theia Elastic Kibana laboratory), allows all users of the THEIA platform to improve their skills and learn how to get maximum value from their data.

At the heart of this approach, lies the importance of a reliable and innovative partner who will make all the difference. As Antoine aptly highlights, “For BPCE, THEIA works well because Elastic works well”.