Provides Technology team with detailed data insights
With Elastic, the Met Office can run searches and correlate insights that speed up the resolution of network issues.
Enables Observation team to monitor valuable hardware
The Met Office can observe and protect the performance of advanced meteorological equipment including its new Lightning Detection System with Elastic.
Shields the organization against cyber attacks
With Elastic, the Met Office Cyber Security team has a complete picture of system activity and can protect the network from suspicious behavior.
As the Met Office helps people around the world adapt to future weather and climate challenges, Elastic plays a vital role in protecting valuable information systems and helping teams get more out of their data.
There are few things in life that aren’t affected by the weather. From families planning a weekend vacation to emergency responders on the scene of a natural disaster, accurate forecasts are an important part of our daily routines.
When it comes to making these predictions, The UK’s Met Office is recognized as one of the world’s most reliable meteorological organizations. It uses more than 10 million weather observations and an advanced atmospheric model to create 3,000 tailored forecasts and briefings every day. The Met Office Hadley Centre for Climate Science and Service at its Exeter headquarters, also plays a critical role helping government policy makers assess the risk of global warming and achieve a net zero future.
To make its forecasts, the Met Office uses some of the most advanced technology on the planet. This includes one of the world’s most powerful supercomputers right down to an innovative network of Raspberry Pi devices that observe local weather conditions. It also runs a mix of on-premise, cloud, and software as a service platforms.
Unlocking the value of data
To monitor performance and security in this complex environment, the Met Office must gather logs from multiple systems in many locations. But until recently, this was a fragmented process with different teams managing their own logging systems and storing data in multiple formats.
John MacGrillen, Solutions Architect, Core Services, Met Office, says, “There were pockets where logging was managed extremely well, but there was no consistent approach, making it impossible to unlock the value in our data.”
To alleviate this, the Met Office created a unified platform for centralized storage where different teams could access the data that they needed. The project, called CLAS (Common Logging and Analytics Service) was led by MacGrillen alongside Elinor Woods, Product Manager, Platform Technology who heads up the Logging Monitoring and Alerting Team (LMA).
The Met Office operates the world’s most sophisticated and powerful computer system, the Cray XC40.
The Met Office was already using Elastic on-premise so MacGrillen and his team built a proof of concept to see if it could be used as a hub for the wider organization. They quickly established the basics of ingesting data into the Elasticsearch database, but this was only one part of the story. “The Met Office is a huge consumer of data. By adopting Elastic Cloud running on AWS and Azure we can scale the system to more than two billion data logs every day,” says MacGrillen.
“Moving to Elastic Cloud means we no longer have to worry about scaling up capacity or maintaining our observability environment. We can concentrate resources on getting the greatest possible value from information contained in our logs.”
The openness and versatility of the Elastic Common Schema was also critical. “It means that we can pull in data easily from a wide range of systems. If there’s an issue, we can join the dots to understand the impact of one event on another and fix the root cause.”
Using data to protect against extreme weather conditions
The Met Office now has a clearer view of its systems, especially during periods of high demand. These include its National Severe Weather Warning System, which issues alerts to regions affected by extreme conditions. “We can see at a glance when there’s a strain on the system and scale up capacity to ensure that it is always available,” says MacGrillen.
Meanwhile, Woods is working hard to onboard different teams across the Met Office. These include the Observation team which uses Elastic to monitor its new Lightning Detection System. “They can now monitor data flowing through the processing chain, detect faults and rectify them with the help of a support technician,” she says.
The Technology team also benefits from a more detailed view of their systems. By aggregating multiple logs, they can run searches, spot patterns and plot trends with user-friendly Kibana dashboards. “Before, they had to manually check every log. Now they can build simple queries that search for impactful events on multiple systems,” says Woods.
Using Elastic, Woods and her team were able to detect faults in the Met Office’s new Lightning Detection System.
Elastic also helps to boost system security. “During major vulnerability responses such as Log4j, our Cyber Security team was able to check if any on-premise systems had attempted to connect to a malicious IP address. Together with our network evidencing tools, Elastic provides a complete picture of system activity and suspicious behavior” says Woods.
On another occasion, Met Office employees working remotely were unable to log in using the organization’s VPN. MacGrillen says, “We were able to trace the issue back to just one broadband provider. The CLAS system, powered by Elastic helps keep employees connected and productive at a time when home working is increasingly popular.”
A bright outlook for the future
Both MacGrillen and Woods praise the support offered by the Elastic team, especially when driving adoption of the CLAS solution. “They’ve helped our users get the most out of the technology. Elastic also plays a key role in our CLAS community, including our online channel, comms website, how to guides, and video demos,” says Woods.
As the role of the Met Office adapts to future weather and climate challenges, MacGrillen is confident that Elastic will continue to play a vital role protecting valuable information systems and helping teams get even more out of their data. This includes the delivery of ever more detailed weather alerts, early warning systems, and advanced climate science.
