How DTN scaled weather forecast data to petabytes per day

Weather forecasting is one of the oldest data challenges. Data scientists are constantly exploring how new modeling techniques or better data architectures could enable more acute and timely forecasts. Consumers might only be interested in knowing whether to bring an umbrella or sunscreen, but better weather forecasts can help businesses of all types improve operations and reduce the impact of severe weather events. 

DTN, the largest private weather service, has recently taken advantage of new Amazon Web Services (AWS) high-performance compute services to dramatically scale its weather forecasting pipeline size, accuracy and timeliness. These improved forecasts are already showing dividends for DTN’s operational intelligence services for agriculture, shipping, utilities and other industries. 

“Our ability to utilize the cloud and high computing scalable infrastructure is improving our ability to leverage weather forecasts by the day,” DTN’s vice president of weather operations, Renny Vandewege, told VentureBeat. 

Vandewege said that weather data currently powers a $2 billion — $3 billion market for operational intelligence services and is growing quickly

A combination of several new AWS services have helped DTN to scale the amount of data it processes from terabytes to petabytes per day; increase resolution from 10 km down to finer 1 km “pixels;” and increase the number of forecast from two per day to four per day. DTN plans to deliver hourly updates in the near future. 

Moving to the cloud

DTN started moving more of its data infrastructure to the AWS cloud to help scale its data and operational intelligence offerings. However, it still relied on its own managed supercomputers to power all of its forecasts. Over the last few years, Vandewege’s team started working with Amazon on a proof of concept to run high-resolution models using parallel clusters. 

The new DTN improvements take advantage of new AWS Hpc6a instances built for tightly coupled high-performance computing (HPC) workloads. This allows DTN to assemble a virtual supercomputer across virtual machines running on third-generation AMD EPIC processors. These also provide a 65% better price-to-performance ratio than prior offerings. 

More significantly, DTN uses AWS Parallel Cluster service to dynamically provision and manage new HPC clusters automatically. This helped DTN take advantage of new ensemble forecasting techniques, in which the results of multiple models and slightly different starting assumptions are synthesized into a more accurate forecast. 

Ensemble forecasting takes advantage of the fact that different weather forecasting models work slightly better in different conditions. Rather than trying to improve the accuracy of a single model for all conditions, data scientists run multiple models in parallel and then synthesize the results into a single forecast. These ensemble forecast results tend to be more accurate across a range of geographies, climates and weather conditions. 

But running numerous models requires finding a way to aggregate raw data from satellites, weather stations and radar, spin up new simulations to run different models and then bring all the results into one official forecast. The new AWS infrastructure also allows DTN to dynamically scale its forecasting operations in response to severe weather events like hurricanes and tornadoes to help its customer prioritize preventative and restorative operations. 

“We worked with Amazon to run these models fast enough that the data is useful,” Vandewege said.

Creating new value

DTN began as a farm information service in the 1980s that was delivered over the radio to dedicated video terminals, hence the name ‘Digital Transmission Network’ that was later abbreviated. It always had a strong focus on weather. The company went through several significant changes over the years with the growth of the Internet and competition from other weather services. It pivoted to operational intelligence services for various industries, powered by its core weather forecasting capabilities. 

For example, the company has developed an offering for maritime shipping to optimize routing and speeds using a digital twin of specific ships coupled with precise weather and ocean current data. 

“A 5% improvement in fuel efficiency can mean millions or even billions of dollars in savings for the industry over the course of a year,” Vandewege said.

DTS has also developed new predictive maintenance models to help utilities prioritize replacement or harden equipment most likely to be impacted by a heatwave, storm, or other weather-related events. 

These new services benefit from finding new ways to combine more precise weather data with other information. For instance, DTN data scientists reconstructed weather conditions that paired up with power outages or specific types of equipment failures over the last decade. 

 “Climate change is a very important factor for us,” Vandewege said. “DTN is focused on watching how climate change manifests as weather risk and helping businesses to prepare for those individual weather risks and take appropriate action.” 

New approaches to creating and modeling forecasts at scale will only grow in importance with the rising uncertainty of climate change, political upheavals and various supply chain shocks.

Originally appeared on: TheSpuzz