Why AI is an increasingly important tool in weather prediction
Artificial intelligence has been used to analyze data about weather and climate for years. Today, though, with a boost from increasingly powerful high-performance computers (HPC) and massive loads of data, scientists are beginning to apply AI to create forecasts that are more accurate, more granular, and further reaching.
That means technology is coming together to provide better climate predictions for the next 100 years as well as more pinpointed weather forecasts offering more warning for people to take shelter from events such as tornadoes and hurricanes. And this powerful combination of predictive technology, already being tested, could be working in the next three to five years.
"I think we're about to see real breakthroughs," says Sue Ellen Haupt, senior scientist and deputy director of the Research Applications Laboratory at the U.S. National Center for Atmospheric Research (NCAR) in Boulder, Colorado. "Using AI for forecasting isn't new, but the push to use it more and to use it differently is new. We're beginning to use AI to determine what storms will have extreme events, like hail or tornadoes. We might be able to get more than a few minutes' warning. We hope to maybe even get hours. AI is going to be the key to better forecasting."
And being able to improve an extreme weather prediction by even minutes could save lives and millions of dollars in property, according to David John Gagne, a machine learning scientist at NCAR who works with scientists at different weather and climate labs to help them develop AI and machine learning systems.
"Think about what it could mean for even an extra two minutes of warning for a tornado," he says. "In some cases, that might allow you to get to shelter. If we can make more specific forecasts, we also might be able to say a tornado likely will take this path but it also could go over there, so you might want to take shelter there, too."
AI and weather have history
Weather and climate forecasting scientists at organizations such as NCAR, the National Oceanic and Atmospheric Administration (NOAA), and the U.S. National Weather Service are no strangers to technology, harnessing massive compute power and AI to improve forecasts and provide more accurate, far-reaching weather, climate, ocean, and space weather information.
The accuracy of daily weather forecasts, as well as warnings of severe weather, depends on smart algorithms and supercomputers. Weather forecasting, which involves managing, analyzing, and visualizing vast amounts of data, has depended on AI for years. Twenty years ago, when the Dynamic Integrated foreCast (DICast) system was created to take in meteorological data and produce automated and accurate forecasts, AI was part of it. And the system has been used by top commercial weather service companies.
"AI has been used in weather forecasting for a long time, but now there's a resurgence because of advances in machine learning, driven by the availability of massive amounts of data and the power of GPUs," says Ilene Carpenter, Earth Sciences segment manager at Hewlett Packard Enterprise. "Weather forecasting centers were among the first supercomputer users. Now, they are combining physical modeling on supercomputers with AI and data-driven approaches to enable better predictions."
Each technology is dependent on the others because alone, they simply couldn't get the job done. And what is happening in weather and climate forecasting is a taste of what will be happening in other industries around the globe.
"The advance of these technologies is creating a natural combination for more than just weather and climate but for everything going forward," says Jeff Kagan, an independent industry analyst. "AI, of course, has been around for a while, but it's just grown dramatically, and HPC has grown to a new level. If you can harness the power of those technologies together, then it will create a new way of doing things, a new paradigm. These technologies going forward will be used to change every industry. We're just taking the first steps into this new world."
Using AI to tackle hail predictions
Gagne and other scientists at NCAR have been using a neural network as part of a weather model that analyzes storm factors like temperatures at different altitudes, wind direction and speed, updraft, and humidity. The model then recognizes patterns that signal a storm could produce hail, which can fall at a speed of up to 120 miles per hour and annually causes about $8 billion to $10 billion in damages in the U.S. alone.
Gagne, who is leader of the hail project, says they use machine learning to estimate the possibility of hail in a specific area up to a day in advance. Working with a team from the University of Oklahoma, he wrote the initial code to train the machine learning models, which have been able to find different pieces of information in massive datasets that, when pieced together, indicate a hailstorm is brewing. The system can even predict if the storm will produce small or large hail.
Running in real time at the University of Oklahoma, the system is being tested using data from NOAA. "It's been successfully predicting hailstorms," Gagne says. "It's doing very, very well."
To make this kind of predictive system work, scientists need to gather, organize, and analyze large amounts of weather data. And that takes advanced AI and HPC.
Technologies come together
Around the globe, weather sensors on satellites, on the ground, and in the oceans are providing a fire hose of weather and climate data. It's far too vast and overwhelming to be analyzed and scanned for patterns by humans or even traditional computer systems. And that's a problem, because without the ability to make sense of the deluge of information, it's a wasted opportunity.
"We have so much data that forecasters are overwhelmed with the amount of information to sort through to make a decision," says Gagne. "We're already well past the point where there's so much data and so much information noise that humans struggle to recognize the patterns in them. We need to develop really good visualization analysis tools, ways to dig in."
Scientists are doing that with AI systems, machine learning, neural networks, and deep learning, because their pattern-recognition capabilities are tailor-made for the job. The systems can be fed massive amounts of data and learn how to spot a storm that might produce lightening or tornadoes. It can see patterns that are likely to lead to devastating hurricanes or brutal snowstorms. This kind of pattern recognition works with both weather and climate datasets.
But how do you handle that massive of a load? That's where HPC comes into play.
"Typical discussions around climate change revolve around highly simplified, coarsened quantities, such as global, annual surface temperature increase, or sea-level rise," says Prabhat, group leader of the data and analytics services team at the National Energy Research Scientific Computing Center (NERSC). "But increasingly, people want to know how climate change will affect where they live. We need simulations that will more precisely resolve at a finer level. That will produce massive datasets, and you need to conduct precision analytics to ask, say, how Maine will be impacted or how will my city be impacted. AI is poised to help us with more accurate simulations and precision analytics."
For the past three years, NERSC's flagship supercomputer has been a Cray XC40 machine, dubbed Cori. With 622,336 Intel processor cores, it theoretically can perform 30 quadrillion operations per second. (Hewlett Packard Enterprise acquired Cray in 2019.)
NERSC, which is the primary scientific computing facility for the Office of Science in the U.S. Department of Energy, is getting ready for an upgrade, though. Later this year, the organization is expected to begin receiving pieces of the next-generation HPE Cray supercomputing architecture, Shasta, which will feature AMD EPYC processors and NVIDIA GPUs.
And that kind of compute power is just what NERSC is going to need to take on the kind of upgraded weather and climate simulations it's looking to do.
One of the projects that scientists at NERSC are focused on is small-scale predictions—small in geographical size but massive in terms of the need for compute power.
For this kind of forecasting, think about microclimates. For example, instead of predicting that San Francisco will be 70 degrees and cloudy, a microclimate prediction might specify which neighborhoods will have cloud cover and higher humidity and which ones will be sunnier and drier.
"Those kinds of predictions could potentially be done coupling AI (running on GPUs) with classical partial differential equation (PDE) solvers (running on CPUs) on a modern supercomputer," says Prabhat. "Ideally, we want to be able to run a global model at this scale. We want much finer predictions over the entire globe. I think with the machines we have now, we couldn't get there."
He adds that, in a time of dramatic and life-threatening climate change, improving our climate models will similarly call for advanced AI and supercomputers.
At NOAA's Earth System Research Laboratories, Jebb Stewart, informatics and visualization branch chief and leader of cloud computing and machine learning efforts in the lab, says, "We couldn't handle some of the machine learning apps without high-performance computer processing. For some of the neural networks we develop, for both the depth and the data size, we need highly scalable and parallel processing to make it work. Running that on a single node could take hours, if not days, to process several terabytes of that kind of data. But with HPC, we can get it down to minutes or hours. We can use different neural network structures because of this."
Testing, testing, testing
While different weather and climate prediction agencies are focused on adopting machine learning and deep learning, don't expect to see major changes to your daily forecasts immediately. When it comes to weather in particular, scientists must do a lot of testing to make sure the addition of any new technology is not going to cause any problems to such an important service.
"Researchers are taking a close look at how best to integrate AI techniques into [weather and climate] simulation codes," says Prabhat, noting that he thinks we'll start to see advanced AI at work in weather and climate prediction in another five years. "With any field of scientific inquiry, you don't just adopt something because it's hot. We've largely relied on simulations based on applied mathematics, not AI, for 40-plus years. We're not going to replace PDE solvers with AI overnight until AI has been investigated very carefully."
Over at NCAR, Gagne says it could take decades for AI to reach its full potential in the field.
"The weather community moves kind of slowly, but there's a ramp-up in investment, and I would expect to see even more in the next five years," he adds. "In some cases, the predictive improvement could maybe be another 10 percent in accuracy. It may not sound like a lot, but even a small improvement makes a big difference. We want people to have time to take shelter from a tornado or put their car in the garage before a hailstorm hits."
Stewart agrees that even incremental improvements in weather forecasting could be critical for individuals, farmers, and companies. And that is exactly what he hopes is coming.
"There are AI applications being tested but … there's a ton more in development," he says. "In the next couple of years, due to AI, we'll see a significant improvement in weather forecasting. First, we have to verify that we're not making the weather models worse. We have to verify we're making it better. There's a lot coming. That's very exciting."
AI in weather prediction: Lessons for leaders
- AI paired with machine learning, neural networks, deep learning, and high-performance computing is expected to significantly improve weather forecasting over the next three to five years.
- Climate scientists and weather forecasters are increasingly looking to these tools to sift through massive amounts of data and detect patterns indicating severe weather or climate changes.
- This combination of powerful predictive technologies will not only greatly enhance weather forecasting but provide similar capabilities in other industries as well.
"I think we're about to see real breakthroughs. We're beginning to use AI to determine what storms will have extreme events, like hail or tornadoes."
This article/content was written by the individual writer identified and does not necessarily reflect the view of Hewlett Packard Enterprise Company.