Meteorologists predict better weather forecasting with AI

The value of having data now is higher than it’s ever been,” says Greg Herman, a research scientist at the Climate Corporation in San Francisco, which uses weather information to help farmers manage risk. Weather forecasting is joining a growing number of fields, including financial management, internet advertising, oil and gas exploration, and scientific research, that are adopting artificial intelligence (AI) methods to analyze huge data quantities. (For other examples, see Physics Today, February 2019, page 17 , and the article by Sankar Das Sarma, Dong-Ling Deng, and Lu-Ming Duan, March 2019, page 48 .) By processing more data in a shorter time, weather forecasters could use AI methods to improve the prediction accuracy of tornadoes and other weather patterns over traditional methods. Meteorologists hope those advancements may help save lives and money.

Despite the promise of AI’s predictive capabilities, scientists face challenges in developing AI for operational forecasting. Preparing data for an AI algorithm, though faster than traditional methods, takes up to 90% of AI researchers’ time, according to Herman. Climate change alters the multidecadal background state of the atmosphere, which makes it difficult for meteorologists to produce accurate predictions even with AI methods. And decision makers at the National Weather Service (NWS), for example, need to choose whether to allocate the limited financial and computational resources to AI-guided forecasts or other existing capabilities.

Artificially intelligent forecasts

Meteorologists currently forecast weather by collecting and processing observations such as temperature and pressure, assimilating those variables into numerical computer models, extrapolating the models into the future, and using the projections as the starting conditions for the next iteration. More data on finer scales improve forecasting, but traditional dynamical modeling and existing computing capabilities can’t keep up. AI methods promise improvement by more efficiently mining data for useful features as both inputs and corrections to the outputs of weather forecasting models.

One class of AI method, machine learning, builds on traditional modeling techniques by learning to recognize weather features using training data—satellite imagery and ground observations, for example—that characterize multidimensional and nonlinear processes. NOAA’s weather-satellite observing system now produces kilometer-resolution imagery every few minutes. With AI methods, programmers can write algorithms to scale as more and better data become available. Thus they can avoid some of the shortcomings of more-established statistical methods, says David John Gagne of the National Center for Atmospheric Research in Boulder, Colorado.

Preliminary research suggests that machine learning results may outperform traditional approximations of some meteorological variables in a forecast. Those variables include soil moisture and temperature, which can fuel thunderstorms under the right atmospheric conditions, says Jebb Stewart, a scientist at NOAA’s Earth System Research Laboratory in Boulder. With machine learning, more data are incorporated into the model without increasing the time required to complete the approximation. The traditional approach for estimating soil moisture saves scientists time and money by using only a fraction of the available data: the previously modeled value and the present estimate of surface temperature. By starting with a more accurate initial state of soil moisture, machine learning helps weather models run faster and make more accurate predictions.

Although computational power and memory continue to increase, weather models still must approximate solutions to the Navier–Stokes equations, which describe the fluid dynamics governing the atmosphere’s behavior. Machine learning can identify biases and model errors, such as persistently lower surface temperatures compared with observations, and can apply corrections to the output more quickly and systematically than traditional statistical methods. People only acquire intuition for error identification over time.

Another AI technique, deep learning, works similarly to machine learning but extracts features such as low-pressure zones and tropical cyclones from an image by breaking them down to smaller components. Deep learning is often referred to as a black box because the smaller, hidden components of image features are unrecognizable to a human. The algorithm compares those deconstructed components of features with information it’s been trained to recognize “from an independent dataset. Forecasters can assimilate the satellite imagery analyzed by the deep-learning method into the weather model to make better predictions. Last year, the amount of weather data increased by several orders of magnitude after NOAA began releasing data from its latest Geostationary Operational Environmental Satellite, GOES-17.

A healthy skepticism

Some computer scientists have tried generalizing an AI algorithm designed for forecasting a specific weather feature, such as hail, to another, such as severe wind. But each feature has its own dynamical properties that interact with each other in different ways. “You can’t just take the same black box and apply it to different problems. If the model doesn’t make sense physically, meteorologists aren’t going to use it,” says Amy McGovern, a computer science professor at the University of Oklahoma.

An AI method is only as good as the training data it uses. Some of the best sources of data, including the European Centre for Medium-Range Weather Forecasts, charge for access. And, says McGovern, “companies jumping in aren’t willing to share data. They live in a world where their competitors can get an edge.”

How data are collected may change to bolster the use of AI in forecasting. The training data must first be labeled and classified by a person or pass through a digital quality-control check. Quick and effective labeling by meteorologists or by the data-collection system itself would speed up the AI development cycle.

Computer scientists are still figuring out how to generalize an algorithm to work in different scenarios. AI methods, particularly deep learning, don’t extrapolate well when the information used in their training changes to an unfamiliar pattern. Climate change makes training data less representative by altering the intensity, frequency, and spatial extent of extreme weather events.

Putting theory into practice

In meteorology and other fields, researchers are still working out how to overcome the limitations of nonrepresentative training data and develop more flexible AI. Despite the challenges, applying AI to weather forecasting is gaining in popularity. Several hundred people attended the AI applications conference at the American Meteorological Society’s annual meeting in January, up from 40 or 50 a decade ago, says John Williams, former AI committee chair and a senior manager at the Weather Company, a forecasting and technology business of IBM.

AI adoption has already started, says Rich Sorkin, CEO and cofounder of Jupiter Intelligence, a Silicon Valley–based company that predicts damage, risk, and cost from severe weather and climate change. “A lot of the new graduates have been exposed to this in their academic studies, or they’re just reasonably well-read,” he says.

Later this year the Weather Company will deploy the IBM Global High-Resolution Atmospheric Forecasting System to provide hourly updates of weather conditions at 3 km resolution on land and forecasts out to 15 hours. The system will use machine learning to automate quality control, fuse data from many different sources—including pressure measurements collected with cell-phone barometers—and apply corrections to the weather model output.

Since 2000 the NOAA Hazardous Weather Testbed in Norman, Oklahoma, has brought together meteorologists, computer scientists, software developers, broadcasters, and students for five to six weeks each spring to experiment with and evaluate the many emerging advances in severe weather forecasting. As part of the program, researchers and forecasters use the latest methods, including AI, to make daily severe weather predictions. Forecasters benefit from exposure to the latest scientific advances, and researchers gain valuable feedback about the strengths and weaknesses of their experimental methods. “Here, promising techniques can be shared with on-duty storm forecasters with almost the flip of a switch,” says Patrick Marsh, a meteorologist at NOAA’s Storm Prediction Center in Norman.

The promise that AI methods show in the experimental test bed doesn’t guarantee success in an operational setting, cautions Marsh. The NWS weighs the benefits against the costs of each forecasting approach to uphold its mission of protecting life and property. To focus on producing superior hurricane-track forecasts using AI at the expense of accurate hurricane intensity, for example, may result in an overall worse prediction. “The AI is the easy part,” says Marsh. “Prioritizing the results is the really hard part.”

As AI weather forecasting enters the economy, some meteorologists worry that computers may take the jobs of people. “This is part of a much larger conversation we’re having in our society,” says Marsh. “Most people prefer a human there when the machine goes off the rails.” And if forecasters aren’t practicing their craft, skills and knowledge will atrophy over time, he says. A successful workflow requires humans to be in the loop, he adds. Humans can act both as a check for the AI and as a resource for AI developers to strive for the best possible weather forecast.