AI techniques including machine learning and data assimilation can improve weather forecasting accuracy, according to a new study.

Data scientists from Imperial’s Data Science Institute and the Department of Earth Science and Engineering used local atmospheric data and data assimilation (combining real-time observational data with machine learning models) - to improve the accuracy of weather forecasts.

“Our study shows a potential path for future research to elevate prediction accuracy by appropriate selection, processing, integration, and computation of diverse data sources. The findings indicate that such selective integration can substantially improve forecasting accuracy moving forward.” Wenqi Wang Lead Author

The study was presented in the ‘Tackling Climate Change with Machine Learning' workshop at the NeurIPS 2023 Conference, and demonstrates the potential of integrating machine learning with data assimilation techniques to enhance regional weather forecasting.

By harnessing temperature data from the local area’s lower atmosphere (at around 1500 metres above sea level) and improving a widely used global weather forecasting model known as U-STN to tailor its predictions to the UK's climate features, the researchers have made significant strides in enhancing forecast results.

According to Lead Author Wenqi Wang, “Our study shows a potential path for future research to elevate prediction accuracy by appropriate selection, processing, integration, and computation of diverse data sources. The findings indicate that such selective integration can substantially improve forecasting accuracy moving forward.”

From Data to Forecasting: Bridging the gap with machine learning and data assimilation

In terms of predicting the weather, the ability to seamlessly integrate data from diverse sources is essential.

Currently, the integration of varied and ambiguously related data sources poses a significant hurdle for traditional forecasting methods. However, by utilising machine learning, scientists can integrate different datasets effectively to enhance model accuracy.

Numerical Weather Prediction (NWP) is one method that meteorologists have used and continues to widely use to forecast the weather by simulating the behaviour of the atmosphere using mathematical models. However, the machine learning model’s increasing efficiency, marked by a substantial reduction in computational resources and processing time, has shown considerable application potential.

These machine learning models take into account various atmospheric variables such as temperature, pressure, humidity and wind to predict how the weather will evolve. The models have evolved over the years, incorporating advanced algorithms, high-resolution data and improved computational power to enhance forecast accuracy and reliability.

Data assimilation is a crucial component of the weather forecasting process. It incorporates various processed atmospheric data and real-time surface data from ground observational sites into the models to produce more accurate forecasts.

According to Wang, “We trained a machine learning model to forecast short-term weather in the UK region, utilising the processed data. Imagine this as creating a map where, based on specific weather conditions (our parameters), you would have been able to pinpoint what the weather will be like.  Initially, our predictions had inaccuracies, similar to aiming for a target on the map but missing the precise spot. Data assimilation steps in as a critical tool to correct these inaccuracies. It's about making the computer's guess as close to the real thing as possible.

We compared the differences between observations and the model’s predictions through the data assimilation operator. This comparison allowed me to assess how well the observation data could correct the initial model —like using a compass to correct the course during navigation. This step helped me determine whether the model’s forecasts were on the right track or needed some tweaking.” 

By combining real-time observations with existing model predictions, data assimilation helps meteorologists improve their forecasts and better understand the evolving state of the atmosphere.

Future of forecasting

"Europe has just invested 150M€ in an initiative called Destination Earth which will give us an unprecedented availability of data to be used in Data Learning models developed in the DSI and more broadly in the Data & AI 4 Climate community." Dr Rossella Arcucci DSI Director of Research

As meteorological data continues to evolve and expand, the ability to adapt to new information in real-time becomes increasingly crucial.

The Data Learning Group and the new Data & AI 4 Climate Lab at Imperial are at the forefront of this adaptive approach, utilising machine learning models that can continuously self-optimise and adjust to new data inputs.

Lead of the Data Learning Group Dr Rossella Arcucci said: "Europe has just invested 150M€ in an initiative called Destination Earth which will give us an unprecedented availability of data to be used in Data Learning models developed in the DSI and more broadly in the Data & AI 4 Climate community."

Moving forward, the researchers hope to optimise the selected data, evolve the model’s dimensionality and expand the geographical coverage.

Wang said: “Our research is based on a 3D model based solely on a singular observation source. The object is to transition to a 4D model. This enhancement implies utilising data from identical geographical points across different atmospheric pressure levels.”

As we look ahead to a world where weather prediction plays an increasingly vital role in our daily lives, machine learning and data assimilation will be crucial in atmospheric and climate science.

-

‘Data Assimilation using ERA5, ASOS, and the U-STN model for Weather Forecasting over the UK’ by Wang et al., was presented in the ‘Tackling Climate Change with Machine Learning workshop at the NeurIPS 2023 Conference.