Journal Article

Integrating Recurrent Neural Networks With Data Assimilation for Scalable Data‐Driven State Estimation

Goodliff, M., Penny, S. G., Chen, T.‐C., Platt, J. A., Abarbanel, H. D. I., Smith, T. A., Lin, H.‐Y.

2022

Overview

Data assimilation (DA) is integrated with machine learning in order to perform entirely data‐driven online state estimation. To achieve this, recurrent neural networks (RNNs) are implemented as pretrained surrogate models to replace key components of the DA cycle in numerical weather prediction (NWP), including the conventional numerical forecast model, the forecast error covariance matrix, and the tangent linear and adjoint models. It is shown how these RNNs can be initialized using DA methods to directly update the hidden/reservoir state with observations of the target system. The results indicate that these techniques can be applied to estimate the state of a system for the repeated initialization of short‐term forecasts, even in the absence of a traditional numerical forecast model. Further, it is demonstrated how these integrated RNN‐DA methods can scale to higher dimensions by applying domain localization and parallelization, providing a path for practical applications in NWP. Plain Language Summary Weather forecast models derived from fundamental equations of physics continue to increase in detail and complexity. While this evolution leads to consistently improving daily weather forecasts, it also leads to associated increases in computational costs. In order to make a forecast at any given moment, these models must be initialized with our best guess of the current state of the atmosphere, which typically includes information from a limited set of observations as well as forecasts from the recent past. Modern methods for initializing these computer forecasts typically require running many copies of the model, either simultaneously or in sequence, to compare with observations over the recent past and ensure that our best guess estimate of the current state of the atmosphere agrees closely with those observations before making a new forecast. This repeated execution of the computer forecast model is often a time‐consuming and costly bottleneck in the initialization process. Here, it is shown that techniques from the fields of artificial intelligence and machine learning (AI/ML) can be used to produce simple surrogate models that provide sufficiently accurate approximations to replace the original costly model in the initialization phase. The resulting process is self‐contained, and does not require any further utilization of the original computer model when making daily forecasts. Key Points Recurrent neural networks (RNNs) can replace conventional forecast models, producing accurate ensemble forecast statistics and linearized dynamics Data assimilation (DA) is compatible with RNNs by applying state estimation in the hidden state space using a modified observation operator The integrated RNN‐DA methods can be scaled to higher dimensions by applying domain localization techniques

• Publisher: John Wiley & Sons, Inc,American Geophysical Union (AGU)
• Subject: 4D‐var / Adjoint models / Artificial intelligence / Data assimilation / Data collection / Dimensions / ensemble kalman filter / Machine learning / Methods / Modelling / Neural networks / Numerical forecasting / Numerical forecasting models / recurrent neural networks / Weather forecasting