Journal Article

WeatherBench 2: A Benchmark for the Next Generation of Data‐Driven Global Weather Models

Rasp, Stephan, Sha, Fei, Bromberg, Carla, Bell, Aaron, Battaglia, Peter, Hoyer, Stephan, Ben Bouallegue, Zied, Russell, Tyler, Carver, Rob, Barrington, Luke, Chantry, Matthew, Yang, Vivian, Sisk, Jared, Langmore, Ian, Agrawal, Shreya, Sanchez‐Gonzalez, Alvaro, Dueben, Peter, Merose, Alexander

2024

Overview

WeatherBench 2 is an update to the global, medium‐range (1–14 days) weather forecasting benchmark proposed by (Rasp et al., 2020, https://doi.org/10.1029/2020ms002203), designed with the aim to accelerate progress in data‐driven weather modeling. WeatherBench 2 consists of an open‐source evaluation framework, publicly available training, ground truth and baseline data as well as a continuously updated website with the latest metrics and state‐of‐the‐art models: https://sites.research.google/weatherbench. This paper describes the design principles of the evaluation framework and presents results for current state‐of‐the‐art physical and data‐driven weather models. The metrics are based on established practices for evaluating weather forecasts at leading operational weather centers. We define a set of headline scores to provide an overview of model performance. In addition, we also discuss caveats in the current evaluation setup and challenges for the future of data‐driven weather forecasting. Plain Language Summary Traditionally, weather forecasts are made by models that attempt to replicate the physical processes of the atmosphere. This has been very successful over the last few decades as better computers, better observations and model upgrades have lead to steadily improving weather forecasts. However, with rapid advances in artificial intelligence (AI), the question can be asked whether one can simply learn a weather model from past observations or reanalyzes. In the last couple of years, we have seen tremendous progress with state‐of‐the‐art AI models rivaling the best “traditional” weather models in skill. WeatherBench 2 is a benchmark data set designed to evaluate and compare the quality of AI and traditional models. By setting a standard for evaluation, alongside providing open‐source data and code, this project aims to accelerate this research direction and lead to better weather prediction. Key Points WeatherBench 2 is a framework for evaluating and comparing data‐driven and traditional numerical weather forecasting models It provides an evaluation framework, publicly available data sets and a website to assess the state‐of‐the‐art weather models The evaluation protocol has been designed following best practices established in the operational weather forecasting community

• Publisher: John Wiley & Sons, Inc,American Geophysical Union (AGU)
• Subject: Algorithms / Artificial intelligence / Baseline studies / benchmark / Benchmarks / Data assimilation / Datasets / Forecasting data / Global weather / Machine learning / NWP / Weather forecasting