Journal Article

Addressing Out‐of‐Sample Issues in Multi‐Layer Convolutional Neural‐Network Parameterization of Mesoscale Eddies Applied Near Coastlines

Perezhogin, Pavel, Zanna, Laure, Zhang, Cheng, Adcroft, Alistair

2025

Overview

This study addresses the boundary artifacts in machine‐learned (ML) parameterizations for ocean subgrid mesoscale momentum forcing, as identified in the online ML implementation from a previous study (Zhang et al., 2023, https://doi.org/10.1029/2023ms003697). We focus on the boundary condition (BC) treatment within the existing convolutional neural network (CNN) models and aim to mitigate the “out‐of‐sample” errors observed near complex coastal regions without developing new, complex network architectures. Our approach leverages two established strategies for placing BCs in CNN models, namely zero and replicate padding. Offline evaluations revealed that these padding strategies significantly reduce root mean squared error (RMSE) in coastal regions by limiting the dependence on random initialization of weights and restricting the range of out‐of‐sample predictions. Further online evaluations suggest that replicate padding consistently reduces boundary artifacts across various retrained CNN models. In contrast, zero padding sometimes intensifies artifacts in certain retrained models despite both strategies performing similarly in offline evaluations. This study underscores the need for BC treatments in CNN models trained on open water data when predicting near‐coastal subgrid forces in ML parameterizations. The application of replicate padding, in particular, offers a robust strategy to minimize the propagation of extreme values that can contaminate computational models or cause simulations to fail. Our findings provide insights for enhancing the accuracy and stability of ML parameterizations in the online implementation of ocean circulation models with coastlines. Plain Language Summary This study focuses on improving machine learning (ML) models used to predict ocean forces near coastlines, where errors arise because these models lack information in the area. We investigated how boundary conditions are handled in existing convolutional neural network models to reduce these errors without creating complex new architectures. By using two methods, that is, zero padding and replicate padding, we found that replicate padding significantly decreases prediction errors in coastal areas. While zero padding sometimes worsens issues in certain models, our results show that replicate padding is more reliable for effectively minimizing extreme value errors. This work highlights the importance of proper boundary condition treatment in ML models for coastal applications, ultimately aiming to enhance the accuracy and reliability of ocean circulation predictions. Key Points This study validates specialized boundary condition treatments in CNN models to reduce boundary artifacts in ocean parameterizations This approach can be applied directly to already trained CNN models to ensure accurate and stable implementation of mesoscale eddies parameterizations Replicate padding outperforms zero padding by minimizing boundary artifacts and preventing extreme values that compromise simulations

• Publisher: John Wiley & Sons, Inc,American Geophysical Union (AGU)
• Subject: boundary condition treatments / Boundary conditions / Climate change / Coastal circulation / Coastal zone / Coasts / convolutional neural network / Eddies / General circulation models / Machine learning / Mesoscale eddies / mesoscale eddy parameterizations / Momentum / Neural networks / Ocean circulation / ocean circulation model / Ocean circulation models / Oceans / out‐of‐sample prediction / Parameterization / Shorelines / Water circulation