Journal Article

Denoising Autoencoder for Reconstructing Sensor Observation Data and Predicting Evapotranspiration: Noisy and Missing Values Repair and Uncertainty Quantification

Varadharajan, Charuleka, Johnsen, Timothy K, Bi, Xiangyu, Chou, Chunwei, Ramakrishnan, Lavanya, Wu, Yuxin, Skone, Jonathan

2025

Overview

Machine learning (ML) methods applied in scientific research often deal with interrelated features in high‐dimensional data. Reducing data noise and redundancy is needed to increase prediction accuracy and efficiency especially when dealing with data from field sensors. We explored an unsupervised learning method, the denoising autoencoder (DAE), to extract the underlying data structure from noisy raw data in the context of predicting hydrologic quantities from multiple field sensors. These sensors have intrinsic instrumental noise and occasional malfunctions that cause missing values. Our DAE neural network reconstructed meteorological sensor data containing noise and missing values to predict evapotranspiration in a mountainous watershed. The DAE reconstructed the sensor variables with a mean coefficient of determination r2 $\\left({r}^{2}\\right)$ value of 0.77 across 15 dimensions representing individual sensors. It reduced variance and bias uncertainties compared to a classical autoencoder model. The reconstruction quality varied across dimensions depending on their cross‐correlation and alignment with the underlying data structure. Uncertainties arising from the model structure were overall higher than those resulting from data corruption. We attached the DAE structure to a downstream ET‐prediction neural network in three formats and achieved reasonably accurate ET predictions r2≃0.7 $\\left({r}^{2}\\simeq 0.7\\right)$. The use of the DAE notably reduced variance uncertainty in ET prediction. However, excessive variance reduction may be accompanied by an increase in bias due to the intrinsic bias‐variance tradeoff. Our method of evaluating and reducing uncertainties in aggregated data from different sources can be used to improve predictive models, process understanding, and uncertainty quantification for better water resource management.

• Publisher: John Wiley & Sons, Inc
• Subject: Algorithms / Bias / Data reduction / Data structures / Datasets / Estimates / Evapotranspiration / Hydrology / Machine learning / Neural networks / Noise / Noise prediction / Noise reduction / Precipitation / Prediction models / Resource management / Scientific research / Sensors / Uncertainty / Unsupervised learning / Variables / Variance / Water resources / Water resources management