Journal Article

Deep-Learning-Based Multispectral Image Reconstruction from Single Natural Color RGB Image—Enhancing UAV-Based Phenotyping

Balram Marathi, Seishi Ninomiya, Wei Guo, Balaji Naik Banoth, Pachamuthu Rajalakshmi, Jiangsan Zhao, Ajay Kumar, Boris Rewald

2022

Overview

Multispectral images (MSIs) are valuable for precision agriculture due to the extra spectral information acquired compared to natural color RGB (ncRGB) images. In this paper, we thus aim to generate high spatial MSIs through a robust, deep-learning-based reconstruction method using ncRGB images. Using the data from the agronomic research trial for maize and breeding research trial for rice, we first reproduced ncRGB images from MSIs through a rendering model, Model-True to natural color image (Model-TN), which was built using a benchmark hyperspectral image dataset. Subsequently, an MSI reconstruction model, Model-Natural color to Multispectral image (Model-NM), was trained based on prepared ncRGB (ncRGB-Con) images and MSI pairs, ensuring the model can use widely available ncRGB images as input. The integrated loss function of mean relative absolute error (MRAEloss) and spectral information divergence (SIDloss) were most effective during the building of both models, while models using the MRAEloss function were more robust towards variability between growing seasons and species. The reliability of the reconstructed MSIs was demonstrated by high coefficients of determination compared to ground truth values, using the Normalized Difference Vegetation Index (NDVI) as an example. The advantages of using “reconstructed” NDVI over Triangular Greenness Index (TGI), as calculated directly from RGB images, were illustrated by their higher capabilities in differentiating three levels of irrigation treatments on maize plants. This study emphasizes that the performance of MSI reconstruction models could benefit from an optimized loss function and the intermediate step of ncRGB image preparation. The ability of the developed models to reconstruct high-quality MSIs from low-cost ncRGB images will, in particular, promote the application for plant phenotyping in precision agriculture.

• Publisher: MDPI AG,MDPI
• Subject: [INFO.INFO-LG]Computer Science [cs]/Machine Learning [cs.LG] / [INFO.INFO-TI]Computer Science [cs]/Image Processing [eess.IV] / [SDE.IE]Environmental Sciences/Environmental Engineering / [SDV.SA.STA]Life Sciences [q-bio]/Agricultural sciences/Sciences and technics of agriculture / Agricultural engineering / Agricultural sciences / Agriculture / Color / Color imagery / Computer Science / Corn / data collection / Datasets / Deep learning / Digital cameras / Divergence / Electrical Engineering / Environmental Engineering / Environmental Sciences / Experiments / Growing season / hyperspectral imagery / Hyperspectral imaging / Image acquisition / Image enhancement / Image Processing / Image quality / Image reconstruction / Life Sciences / loss function optimization / Machine Learning / multispectral image reconstruction / multispectral image reconstruction; natural color RGB image; deep learning; loss function optimization; precision agriculture / multispectral imagery / natural color RGB image / normalized difference vegetation index / Normalized difference vegetative index / phenotype / Phenotyping / Plant breeding / Precision agriculture / Q / Rice / Robustness / Science / Sciences and technics of agriculture / Sensors / unmanned aerial vehicles