Journal Article

Architectural richness in deep reservoir computing

Micheli, Alessio, Gallicchio, Claudio

2023

Overview

Reservoir computing (RC) is a popular class of recurrent neural networks (RNNs) with untrained dynamics. Recently, advancements on deep RC architectures have shown a great impact in time-series applications, showing a convenient trade-off between predictive performance and required training complexity. In this paper, we go more in depth into the analysis of untrained RNNs by studying the quality of recurrent dynamics developed by the layers of deep RC neural networks. We do so by assessing the richness of the neural representations in the different levels of the architecture, using measures originating from the fields of dynamical systems, numerical analysis and information theory. Our experiments, on both synthetic and real-world datasets, show that depth—as an architectural factor of RNNs design—has a natural effect on the quality of RNN dynamics (even without learning of the internal connections). The interplay between depth and the values of RC scaling hyper-parameters, especially the scaling of inter-layer connections, is crucial to design rich untrained recurrent neural systems.

• Publisher: Springer London,Springer Nature B.V
• Subject: Artificial Intelligence / Computation / Computational Biology/Bioinformatics / Computational Science and Engineering / Computer Science / Data Mining and Knowledge Discovery / Design factors / Dynamical systems / Image Processing and Computer Vision / Information theory / Neural networks / Numerical analysis / Performance prediction / Probability and Statistics in Computer Science / Recurrent neural networks / S.I. : IWANN 2019 SI on Advances in Computational Intelligence