Paper

Beyond Folklore: A Scaling Calculus for the Design and Initialization of ReLU Networks

Defazio, Aaron, Bottou, Léon

2021

Overview

We propose a system for calculating a \"scaling constant\" for layers and weights of neural networks. We relate this scaling constant to two important quantities that relate to the optimizability of neural networks, and argue that a network that is \"preconditioned\" via scaling, in the sense that all weights have the same scaling constant, will be easier to train. This scaling calculus results in a number of consequences, among them the fact that the geometric mean of the fan-in and fan-out, rather than the fan-in, fan-out, or arithmetic mean, should be used for the initialization of the variance of weights in a neural network. Our system allows for the off-line design & engineering of ReLU neural networks, potentially replacing blind experimentation.

• Publisher: Cornell University Library, arXiv.org
• Subject: Conditioning / Multilayer perceptrons / Neural networks / Preconditioning / Scaling laws