Asset Details
Do you wish to reserve the book?
Width and Depth Limits Commute in Residual Networks
Hey, we have placed the reservation for you!
By the way, why not check out events that you can attend while you pick your title.
Oops! Something went wrong.
Looks like we were not able to place the reservation. Kindly try again later.
Are you sure you want to remove the book from the shelf?
Width and Depth Limits Commute in Residual Networks
Do you wish to request the book?
Width and Depth Limits Commute in Residual Networks
Please be aware that the book you have requested cannot be checked out. If you would like to checkout this book, you can reserve another copy
We have requested the book for you!
Your request is successful and it will be processed during the Library working hours. Please check the status of your request in My Requests.
Oops! Something went wrong.
Looks like we were not able to place your request. Kindly try again later.
Paper
Width and Depth Limits Commute in Residual Networks
2023
Overview
We show that taking the width and depth to infinity in a deep neural network with skip connections, when branches are scaled by \\(1/depth\\) (the only nontrivial scaling), result in the same covariance structure no matter how that limit is taken. This explains why the standard infinite-width-then-depth approach provides practical insights even for networks with depth of the same order as width. We also demonstrate that the pre-activations, in this case, have Gaussian distributions which has direct applications in Bayesian deep learning. We conduct extensive simulations that show an excellent match with our theoretical findings.
Publisher
Cornell University Library, arXiv.org
Subject
