Stochastics & Finance seminar: Qu -- Dynamical and computational properties of heavy-tailed deep neural networks

Dear All, 

You are kindly invited to attend the next Stochastics and Finance seminar.  On Tuesday
April 4 at 2pm, Kevin Qu will give a talk at EA LT 315 (and on zoom).  

Location: Eastern Avenue Lecture Theatre 315 

Zoom link: https://uni-sydney.zoom.us/j/84240252220 

Speaker: Kevin Qu (School of Physics, University of Sydney) 

Title: Dynamical and computational properties of heavy-tailed deep neural networks 

Abstract: In deep neural networks (DNNs), the assumption of Gaussian statistics in
theoretical studies has been pervasive.  However, empirical data has shown that heavy-
tailed connectivity is prevalent in commonly used pretrained DNNs.  To elucidate the
fundamental impact of such heavy-tailed connectivity on the dynamical and computational
properties of DNNs, we develop a novel mean field theory that integrates theories of
heavy-tailed random matrices and non-equilibrium statistical physics.  Our theoretical
framework demonstrates that heavy-tailed weights enable the emergence of the extended
criticality (edge-of-chaos) in a broad parameter region, leading to improved and faster
learning of real-world tasks without the need to fine-tune the weight statistics,
compared to networks with Gaussian weights.  Notably, we find that heavy-tailed DNNs
exhibit multifractal eigenvectors and internal neural representations of input data,
which are associated with profound computational advantages such as enhanced robustness
and increased signal-to-noise ratio in the context of few-shot learning.  Overall, our
findings challenge the commonly held assumption of Gaussian statistics in DNNs and offer
new insights into the underlying mechanisms of deep learning.  

This is a joint project with Asem Wardak and Pulin Gong under the Complex Systems,
School of Physics.  

https://www.maths.usyd.edu.au/u/SemConf/Stochastics_Finance/seminar.html 

Please feel free to forward this message to anyone who might be interested in this
talk.  

Best wishes, Anna