Barbara, USA
Title: Neural Networks: Spurious Minima, the Standard Representation of Symmetric Group, and Bias.
Abstract: We start with a review of past collaborative work with Yossi Arjevani (The Hebrew University, Jerusalem) on bias-free neural networks. One application discussed uses the codimension one theory of equivariant bifurcation on the standard representation of the symmetric group. Although the codimension 1 theory of equivariant bifurcation on the standard representation of the symmetric group has been known for about 35 years (work of RW Richardson and MF), there are some intriguing features of the bifurcation that seem not to be so widely known and these turn out to be helpful in understanding theoretical problems in machine learning. Specifically, the creation and annihilation of spurious minima in terms of the width of the network and the number of neurons. A spurious minimum is local minimum which is not the global minimum (zero) of the loss function. On the other hand, it is not clear if results on bias free networks extend to biased networks. Without bias, neural networks do not give approximation to general continuous functions (recent work of Zhang, Saxe & Latham).
In the second half of the talk this matter is addressed and largely resolved (positively).
Bio: Professor Mike Field’s research is mainly in dynamical systems and includes statistical properties (ergodic theory and mixing), network dynamics with special reference to asynchronous networks, adaptation, event driven dynamics and non-smooth dynamics, and the geometric theory of dynamical systems with symmetry and the mechanisms whereby symmetry can lead to complex dynamics in low dimensional systems. Areas of application include spiking neuronal networks, plasticity and visualization. Currently, he is working on theoretical problems in machine learning related to non-convex optimization and gradient descent.