Please use this identifier to cite or link to this item:
Authors: Hartmann, David
Franzen, Daniel
Brodehl, Sebastian
Title: Studying the evolution of neural activation patterns during training of feed-forward ReLU networks
Online publication date: 24-Mar-2022
Year of first publication: 2021
Language: english
Abstract: The ability of deep neural networks to form powerful emergent representations of complex statistical patterns in data is as remarkable as imperfectly understood. For deep ReLU networks, these are encoded in the mixed discrete–continuous structure of linear weight matrices and non-linear binary activations. Our article develops a new technique for instrumenting such networks to efficiently record activation statistics, such as information content (entropy) and similarity of patterns, in real-world training runs. We then study the evolution of activation patterns during training for networks of different architecture using different training and initialization strategies. As a result, we see characteristic- and general-related as well as architecture-related behavioral patterns: in particular, most architectures form bottom-up structure, with the exception of highly tuned state-of-the-art architectures and methods (PyramidNet and FixUp), where layers appear to converge more simultaneously. We also observe intermediate dips in entropy in conventional CNNs that are not visible in residual networks. A reference implementation is provided under a free license1.
DDC: 004 Informatik
004 Data processing
Institution: Johannes Gutenberg-Universität Mainz
Department: FB 08 Physik, Mathematik u. Informatik
Place: Mainz
Version: Published version
Publication type: Zeitschriftenaufsatz
License: CC BY
Information on rights of use:
Journal: Frontiers in artificial intelligence
Pages or article number: 642374
Publisher: Frontiers Media
Publisher place: Lausanne
Issue date: 2021
ISSN: 2624-8212
Publisher DOI: 10.3389/frai.2021.642374
Appears in collections:JGU-Publikationen

Files in This Item:
  File Description SizeFormat
studying_the_evolution_of_neu-20220322111055686.pdf2.75 MBAdobe PDFView/Open