Gutenberg Open Science: Studying the evolution of neural activation patterns during training of feed-forward ReLU networks

Please use this identifier to cite or link to this item: http://doi.org/10.25358/openscience-6840

Full metadata record

DC Field	Value	Language
dc.contributor.author	Hartmann, David	-
dc.contributor.author	Franzen, Daniel	-
dc.contributor.author	Brodehl, Sebastian	-
dc.date.accessioned	2022-03-24T11:00:09Z	-
dc.date.available	2022-03-24T11:00:09Z	-
dc.date.issued	2021	-
dc.identifier.uri	https://openscience.ub.uni-mainz.de/handle/20.500.12030/6851	-
dc.description.abstract	The ability of deep neural networks to form powerful emergent representations of complex statistical patterns in data is as remarkable as imperfectly understood. For deep ReLU networks, these are encoded in the mixed discrete–continuous structure of linear weight matrices and non-linear binary activations. Our article develops a new technique for instrumenting such networks to efficiently record activation statistics, such as information content (entropy) and similarity of patterns, in real-world training runs. We then study the evolution of activation patterns during training for networks of different architecture using different training and initialization strategies. As a result, we see characteristic- and general-related as well as architecture-related behavioral patterns: in particular, most architectures form bottom-up structure, with the exception of highly tuned state-of-the-art architectures and methods (PyramidNet and FixUp), where layers appear to converge more simultaneously. We also observe intermediate dips in entropy in conventional CNNs that are not visible in residual networks. A reference implementation is provided under a free license1.	en_GB
dc.description.sponsorship	Open Access-Publizieren Universität Mainz / Universitätsmedizin Mainz	de
dc.language.iso	eng	de
dc.rights	CC BY	*
dc.rights.uri	https://creativecommons.org/licenses/by/4.0/	*
dc.subject.ddc	004 Informatik	de_DE
dc.subject.ddc	004 Data processing	en_GB
dc.title	Studying the evolution of neural activation patterns during training of feed-forward ReLU networks	en_GB
dc.type	Zeitschriftenaufsatz	de
dc.identifier.doi	http://doi.org/10.25358/openscience-6840	-
jgu.type.dinitype	article	en_GB
jgu.type.version	Published version	de
jgu.type.resource	Text	de
jgu.organisation.department	FB 08 Physik, Mathematik u. Informatik	de
jgu.organisation.number	7940	-
jgu.organisation.name	Johannes Gutenberg-Universität Mainz	-
jgu.rights.accessrights	openAccess	-
jgu.journal.title	Frontiers in artificial intelligence	de
jgu.journal.volume	4	de
jgu.pages.alternative	642374	de
jgu.publisher.year	2021	-
jgu.publisher.name	Frontiers Media	de
jgu.publisher.place	Lausanne	de
jgu.publisher.issn	2624-8212	de
jgu.organisation.place	Mainz	-
jgu.subject.ddccode	004	de
jgu.publisher.doi	10.3389/frai.2021.642374	de
jgu.organisation.ror	https://ror.org/023b0x485
Appears in collections:	JGU-Publikationen

Files in This Item:

	File	Description	Size	Format
	studying_the_evolution_of_neu-20220322111055686.pdf		2.75 MB	Adobe PDF	View/Open

Show simple item record