2024-08-14-aswani24a.md

title	abstract	openreview	layout	series	publisher	issn	id	month	tex_title	firstpage	lastpage	page	order	cycles	bibtex_author	author	date	address	container-title	volume	genre	issued	pdf	extras
Examining Changes in Internal Representations of Continual Learning Models Through Tensor Decomposition	Continual learning (CL) has spurred the development of several methods aimed at consolidating previous knowledge across sequential learning. Yet, the evaluations of these methods have primarily focused on the final output, such as changes in the accuracy of predicted classes, overlooking the issue of representational forgetting within the model. In this paper, we propose a novel representation-based evaluation framework for CL models. This approach involves gathering internal representations from throughout the continual learning process and formulating three-dimensional tensors. The tensors are formed by stacking representations, such as layer activations, generated from several inputs and model ‘snapshots’, throughout the learning process. By conducting tensor component analysis (TCA), we aim to uncover meaningful patterns about how the internal representations evolve, expecting to highlight the merits or shortcomings of examined CL strategies. We conduct our analyses across different model architectures and importance-based continual learning strategies, with a curated task selection. Often, the results of our approach mirror the difference in performance of various CL strategies on various architectures. Ultimately, however, we found that our methodology did not directly highlight specialized clusters of neurons, nor provide an immediate understanding the evolution of filters. We believe a scaled down variation of our approach will provide insight into the benefits and pitfalls of using TCA to study continual learning dynamics.	pyjLrj4o8y	inproceedings	Proceedings of Machine Learning Research	PMLR	2640-3498	aswani24a	0	Examining Changes in Internal Representations of Continual Learning Models Through Tensor Decomposition	62	82	62-82	62	false	Aswani, Nishant Suresh and Guesmi, Amira and Hanif, Muhammad Abdullah and Shafique, Muhammad	given family Nishant Suresh Aswani given family Amira Guesmi given family Muhammad Abdullah Hanif given family Muhammad Shafique	2024-08-14		Proceedings of the 1st ContinualAI Unconference, 2023	249	inproceedings	date-parts 2024 8 14	https://raw.githubusercontent.com/mlresearch/v249/main/assets/aswani24a/aswani24a.pdf