2024-10-09-grazzi24a.md

title	openreview	abstract	layout	series	publisher	issn	id	month	tex_title	firstpage	lastpage	page	order	cycles	bibtex_author	author	date	address	container-title	volume	genre	issued	pdf	extras
Is Mamba Capable of In-Context Learning?	rJhOG0P8nr	The surprising generalization capabilities of foundation models have been enabled by in-context learning (ICL), a new variant of meta-learning that denotes the learned ability to solve tasks during a neural network forward pass, exploiting contextual information provided as input to the model. This useful ability emerges as a side product of the foundation model’s massive pretraining. While transformer models are currently the state of the art in ICL, this work provides empirical evidence that Mamba, a newly proposed state space model which scales better than transformers w.r.t. the input sequence length, has similar ICL capabilities. We evaluated Mamba on tasks involving simple function approximation as well as more complex natural language processing problems. Our results demonstrate that, across both categories of tasks, Mamba closely matches the performance of transformer models for ICL. Further analysis reveals that, like transformers, Mamba appears to solve ICL problems by incrementally optimizing its internal representations. Overall, our work suggests that Mamba can be an efficient alternative to transformers for ICL tasks involving long input sequences. This is an exciting finding in meta-learning and may also enable generalizations of in-context learned AutoML algorithms (like TabPFN or Optformer) to long input sequences. The anonymous code to reproduce our experiments is available at \url{https://anon-github.automl.cc/r/is_mamba_capable_of_in_context_learning-7C49/README.md}.	inproceedings	Proceedings of Machine Learning Research	PMLR	2640-3498	grazzi24a	0	Is Mamba Capable of In-Context Learning?	1/1	26	1/1-26	1	false	Grazzi, Riccardo and Siems, Julien Niklas and Schrodi, Simon and Brox, Thomas and Hutter, Frank	given family Riccardo Grazzi given family Julien Niklas Siems given family Simon Schrodi given family Thomas Brox given family Frank Hutter	2024-10-09		Proceedings of the Third International Conference on Automated Machine Learning	256	inproceedings	date-parts 2024 10 9	https://raw.githubusercontent.com/mlresearch/v256/main/assets/grazzi24a/grazzi24a.pdf