Expanding the attention mechanism in language models, rather than compressing it, has been found to significantly accelerate learning speed. By modifying the standard attention computation to include a learned projection matrix U, where the rank of U is greater than the dimensionality d_k, the model can achieve faster convergence despite more compute per step. This approach was discovered accidentally through hyperparameter drift, resulting in a smaller model that quickly acquired coherent English grammar. The key insight is that while attention routing benefits from expanded "scratch space," value aggregation should remain at full dimensionality. This finding challenges the common focus on compression in existing literature and suggests new possibilities for enhancing model efficiency and performance. Summary: Expanding attention mechanisms in language models can dramatically improve learning speed, challenging the traditional focus on compression for efficiency.
