Research into the Llama 3.2 3B fMRI model reveals intriguing patterns in the correlation of hidden activations across layers. Most correlated dimensions are transient, appearing briefly in specific layers and then vanishing, suggesting short-lived subroutines rather than stable features. Some dimensions persist in specific layers, indicating mid-to-late control signals, while a small set of dimensions recur across different prompts and layers, maintaining stable polarity. The research aims to further isolate these recurring dimensions to better understand their roles, potentially leading to insights into the model's inner workings. Understanding these patterns matters as it could enhance the interpretability and reliability of complex AI models.
