Grafted Titans: Enhancing LLMs with Neural Memory

An experiment with Test-Time Training (TTT) aimed to replicate Google’s “Titans” architecture by grafting a trainable memory module onto a frozen open-weight model, Qwen-2.5-0.5B, using consumer-grade hardware. This new architecture, called “Grafted Titans,” appends memory embeddings to the input layer through a trainable cross-attention gating mechanism, allowing the memory to update while the base model remains static. In tests using the BABILong benchmark, the Grafted Titans model achieved 44.7% accuracy, outperforming the vanilla Qwen model’s 34.0% accuracy by acting as a denoising filter. However, the model faces limitations such as signal dilution and susceptibility to input poisoning, and further research is needed to address these issues. This matters because it explores innovative ways to enhance neural network performance without extensive computational resources, potentially democratizing access to advanced AI capabilities.

The concept of “Grafted Titans” introduces an innovative approach to enhancing the capabilities of pre-trained language models by integrating a trainable memory module. This method allows a static model, such as the Qwen-2.5-0.5B, to dynamically update its memory without altering its base weights. By appending memory embeddings through a cross-attention gating mechanism, this architecture aims to improve information retrieval and processing efficiency. This is particularly significant in scenarios where computational resources are limited, as it circumvents the need for extensive retraining of large models, making advanced AI capabilities more accessible to a broader audience.

The results from the BABILong benchmark highlight the potential of this approach. The Grafted Titans architecture achieved a notable 44.7% accuracy using only the memory state, outperforming the vanilla Qwen model, which required the full context and scored 34.0%. This suggests that the neural memory module effectively acts as a denoising filter, refining the attention mechanism of the model. By compressing the input signal into specific vectors, the memory module enhances the model’s ability to retrieve relevant information, demonstrating an improvement over traditional attention mechanisms that can become overwhelmed by noise in large contexts.

However, the approach is not without its challenges. One significant limitation is the potential for signal dilution, as the memory is injected at the initial layer, which could lead to a vanishing gradient effect as the signal propagates through the layers. This issue suggests that future iterations could benefit from multi-layer injection to maintain signal integrity throughout the model. Additionally, the current setup’s susceptibility to input poisoning highlights the need for robust guardrails, particularly in multi-turn conversations where the model might be exposed to misleading or harmful information.

Overall, the Grafted Titans architecture presents a promising direction for enhancing open-weight language models with minimal computational overhead. By addressing the identified limitations, such as signal dilution and input gullibility, this approach could significantly improve the stability and reliability of AI systems in complex conversational settings. As the project moves towards open-sourcing, it invites further experimentation and collaboration, potentially leading to breakthroughs in memory retrieval and model adaptability. This matters because it represents a step towards more efficient and effective AI systems that can operate within resource constraints, broadening the scope of AI applications and accessibility.

Read the original article here