language modeling
-
End-to-End Test-Time Training for Long Context
Read Full Article: End-to-End Test-Time Training for Long Context![[R] End-to-End Test-Time Training for Long Context](https://www.tweakedgeek.com/wp-content/uploads/2025/12/featured-article-7245-300x171.png)
Long-context language modeling is approached as a continual learning problem, utilizing a standard Transformer architecture with sliding-window attention. The model continues to learn during test time by predicting the next token based on the given context, effectively compressing the context into its weights. By employing meta-learning during training, the model's initialization is enhanced for learning at test time. This End-to-End Test-Time Training (TTT-E2E) method demonstrates scalability similar to full attention Transformers while maintaining constant inference latency, offering a significant speed advantage. This development is crucial as it provides a more efficient approach to handling long-context language tasks, improving both performance and speed.
-
Nested Learning: A New ML Paradigm
Read Full Article: Nested Learning: A New ML Paradigm
Nested Learning is a new machine learning paradigm designed to address the challenges of continual learning, where current models struggle with retaining old knowledge while acquiring new skills. Unlike traditional approaches that treat model architecture and optimization algorithms as separate entities, Nested Learning integrates them into a unified system of interconnected, multi-level learning problems. This approach allows for simultaneous optimization and deeper computational depth, helping to mitigate issues like catastrophic forgetting. The concept is validated through a self-modifying architecture named "Hope," which shows improved performance in language modeling and long-context memory management compared to existing models. This matters because it offers a potential pathway to more advanced and adaptable AI systems, akin to human neuroplasticity.