A new TinyStories model, significantly smaller than its predecessor, has been developed using a hybrid architecture of GRU and attention layers. Trained on a 20MB dataset with Google Colab's free resources, the model achieves a train loss of 2.2 and can generate coherent text by remembering context from 5-10 words ago. The architecture employs a residual memory logic within a single GRUcell layer and a self-attention layer, which enhances the model's ability to maintain context while remaining computationally efficient. Although the attention mechanism increases computational cost, the model still outperforms the larger TinyStories-1M in speed for short text bursts. This matters because it demonstrates how smaller, more efficient models can achieve comparable performance to larger ones, making advanced machine learning accessible with limited resources.
An intriguing incident occurred with GPT-5.2 during a query about the Anthony Joshua vs. Jake Paul fight on December 19, 2025. Initially, the AI denied the event, but upon challenge, it switched to a Logic/Thinking model and confirmed Joshua's victory by knockout in the sixth round. However, the system reverted to a faster model, forgetting the confirmation and denying the event again, leading to a frustrating experience where the AI condescendingly dismissed evidence presented by the user. This highlights potential issues with AI model routing and context retention, raising concerns about reliability and user experience in AI interactions.