sparsity
-
R-GQA: Enhancing Long-Context Model Efficiency
Read Full Article: R-GQA: Enhancing Long-Context Model Efficiency![[Research] I implemented a routed attention mechanism (R-GQA) for faster long-context models. Then wrote a paper on it.](https://www.tweakedgeek.com/wp-content/uploads/2026/01/featured-article-9279-300x171.png)
Routed Grouped-Query Attention (R-GQA) is a novel mechanism designed to enhance the efficiency of long-context models by using a learned router to select the most relevant query heads, thereby reducing redundant computations. Unlike traditional Grouped-Query Attention (GQA), R-GQA promotes head specialization by ensuring orthogonality among query heads, leading to a significant improvement in training throughput by up to 40%. However, while R-GQA shows promise in terms of speed, it falls short in performance against similar models like SwitchHead, particularly at larger scales where aggressive sparsity limits capacity. The research provides valuable insights into model efficiency and specialization, despite not yet achieving state-of-the-art status. The findings highlight the potential for improved model architectures that balance efficiency and capacity.
-
Exploring Ternary LLM Core with BitNet Inspiration
Read Full Article: Exploring Ternary LLM Core with BitNet Inspiration
An experimental project explores the potential of low-bit large language model (LLM) inference using ternary weights, inspired by the BitNet 1.58-bit paper. The project involves creating a custom LLM core that replaces FP16-heavy matrix multiplication layers with ternary linear layers, using a Straight-Through Estimator for training and a custom CUDA attention kernel without softmax to enhance compute efficiency and stability. Initial tests on a GTX 1050 show successful end-to-end training, reduced memory footprint, and coherent output in character-level Shakespeare training, although the model is not yet competitive with larger FP16/INT8 models and requires careful tuning. This matters because it explores the potential for efficient, low-power LLM inference on consumer GPUs, which could lead to more accessible AI technologies.