![[D] Reasoning over images and videos: modular pipelines vs end-to-end VLMs](https://www.tweakedgeek.com/wp-content/uploads/2026/01/featured-article-7835-300x171.png)
Exploring the best approach for reasoning over images and videos, the discussion contrasts modular pipelines with end-to-end Vision-Language Models (VLMs). While end-to-end VLMs show impressive capabilities, they often struggle with brittleness in complex tasks. A modular setup is proposed, where specialized vision models handle perception tasks like detection and tracking, and a Language Model (LLM) reasons over structured outputs. This approach aims to improve tasks such as event-based counting in traffic videos, tracking state changes, and grounding explanations to specific objects, while avoiding hallucinated references. The tradeoff between these methods is examined, questioning where modular pipelines excel and what reasoning tasks remain challenging for current video models. This matters because improving how machines interpret and reason over visual data can significantly enhance applications in areas like autonomous driving, surveillance, and multimedia analysis.
