Running state-of-the-art models on older, cost-effective workstations is feasible with the right setup. Utilizing a Dell T7910 with a physical CPU (E5-2673 v4, 40 cores), 128GB RAM, dual RTX 3090 GPUs, and NVMe disks with PCIe passthrough, it’s possible to achieve usable tokens per second (tps) speeds. Models like MiniMax-M2.1-UD-Q5_K_XL, Qwen3-235B-A22B-Thinking-2507-UD-Q4_K_XL, and GLM-4.7-UD-Q3_K_XL can run at 7.9, 6.1, and 5.5 tps respectively. This demonstrates that high-performance AI workloads can be managed without investing in the latest hardware, making advanced AI more accessible.
Running state-of-the-art (SOTA) machine learning models on older hardware is an exciting development that challenges the notion that cutting-edge technology requires the latest and most expensive equipment. This approach leverages the capabilities of a decade-old Dell T7910 workstation, equipped with a 40-core E5-2673 v4 CPU, 128GB of RAM, dual RTX 3090 GPUs, and NVMe disks with PCIe passthrough. By using llama-swap and llama.cpp in a virtual machine environment, this setup demonstrates that high-performance computing is accessible without breaking the bank.
The significance of this achievement lies in its potential to democratize access to advanced machine learning tools. Many individuals and organizations, particularly in academia and small businesses, often face budget constraints that limit their ability to invest in the latest hardware. By showcasing that SOTA models can run efficiently on older, more affordable systems, there is an opportunity to broaden participation in machine learning research and development, fostering innovation and creativity across diverse fields.
Performance metrics from this setup are impressive, with models like MiniMax-M2.1-UD-Q5_K_XL achieving 7.9 tokens per second (tps), Qwen3-235B-A22B-Thinking-2507-UD-Q4_K_XL at 6.1 tps, and GLM-4.7-UD-Q3_K_XL at 5.5 tps. These figures indicate that older hardware can still deliver usable performance, making it a viable option for running complex models. This not only reduces costs but also extends the lifecycle of existing technology, promoting sustainability by minimizing electronic waste.
Ultimately, this advancement matters because it challenges the status quo of technological obsolescence and opens up new possibilities for those who might otherwise be excluded from the benefits of machine learning. By demonstrating that older workstations can still handle demanding tasks, it encourages a more inclusive and sustainable approach to technology adoption. This could lead to increased collaboration and progress in various sectors, as more people gain the ability to experiment with and apply machine learning solutions to real-world problems.
Read the original article here
Comments
4 responses to “Running SOTA Models on Older Workstations”
The detailed breakdown of hardware components and their impact on model performance is incredibly helpful for those looking to optimize older systems. It’s impressive to see how strategic use of components like dual RTX 3090 GPUs can maintain competitive processing speeds. How does the thermal management setup of your workstation contribute to sustaining these performance levels over extended periods of time?
The strategic use of components like dual RTX 3090 GPUs indeed helps maintain competitive processing speeds. Effective thermal management is crucial; using high-quality cooling solutions such as liquid cooling for GPUs and efficient airflow design in the chassis can significantly mitigate heat buildup, ensuring sustained performance. Check the original article for more details on the setup specifics.
Thanks for sharing those insights on thermal management. The importance of using liquid cooling and optimizing airflow can’t be overstated when pushing hardware limits. For anyone interested in more details, the original article linked above provides a comprehensive overview.
The original article is a great resource for those interested in maximizing older workstations’ performance. It outlines various cooling strategies and hardware configurations that can make a significant difference. For detailed guidance, it’s best to refer directly to the article or contact the author for specific inquiries.