vLLM
-
Benchmarking 4-bit Quantization in vLLM
Read Full Article: Benchmarking 4-bit Quantization in vLLM
A comprehensive analysis of vLLM quantization methods reveals varied performance across different techniques. Marlin achieved the highest token processing speed at 712 tokens per second, significantly outperforming the baseline FP16's 461 tok/s, while GPTQ without Marlin's kernel lagged behind at 276 tok/s. BitsandBytes maintained the smallest quality drop and required no pre-quantized weights, whereas GGUF had the worst perplexity but excelled in HumanEval scores. AWQ showed unexpectedly slow performance in vLLM, processing only 67 tok/s. Understanding these differences is crucial for optimizing model efficiency and performance in machine learning applications.
-
7900 XTX + ROCm: Llama.cpp vs vLLM Benchmarks
Read Full Article: 7900 XTX + ROCm: Llama.cpp vs vLLM Benchmarks
After a year of using the 7900 XTX with ROCm, improvements have been noted, though the experience remains less seamless compared to NVIDIA cards. A comparison of llama.cpp and vLLM benchmarks on this hardware, connected via Thunderbolt 3, reveals varying performance with different models, all fitting within VRAM to mitigate bandwidth limitations. Llama.cpp shows a range of generation speeds from 22.95 t/s to 87.09 t/s, while vLLM demonstrates speeds from 14.99 t/s to 94.19 t/s, highlighting the ongoing challenges and progress in running newer models on AMD hardware. This matters as it provides insight into the current capabilities and limitations of AMD GPUs for local machine learning tasks.
-
Run MiniMax-M2.1 Locally with Claude Code & vLLM
Read Full Article: Run MiniMax-M2.1 Locally with Claude Code & vLLM
Running the MiniMax-M2.1 model locally using Claude Code and vLLM involves setting up a robust hardware environment, including dual NVIDIA RTX Pro 6000 GPUs and an AMD Ryzen 9 7950X3D processor. The process requires installing vLLM nightly on Ubuntu 24.04 and downloading the AWQ-quantized MiniMax-M2.1 model from Hugging Face. Once the server is set up with Anthropic-compatible endpoints, Claude Code can be configured to interact with the local model using a settings.json file. This setup allows for efficient local execution of AI models, reducing reliance on external cloud services and enhancing data privacy.