The recent update to llama.cpp introduces experimental native mxfp4 support for Blackwell, resulting in a 25% preprocessing speedup compared to the previous version. While this update is currently 10% slower than the master version, it shows significant promise, especially for gpt-oss models. To utilize this feature, compiling with the flag -DCMAKE_CUDA_ARCHITECTURES="120f" is necessary. Although there are some concerns about potential correctness issues due to the quantization of activation to mxfp4 instead of q8, initial tests indicate no noticeable quality degradation in models like gpt-oss-120b. This matters because it enhances processing efficiency, potentially leading to faster and more efficient AI model training and deployment.
Read Full Article: Llama.cpp: Native mxfp4 Support Boosts Speed