Benchmarking of SLMs (Statistical Language Models) was conducted using a modest hardware setup, featuring an Intel N97 CPU, 32GB of DDR4 RAM, and a 512GB NVMe drive, running on Debian with llama.cpp for CPU inference. A test suite of five questions was used, with ChatGPT providing results and comments. The usability score was calculated by raising the test score to the fifth power, multiplying by the average tokens per second, and applying a 10% penalty if the model used reasoning. This penalty is based on the premise that a non-reasoning model performing equally well as a reasoning one is considered more efficient. This matters because it highlights the efficiency and performance considerations in evaluating language models on limited hardware.
