Which open model fits, and how good is it?
All models are wrong. Some fit in your memory. We measure which ones are the most useful.
Why not just use cloud benchmarks?
-
Most tasks don’t need a frontier model.
Summarising, translating, extracting and classifying are everyday work. Cloud leaderboards chase the top of the curve; we ask a more practical question: which smaller, cheaper model is good enough for this task?
-
Quantisation matters.
The same model at Q4 and at Q8 is not the same model. Cloud benchmarks test one hosted configuration and hide the trade-off. Here every quant is its own run, so you can see what a smaller file actually costs you in quality.
-
Fit your hardware and your budget.
The best model is the one that runs on the machine you have. Comparing quants and sizes side by side lets you pick for your memory, your latency target or your monthly spend, not for a leaderboard crown.
-
Niche models get measured too.
Fine-tunes, merges, abliterated and uncensored variants rarely show up in cloud benchmarks. We measure them next to the mainstream models, so you can see what, if anything, those changes cost or buy.
Available benchmarks
-
Round-trip translation
Translates text into another language and back, then scores how close the result is to the original using embedding distance. Higher is better.
FAQ
Why does the quantisation show up in the results?
Quantisation trades quality for size and speed. Seeing the score for each quant of the same model tells you whether the smaller one is good enough for your task.
Will you cover abliterated or uncensored models?
Yes, that is part of the point. They are widely used and rarely measured. Results are reported like any other run, without judgement on whether you should use them.
Are these scores comparable to cloud scores?
No. Different tasks, different settings, different scoring. Compare models within a benchmark and version, not across leaderboards.
Does a high score mean it is the best model?
Only for what that benchmark measures. Each benchmark page says what it is good for, and what it is not good for.