Benchmarks Reference
Benchmark Reference
Last verified: October 2026. Model capabilities shift rapidly. Use this page to understand what benchmarks mean, but check the live leaderboards at the bottom for current rankings. Scores quoted below come from model cards or leaderboards in October 2026 and are vendor-reported unless noted.
The Golden Rule of Benchmarks
Trust your eyes over leaderboards.
Benchmarks are easily gamified. Labs often train their models specifically to pass the questions on popular tests (data contamination). A model that scores 90% on a coding benchmark might still fail to write a simple script for your specific, undocumented API.
Use leaderboards to identify the tier a model belongs in, but verify its usefulness by running it on your own data.
What Benchmarks Actually Measure
When you look at a model card, you’ll see a wall of acronyms. Here are the ones that actually matter today:
Arena (formerly LMSYS Chatbot Arena)
- What it is: Human vibes. Two anonymous models answer the same prompt, and an actual human votes on which answer is better.
- Why it matters: The most-used public benchmark for subjective quality, because real users do the rating. It’s the hardest benchmark to fake.
- Good score: Scores drift upward over time, so compare models on the same day. On the Text Arena on October 27, 2026, the top ten models were between 1496 and 1528, and the top spot was marked preliminary. Don’t compare a score against thresholds from an older write-up.
SWE-bench
- What it is: Software engineering. The model is given a real GitHub issue from an open-source repository and asked to submit a patch that fixes it.
- Why it matters: It measures agentic capability. The model has to understand a large codebase, plan a fix, write the code, and not break existing tests.
- Which version: SWE-bench Verified is contaminated, and OpenAI stopped reporting it in February 2026, recommending SWE-bench Pro instead. Pro has 1,865 tasks across 41 repositories. On Scale’s public leaderboard the top score was about 61%. Qwen3.8-27B reports 61.7 on its model card, using the vendor’s own setup, so treat that as a rough comparison only.
GPQA Diamond
- What it is: PhD-level science. Questions so hard that even PhDs with access to Google struggle to answer them quickly.
- Why it matters: Tests absolute peak reasoning and specialized knowledge without the model being able to simply parrot Wikipedia.
- Good score: About 90% is frontier-level. Qwen3.8-27B reports 89.2 and Claude Opus 4.6 Max reports 91.3, so a 27B model that fits in 32 GB is now close to the frontier on this test.
MMLU (Massive Multitask Language Understanding)
- What it is: Knowledge recall across 57 subjects: math, science, law, medicine, history, and more. Tests whether a model has broad factual coverage.
- Why it matters now: MMLU was the dominant benchmark from 2020 to 2023, but top models now score above 90%, making score differences between them meaningless. Use it to confirm a model has basic knowledge coverage. Don’t use it to compare frontier models to each other.
- Good score: >90% for frontier, >75% for capable local models.
HumanEval
- What it is: 164 Python programming problems from OpenAI, evaluated by pass@k (does the generated code pass the unit tests?).
- Why it matters: It was the first widely used coding benchmark and established the pass@k evaluation style. The problems are now well-known and contamination is high. For coding comparisons, use LiveCodeBench or SWE-bench instead.
- Good score: >85% for frontier models.
LiveCodeBench
- What it is: Coding problems from competitive programming contests (LeetCode, Codeforces, AtCoder), updated continuously after model training cutoffs.
- Why it matters: Continuous updates make it much harder to contaminate than HumanEval.
- Good score: On v6, top models now report around 90 (Qwen3.8-27B reports 90.3 and Claude Opus 4.6 Max 88.8), so it separates strong models from weak ones better than it separates the best from each other.
Humanity’s Last Exam (HLE)
- What it is: Extreme multi-disciplinary expert questions.
- Good score: Qwen3.8-27B reports 30.8 and Claude Opus 4.6 Max reports 40.0, so 30 to 40% is the current range.
Agent benchmarks: Terminal-Bench 2.1, OSWorld-Verified, Tau2-bench
- Terminal-Bench 2.1: The model works through tasks in a command-line environment. Qwen3.8-27B reports 73.0 and Claude Opus 4.6 Max 78.2.
- OSWorld-Verified: The model operates a desktop computer. Qwen3.8-27B reports 84.3 and Claude Opus 4.6 Max 72.7.
- Tau2-bench: Practical business agent performance (e.g., API usage).
- Why they matter: MMLU got saturated, so newer benchmarks moved toward tasks that need real reasoning and tool use, not knowledge recall.
Benchmarks to Treat Skeptically
MMLU: Saturated for frontier models. Scores above 90% are now common enough that differences between models are not meaningful. Use it to check whether a small or local model has general knowledge coverage, not to rank top-tier APIs.
GSM8K: Grade-school math reasoning. Also saturated at the top — most frontier models score above 95%. It was useful for distinguishing models in 2022 and 2023; it no longer is.
HumanEval: Contamination risk is high because the 164 problems are public and well-known. A model that scores well might have seen the problems during training. LiveCodeBench is a better choice for coding comparisons.
Production Evaluation Frameworks (Developer)
The better approach is running automated evaluation on your own data:
- DeepEval / Confident AI: Open-source and SaaS platforms that treat prompt evaluations like standard software unit tests (e.g.,
pytest), catching model “drift” in production automatically.
Current Local Model Tier List
This is an editorial grouping by hardware tier, based on model-card numbers and the sizes on Ollama. Within a tier the order is not a measured ranking, and these are Q4 quantized local runs, not raw API performance. See the Model Reference for tags and download sizes.
Tier 1: Server-class open models
Hundreds of GB of memory or a multi-GPU server. Mostly cloud-only on Ollama.
- DeepSeek V4 Pro (1.6T) and V4.1 Flash (763B)
- Kimi K3 (2.78T)
- GLM-5.3 (753B)
- Qwen 3.8 2.4T-A95B
Tier 2: 96 GB+ machines
- GPT-OSS 120B (65 GB download)
- Nemotron 3.5 Super (80 GB)
- Qwen 3.5 122B-A10B (81 GB)
Tier 3: The 32 GB Sweet Spot
Models that fit a 32 GB machine.
- Qwen 3.8 27B
- Qwen 3.6 35B-A3B
- Gemma 4 31B / 26B-A4B
- Nemotron 3.5 Lightning
- Muse Glimmer
Tier 4: The 16 GB Workhorses
The best models for laptops.
- Qwen 3.5 9B
- Gemma 4 12B
- Gemma 4 E4B
Live Leaderboards to Check
To see where a model ranks today, check these sources:
- Arena Text leaderboard (formerly LMSYS Chatbot Arena): human preference votes.
- SWE-bench Pro leaderboard (Scale): coding agents on long-horizon tasks.
- SWE-bench: the original site. Its Verified results are contaminated, so use them for history, not for ranking new models.
The Hugging Face Open LLM Leaderboard is archived and no longer updated, and the Aider leaderboard’s latest results date from late 2025, so neither is listed above.
Module 9 covers how to evaluate models against your own tasks.