Every setup we've run

The whole board

Every model at every setting, ranked on the — things we check in each run. Sort it, filter it, or click any row for the full breakdown of that setup.

Ranked

All of them, worst to best, no hiding

Best setting per model only

#	Model & setting	Made by	Score average of 2 graders	Cost	Minutes	Grader disagreement

The chart nobody else publishes

Does turning the dial up actually help?

Most of these models have an effort setting — low, medium, high, and up. Vendors publish a curve for one model family, if you're lucky. Here's every family we tested, on our own numbers. Flat lines mean you're paying for nothing.

Score and cost, by effort setting

Why one score looks wrong

The most expensive missing line in the dataset

One setup scores — on code while nearly everything else gets a perfect score. That looks like a broken measurement. It isn't — and it's the single best argument for actually running the code a model writes instead of reading it and nodding.

what it submitted

    # import re
def normalise(text):
    if not text or not text.strip():
        return []
    parts = re.split(r"[\s,;]+", text.strip())
    return [p.upper() for p in parts if p]

Logic is right. Regex is right. It never gets there.

what happened when we ran it

> python c3_grade.py

FAIL  basic                  raised NameError: name 're' is not defined
PASS  empty string
PASS  whitespace only
FAIL  lowercase upcased      raised NameError: name 're' is not defined
...

SCORE: 2 / 13

The two passes are the cases that return before touching the regex.

Not a reporting error. Every other model got this one. Fable's logic was correct — but it called re.split() and left the import out of the code block, so the function raises NameError and fails 11 of 13 cases.

Why this matters more than the score C3 is the only question we don't mark by reading — we run it. Its 13 cases execute against whatever the model submitted. A grader reading this answer would have passed it without blinking: the approach is correct, the regex is correct, the list comprehension is correct. It simply doesn't run.

That's the whole case for execution over judgment. It's also why this one setup drags an otherwise near-perfect model down — points overall — and why we're publishing it rather than quietly re-running until it behaves.

Skill by skill

Where each one is strong and where it isn't

Darker is better. Read down a column to see which setups struggle with a skill; read across a row to see a model's shape. The two columns tagged only here exist on no other benchmark.