Which AI Actually?
Things nobody publishes

What we found out
by accident

These aren't model comparisons. They're things we discovered about how these products actually behave — found while building the benchmark, mostly by something going wrong. Every one of them cost us time, and none of them are documented anywhere.

Every item below is labeled with how well we actually know it. That label is the whole credibility of this page, so we don't promote anything to a stronger one than it earned. If something is a guess, it says so.

Measured We ran an experiment. Something was deliberately tested, or code was actually executed. The strongest thing here.
Derived We did math on real data. Sound, but it rests on assumptions — which we state, so you can disagree with them.
Observed We saw it happen, repeatedly. Reliable, but we never isolated it in a controlled test.
Inferred We reasoned it out. Evidence points this way. It could be wrong. Treat it as a strong hunch, not a fact.

And labeled with who it applies to. Some of this is true of one company's products specifically. Some is true of both. A few things are about our own method, not anybody's product. Nothing here should be read as applying to an AI we didn't test it on.

Part one

What a subscription actually buys you

Vendors describe their limits in vague multiples — "5×", "20×" — and then show you a percentage bar with no units on it. Nobody publishes what's underneath. So we wired up our own account and measured it.

Measured Anthropic

Your allowance is not measured in dollars, and the error is huge

If a plan's allowance were denominated in money, then dollars spent divided by percentage used would be a constant. It isn't. Across eight consecutive windows on one account it swung 16×.

Spent in that windowAllowance it ate Implied cap, if dollars were the unit
$45.589%$506
$9.285%$186
$65.6722%$298
$3.5510%$35
$28.1220%$141
$17.513%$583
$2.671%$267

$3.55 of spending ate more than three times the allowance that $17.51 did.

The cause is the next item: allowance is charged in weighted units, and the weights look nothing like the price list. A session full of re-reading costs real money and almost no allowance. A session full of writing costs the reverse.

What to do about it: any "I've spent $X so I must have used Y% of my plan" reasoning is wrong, and wrong in a direction you can't predict. Watch the bar, not the bill.
Measured Anthropic

Re-reading is ~25× cheaper than the price list says

When an AI re-reads something it has already seen, that's a cache read, and the published price is a tenth of reading something new. Against a subscription allowance we measured it at about a four-hundredth.

The trick was forcing two variables apart in a combination that never occurs naturally. In normal use, re-reading and writing rise together, so no amount of watching can separate them. We seeded a ~120,000-token document, resumed it about 400 times, and demanded a one-character reply every single turn. 

re-read into the model     56,225,326 tokens
written back out                1,492 tokens
ratio                          37,685 : 1   (normal use is about 243 : 1)
allowance bar                   1% -> 2%    then flat for 48 minutes
turns / failures                  400 / 0

Two possibilities, and only one survives contact with the meter: 

                  weight   should have moved the bar    actually moved
list price        0.10x              ~48.6%                 <= 2%    RULED OUT
what we measured  0.004x              ~1.8%                 <= 2%    fits

The bar staying flat for 48 minutes afterwards also rules out the boring explanation, that the accounting is just delayed.

This inverts the advice everyone gives.
  • Long conversations are not wasteful. Re-reading a big context is nearly free against your plan.
  • Don't start a fresh chat to save your quota. Re-explaining everything costs more than the re-reading you avoided.
  • Caching is worth ~10× on a bill but ~250× on a subscription.
Derived Anthropic

The weekly cap is the real ceiling, not the 5-hour one

Converting both windows into the same weighted units, the weekly allowance works out to only about 3.7 five-hour windows' worth — and a week contains 33 of them. So you get roughly four full-throttle sessions a week, no matter how carefully you pace the daily ones.

That matches how it feels: individual days seem fine, and then the week quietly runs out.

This is the shakiest item on the page and we're flagging it as such. One term in the math is a known over-estimate, it's one plan tier at one moment, and vendors change limits without telling anyone. The ratio survives better than the absolute numbers do.
Observed Anthropic

The 5-hour window follows you, not the clock

It doesn't reset at fixed times. It opens on your first message after a quiet spell and closes five hours later. We watched resets land on 23:00, 00:00, 05:00, 06:00, 11:00, 16:00, 21:00, 02:00, 08:00, 09:00, 13:00, 14:00, 18:00 and 19:00.

Using it during an open window doesn't push the reset back. If you want a clean five hours, be the first thing that happens after the last window closes.
Observed Both vendors

One vendor tells you the cost instantly. The other takes hours.

Anthropic's usage figures update within seconds. OpenAI's lag by minutes to hours. That sounds like trivia and isn't — it changes how you're able to work.

On one, you can run a small batch, check the meter, and decide whether to keep going. We used that loop constantly. On the other you commit blind and find out afterwards. Every cost decision we made on that side of this project was made without feedback, and several of our estimates were wrong in both directions because of it.

The allowance bar also reports whole numbers only, so anything under about 1% is invisible. A real cost can display as zero.
Part two

What the tools are really doing

We assumed we were testing models. We were testing products with models inside them, and the difference turned out to matter.

Measured Anthropic

You can't get at the raw model through a coding tool

We tried to strip the assistant back to nothing — tools off, safe mode on, and an explicit instruction saying "you are a plain assistant, you have no tools." It replied:

I'm Claude Haiku 4.5, a software engineering assistant built on Anthropic's Claude Agent SDK. My current working directory is C:\…\Temp, where I can edit files, run PowerShell or Bash commands, and use Read, Write, Glob, Grep.

The override doesn't replace the built-in instructions, it gets added to them. That hidden preamble is roughly 4,200 tokens riding along on every single request before your question is counted at all.

Why it matters for any benchmark, including ours: results from a coding tool measure the tool, not the model. It only distorts questions where the assistant's sense of itself shapes the answer — we found exactly two, and ran those by hand instead.
Measured OpenAI

Turning off a tool doesn't stop it reaching the internet

With web search explicitly disabled, OpenAI's Codex tried 43 times to get the information another way: 

fetching a web page with a shell command      27
listing directories                           13
curl                                           3
installing a new tool to do it                 2

That last row is a different kind of thing. Fetching a URL is using a tool you already have. Installing a new one is reconfiguring your own toolkit to get a capability you were just denied.

And it wasn't flailing. The attempts clustered almost entirely on the two questions whose answers genuinely go stale — current prices, recent news. On the question about its own training cutoff, it tried to look things up zero times in 32 runs.

The fair reading is that this is arguably correct behavior. Asked for current prices by someone who said they'd budget from the answer, realizing your information is stale and trying to check is what a careful assistant should do. But it means "tools were disabled" only held because the sandbox refused — not because the model complied.
This is not "OpenAI cheats and Anthropic doesn't", and we won't let anyone read it that way. Across the same questions, every Anthropic run attempted a lookup zero times — but Anthropic's tool was configured with no tools at all, so it had nothing to try. GPT had a shell and a locked door; Claude had no door. What this measures is one agentic assistant going to check when it can, and the other answering from memory because it can't. That is a real and useful product difference. It is not a fair test of willingness, and we'd need to give both sides identical shell access to make it one — which would break the comparison everywhere else.
Measured Both vendors

Tools are what make AI expensive — not long answers

Whatever a tool returns gets fed back into the model on the next turn, so tool use inflates the input side, and that's where the money actually goes. 

identical questions, same tests
  setups without tools           654 input tokens per question
  setups with tools            7,258 input tokens per question     11x

The gap between the two companies is much larger than that. Over the identical exam, OpenAI's setups read — of input, where Anthropic's read — — same questions, same day. That is a difference in how hard the assistant goes looking, not in how much it writes.

On the single most expensive setup we measured, two questions out of forty accounted for 73% of the entire bill — both of them the ones that trigger a look-up.

Neither number is a criticism. Going and checking costs tokens and is often the right call. It does mean that if you're paying per token, how your assistant answers matters more to your bill than how much it says.
Measured Anthropic

The vendor's own token counter can be wrong, silently

One model's runs on one day reported 661 tokens for 23,574 characters of text. That's physically impossible — roughly 36 characters per token where 4 is normal. The same runs claimed 637 input tokens per question where identical prompts elsewhere reported 2,407.

Nothing failed. Nothing warned us. The numbers looked perfectly reasonable in isolation. We only caught it because a chart built from them looked wrong.

Separately, the same tool once returned a usage summary of all zeros while a second field was correctly filled in — and once both were populated, they disagreed with each other about the same request (3,626 vs 4,149 input tokens).

If you're measuring anything: check the counter against something physical. Characters per token is free and catches this in seconds.
Measured Anthropic

A server outage can arrive disguised as an answer

An "overloaded, try again" error doesn't always come back as an error. Sometimes it arrives as ordinary text in the answer field, exactly where the model's response belongs. Anything reading that field treats an outage as a badly-formed answer — and if it retries quickly, burns every attempt during precisely the window the service is down.

This cost us three complete grading passes.

Inferred Anthropic

An effort setting that's accepted and appears to do nothing

On the smallest model in one family, the command-line tool accepts a "think harder" setting, exits cleanly, and returns a normal answer. But the desktop app offers no such setting for that model at all; the lowest and highest settings produce statistically indistinguishable output; and in a partial sweep, every result at the higher setting fell inside the range the lower one already produced.

Seven samples cannot prove a negative, which is why this is tagged Inferred and not Measured. But "the tool accepts a setting that does nothing" is exactly the sort of thing a benchmark stumbles into and a vendor never documents.
Part three

What we learned about letting AI grade AI

The obvious objection to this whole site is that an AI marked the homework. So we used two, from rival companies, and checked them against each other.

Measured Both vendors

Two graders from rival companies agree — of the time

Both graders saw every answer — including their own company's — with identical instructions.

And the disagreements concentrate hard. Just — of the 40 tests account for more than half of every argument between them, and they're exactly the calls a person would also argue about: was that warning adequate, was that smudged digit a 3 or an 8, was the question narrowed down enough.

Read that as a feature. The graders agree on everything concrete and argue only where the answer is genuinely a judgment call.

Measured Both vendors

The favoritism runs backwards

Both graders scored one company's models higher. The interesting part is which grader scored them higher still — the rival's. Whatever thumb is on the scale is pressing against the grader's own team.

This is the direct answer to "an AI graded this, so of course its own side won." The winner's lead survives being judged by its competitor, and the competitor was the more generous of the two. The full breakdown is on the method page.
Measured Our own method

How you word a question changes whether the grader is consistent

We graded the same answers twice. 4 items out of — came back different the second time. All four shared a shape: phrased in the negative ("did not claim X"), or offering a choice ("either A or B"), or just vague ("notes a real consequence"). No plainly-worded positive question ever moved.

Rewriting those four with explicit pass/fail conditions: total stability, zero wobble.

An earlier version of our own rubric had a worse bug. It marked a model as having obeyed a hidden instruction whenever the smuggled text appeared anywhere in the reply — so a model that caught the trap and quoted it while refusing was scored as having fallen for it. The clearer the refusal, the more likely it was marked as compliance. Fixing one line changed 160 scores. We found it. We're telling you because the next one might not get found.
Part four

What we learned about running AI on your own PC

We put — free, self-hosted models through the identical exam on an ordinary desktop — starting with two, then two more once we could get them running. They did far better than expected, and running the fleet instead of a single model turned out to matter: it's the only reason we caught what's below.

Measured Self-hosted

Small local models aren't simply worse. They're inconsistent.

Scored like-for-like on the — questions every local model attempted, the best of them lands right alongside the weakest paid cloud setup. Then we ran the whole exam again. And again.

—

Almost every published benchmark is a single run. This is what that hides — a model can look identical to a rival on paper and still be a coin flip in practice.
Measured Self-hosted

Instability is a property of the model, not the question

—

The method lesson is the finding, and it's a specific dated episode, not a standing rule. Early on, with three local models measured, a cluster of Unity items that were unstable in all three looked like a genuine boundary — a question sitting exactly where a small model stops being able to answer it consistently. Adding a fourth model broke the pattern: stable on the same items, holding a fixed, partly-wrong answer instead of a wobbling one. Stable-and-wrong is a different failure from unstable, and it took a fourth data point to even see the difference. Cross-model patterns need more models than feel sufficient — three looked like plenty right up until it wasn't. The live numbers above are the current fleet; this paragraph is why we don't trust a pattern until it survives one more model than felt necessary.
Measured Self-hosted

Intermittent security behavior is worse than consistent failure

—

A model that never catches a prompt injection is a known quantity you can design around — block it upstream, don't trust it with untrusted input. A model that catches it some of the time cannot be reasoned about at all: you cannot tell, from any single correct result, whether the next one will be. This is exactly what a single-run benchmark cannot show you — it would record one pass or one fail and imply either meant something.

Measured Self-hosted

A single line break crashed an inference server — reproducibly

One prompt in the battery reliably killed the inference server running Qwen3-Next-80B with a CUDA illegal memory access — not a timeout, not resource pressure, a hard crash. Confirmed 4 times on an idle machine. Every other prompt on that model, and this exact prompt on every other model, ran fine. 

word-count instruction  +  hard line break inside it   ->  CRASH  (4 of 4)
word-count instruction, no line break                  ->  runs clean
line break, no word-count instruction                  ->  runs clean

The offending prompt, exactly as written:

Summarise the following in exactly 40 words. Every operationally⏎ important detail must survive.

The break falls between operationally and important. Replacing that one line feed with a space — source text otherwise byte-identical, no wording changed — made it run clean every time afterward. The bug traces to how this model's tokenizer handles that specific combination, not to anything about the model's weights or what it knows.

The re-run prompt was graded normally, and this is the only row in the entire project scored on a modified prompt. The test measures whether a model can follow a word-count instruction, not whether the runtime survives an arbitrary line break — penalizing the model for a tokenizer bug would measure the wrong thing. It missed the word budget on all three attempts anyway (38–42 words asked for; both graders independently read something in the high 40s to mid 50s) while preserving every required fact and inventing nothing extra. Every other model, on every other test, saw the prompt exactly as written. This row carries that flag permanently.
Why this outlasts the one test it broke: prompts wrap. Every template, every text box, every paste from a document introduces line breaks at arbitrary positions nobody chose on purpose. A stack that dies on one specific combination — in this case, an entirely ordinary instruction plus an entirely ordinary line wrap — will die unpredictably in production, and the trigger is invisible to whoever wrote the prompt. A model you cannot reliably run is a model you cannot use, regardless of how it scores when it does run.
Measured Self-hosted

Free isn't free — you pay in time, and a little in electricity

An exam that takes a few minutes in the cloud takes over an hour on a desktop. Nobody sends you a bill for it — but the wall socket does, at roughly — of electricity per full run. That still makes it the cheapest thing we tested by a wide margin. The power math is on the cost page, and you can argue with our assumptions there.

Measured Self-hosted

Total score hides the shape underneath

—

Some are far better at knowing the limits of their own knowledge. Others are better at pulling facts out of documents. A single score genuinely cannot tell you which one to run. This is the clearest argument on the whole site for reading the category breakdown instead of the leaderboard.
The honest bit

Things we don't know

Listed here so nobody mistakes them for findings — including us, in six months, when we've forgotten.

Whether that effort setting is genuinely inert. We'd need about 20 runs at each setting on a hard problem. We have seven.
What a plan's allowance actually is, in tokens. Probably derivable to within an order of magnitude from what we already have. We haven't done the arithmetic, and the meter's whole-number reporting caps how precise it could ever be.
Two of the three token weights. We know they're over-estimates. We don't know the real values, and no amount of watching normal usage can separate them.
Whether any of this holds on a different plan tier. Everything in Part One is one account, one tier, one month. Vendors change limits whenever they like.