OrcaRouter Reads the AA 60: Nine Tests, a 64.7-Second First Token and a 39.7% Error Rate

A benchmark score only means something inside its test mix, its thinking time and its index version. In a November 13 piece, OrcaRouter takes Carbon's Artificial Analysis index score of 60 apart exactly that way: what the nine evaluations actually test, what each one costs in time and money, and the two things the score does not tell you — the version trap and the real error rate.

What the nine evaluations test

The index is built from nine evaluations: AA-Briefcase v1.1, AutomationBench-AA, Terminal-Bench 4.0, FrontierCode-AA 2.0, AA-Omniscience v1.1, GDP.pdf, HLE-Diamond, CritPt v2 Hard Subset, and Terminal-Bench-Science 0.1. Three are terminal and automation work, two are document-grounded reasoning over long inputs, two are hard science and mathematics, and one is frontier code. Read that list before generalising: there is no web-agent evaluation, no multi-turn customer support simulation, and no cyber or vulnerability-detection harness — a strong score says nothing about those. Long-horizon coding on a specific repository is only approximated by FrontierCode-AA, not replaced by it.

The 60 and the version trap

Carbon posts 60, rank 4 of 78 on Intelligence Index v5.0.1; the leader is Claude Fable 5.2 at 65. The five-point gap works out to roughly one additional hard task solved per twenty attempted at Carbon's level of competence — the author flags that as a direction, not a rate. The version guard matters more than the rank: models scored on v4.3 or v4.3.2 are not on the same scale, and a chart that mixes them is wrong regardless of who published it. To compare Carbon with a model you evaluated yourself last quarter, either re-run your own harness on both, or restrict the claim to the v5.0.1 cohort and say so in the sentence where you quote it.

The time bill: 64.7 seconds to first token

Time to first token is 64.7 seconds, rank 69 of 77 models with latency data; total response time is 28.4 minutes per task across the nine evaluations. Once Carbon begins emitting, it runs at 72 tokens per second, which is ordinary — the wall clock is dominated by thinking, not by writing. That is a design choice rather than a defect: a harness that lets the model reason until it is satisfied will produce a slow but strong score. The same model on live gateway traffic looks entirely different: a p50 of 6.89 seconds to first token and a p95 of 10.00 seconds, because most real requests are not HLE-Diamond. The two figures measure different things and both belong in a capacity plan.

The money bill: $5.48 per index run

Running the index against Carbon cost $5.48 at $2.00 per million input tokens and $10.00 per million output, for a blended $2.70 per million. AA also reports 205 million output tokens for the run — the token count, not the 2.69 percent verbosity figure, is the number to carry forward: a nine-task agentic evaluation consumed the better part of a quarter billion output tokens. The same index against Claude Fable 5.2 cost $26.54 — 4.8 times as much — at $8.00 in and $40.00 out. Put as a ratio, Carbon pays $0.09 of benchmark spend per index point and Fable 5.2 pays $0.41. On time, Fable 5.2 is slower too: 46.8 minutes per task, 1.6 times Carbon. The author's advice is to route by difficulty: the 90 percent of calls that a 60-point model clears do not need a 65-point one.

What the score leaves out: a 39.7% error rate

There is no measurement here of what happens under load. The live error rate on OrcaRouter traffic for Carbon was 39.7 percent over the last seven days, against 10.8 percent for Fable 5.2. A benchmark harness retries and absorbs that; your application has to decide how — the author suggests logging time to first token, tokens out, and the non-2xx count per route, and if your error rate is above the gateway's 39.7 percent, the problem is on your side of the request.