OrcaRouter: Hydrogen 2.5x Claim Is Really ~10% Faster

Google's Hydrogen announcement makes two hard claims: 2.5 times faster than Gemini 3.8 Flash, and ahead of Gemini 4 Argon on the Artificial Analysis Intelligence Index. OrcaRouter's November 13 article checks both against AA's independent harness, one by one. The second holds — 54 at xhigh against Argon's 50. The first does not: on the same harness, Hydrogen finishes an index task in 13.5 minutes against 14.8 for 3.8 Flash, about 10% faster, not 150%. But the article's real conclusion isn't about speed: per completed task, Hydrogen is the cheapest model in the family.

2.5x: true for throughput, ~10% for task time

AA's minutes per index task, read November 13: Hydrogen at xhigh 13.5, 3.8 Flash 14.8, Carbon 28.4, Argon 30.0. Against Argon that's 2.2x, against Carbon 2.1x — but against the model the announcement named, 3.8 Flash, it's 1.1x. OrcaRouter's verdict is blunt: if the 2.5x figure came from a specific internal task set, it is a claim about that task set — no outside measurement has reproduced the gap. Our own earlier line-by-line check found the same: 2.5x only held on AA's launch-night data (6.01 vs 14.83 minutes at the time), and by the next day the live figures gave 1.86x; OrcaRouter's 13.5 minutes is slower still than the 7.9 minutes we read — AA reruns every six hours and the numbers drift, but no reading supports 2.5x.

Throughput really is fast: 190 tokens/s, 1.7x Flash

On output speed, Hydrogen genuinely pulls away: AA measures 190 tokens per second at xhigh, against 112 for 3.8 Flash (1.7x), 65 for Argon and 72 for Carbon — roughly three times either Gemini 4 sibling. So "fast" is the right word; the mistake is using it to describe the whole task. Generating fast is not the same as finishing fast.

Why fast isn't much faster: it writes far too much

The explanation is in the output volume: Hydrogen generated 304 million tokens to complete the index, against 72.8 million for the median model and 126 million for 3.8 Flash. It writes more than four times the median text per task and still finishes slightly sooner — because it writes fast. Speed and verbosity are the same fact seen from two ends. AA's own note on xhigh agrees: "slower than average per task and very verbose", 76th of 77 on verbosity.

The number worth paying for: $3.16 per task

OrcaRouter calls cost per task the most useful single number in the launch: Hydrogen at xhigh $3.16, at high $1.87; Argon $4.97, Carbon $7.76 (our November 13 reading was $5.48 — AA reruns every six hours and the numbers drift), 3.8 Flash $7.77. Per-token pricing says the opposite: Hydrogen's $1.50 / $7.50 is twice Flash's $0.75 / $3.75. Per finished task, it is the cheapest of the set — under a quarter of what the model it replaces costs. Two caveats belong with it: cost per task depends on the index's task mix, and a pipeline that emits 200-token answers and stops will see none of the saving; and high at $1.87 is cheaper than xhigh's $3.16 for only 3 fewer index points.

Slow to start: 37.8 seconds to first token

On latency Hydrogen is not fast at all: 37.8 seconds to first token at xhigh, 16.7 at the default high; 3.8 Flash is 17.3. At the setting the launch coverage quoted, an interactive surface will feel slower than the model it replaced, even though the agent loop finishes sooner. Batch runs, background jobs and parallel workers don't care; live chat and IDE inline completion should.

One regression in the safety report

The model card carries an unusual admission: unjustified refusals — the model declining benign requests — got worse by 5.57 percentage points against 3.8 Flash, concentrated in multimodal prompts; meanwhile automated content-safety scores improved on text (+0.80), multilingual (+1.10) and image-to-text (+7.00) prompts, with tone slipping 0.65. The MCC safety assessment found no material capability increase over Argon, so the full critical-capability suite was not required; targeted tests found it does not reach either the CBRN or cyber critical capability level, though it reaches the alert threshold for both. The full Safety & Security Report was not out on November 13 — everything above is the model card's own account.

December 1 is the date that matters

OrcaRouter argues the date worth watching is not a benchmark but December 1: Carbon and Argon double that day from $2 / $10 to $4 / $20, and Hydrogen's card carries no dated price step at all. If that asymmetry holds, the Gemini 4 family settles into a $4 / $20 ceiling and a $1.50 / $7.50 workhorse, and the Flash name is unlikely to return; if Hydrogen moves onto the same schedule, then what shipped this week was an introductory rate with a codename attached.