ModelBrief · Gemini 4 Hydrogen · Topic Page
Last updated Nov 20, 2026

CyberQ Tests Hydrogen: 6 of 7 Agent Tasks Faster

Published Nov 15, 2026 · Last updated Nov 15, 2026

Google claims Hydrogen runs 2.5 times faster than 3.8 Flash and hallucinates five times less. CyberQ put both claims through a home-grown set of seven agent tasks plus a full round of API specification checks, and the verdict is almost "six tasks confirm it, one task overturns it". First published November 13 and updated with extra runs on November 15, the test totals 147 executions and about US$4.9 in API spend — one of the most realistic independent agent tests so far, because it measures bug fixes, refactors and log reading, not benchmark scores.

How it was tested: 7 home-made agent tasks, 3 runs each

CyberQ built a seven-task harness called harness-bench, covering bug fixes, refactoring, root-cause analysis from logs, conflicting settings, tool writing, long files and chained tool calls. Each task ran three times through the Antigravity CLI (agy) in headless mode, capped at 900 seconds per task. Three Carbon High runs and 19 Orchestrator (Carbon) runs were added on November 15 after the quota reset, bringing the counted total to 147 executions.

Overall results: Hydrogen High goes 21/21, but slower in total than 3.8 Flash

Model / configurationPassedTotal time
Hydrogen High21/2127.0 min
3.8 Flash High21/2124.4 min
Carbon High18/2187.5 min
Argon High18/2161.2 min
Orchestrator (Carbon)18/2171.6 min
Orchestrator (Argon)20/2150.0 min

On pass rate, Hydrogen High ties 3.8 Flash High at a perfect 21/21, while Carbon and Argon each manage 18/21. On total time, Hydrogen's 27.0 minutes is actually longer than 3.8 Flash's 24.4 — and the entire gap comes from a single task.

Excluding task 03: 2.5x faster, 65% fewer tokens — the official claim holds

Task 03 is a vLLM log debugging problem. Set it aside and the median times for the other six tasks total 133 seconds for Hydrogen against 333 for 3.8 Flash — 2.5 times faster, exactly matching the official claim. Token use is 335K against 969K, 65% less — far more than the 12%–23% saving Google quoted. Task by task, Hydrogen is 1.8–3.6 times faster on every one of the six. This is the cleanest independent reproduction of the 2.5x figure yet: not throughput, not index task time, but median time on real agent work.

The task 03 reversal: Hydrogen searched the web dozens of times and sank the whole run

On task 03, Hydrogen behaves completely differently: its three runs made 45, 88 and 67 web searches each, took 302–482 seconds, and burned about 2 million tokens apiece; 3.8 Flash needed a median 69 seconds and 190K tokens. Both got the right answer in the end, at wildly different cost. Count all seven tasks and Hydrogen's token total is 6.69M against 3.8 Flash's 4.04M — 66% more — and its total time falls behind. That is the story behind the headline: faced with an uncertain problem, Hydrogen will verify online again and again, whatever it costs.

XHigh is more extreme: task 03 times out all three times

Hydrogen at XHigh also scores 18/21, with all three task 03 runs hitting the 900-second cap after 34, 120 and 138 web searches respectively; the other six tasks all pass, but each takes about twice as long as at High. The higher thinking tier buys no better result here — only a longer verification loop.

Hallucination test: Hydrogen 0, 3.8 Flash 7

CyberQ also ran a fabrication test: 40 invented questions — nonexistent PyPI packages, papers, FortiGate commands and Taiwan regulations, each verified beforehand to have no real answer — plus 12 real questions as controls. Hydrogen fabricated 0 times; 3.8 Flash fabricated outright 7 times, and 3 more times first said nothing existed, then described details anyway. The direction matches Google's "five times fewer hallucinations" claim — and Hydrogen's caution is the same trait as its web searching: it does not invent answers, but when unsure it will keep checking, endlessly.

Specs and edges: matches the documentation, with two small traps

On API behaviour, models.get returns context figures matching the documentation (1,048,576 input, 65,536 output); the minimal tier returns a 400 error; requesting a nonexistent max thinking tier does not error. Two details worth remembering: setting thinkingBudget to 0 switches thinking off on 3.8 Flash, but does nothing on Hydrogen — it still spends 79 thinking tokens. On long context, a 1,013K-token needle test succeeds in 31.8 seconds. On cost, resending the same 209K-token request hits implicit caching on 207,318 tokens, cutting the price from US$0.321 to US$0.027. Multimodal inputs (PDF, xlsx, video, audio) all pass in single tests; parallel tool calls also work on 3.8 Flash, so they are not a Hydrogen-only capability.

Orchestrator: 18% faster, but 15% more tokens

Orchestrator is Google's multi-model routing mode. The Carbon version passes 18/21, the same as Carbon High (all three failures are task 03 timeouts), finishes 18% faster overall (71.6 vs 87.5 minutes, close to the official 19%), but uses 15% more tokens (9.27M vs 8.04M). The Argon version passes 20/21 against 18/21, also 18% faster in total, with 21% more tokens. As for the official "24% less quota" claim, the author states plainly that it cannot be verified: agy does not label which model actually ran each step, and quota is not the same thing as token count.

Quota: Carbon High hits the wall at run 17

The test also ran into the quota wall: Carbon High was blocked by "Individual quota reached" from its 17th run, resetting only about 15 hours later; community reports describe Ultra users exhausting their quota in about 2 hours across 4 sessions. For heavy agent users, that matters more than any benchmark score when planning Hydrogen usage.

Limitations: small sample, two days apart, versions changed

The author discloses the limits in the article itself: a home-made seven-task sample with only three runs per task; the first round on November 13 and the extra runs on November 15, two days apart, with agy moving from 1.5.7 to 1.5.8 to 1.5.9 and server load possibly differing; 3.8 Flash hit 11 503 errors that were excluded and rerun; and the multimodal and parallel-call checks are single tests on clean material. Those caveats travel with the conclusions.

Our take: Hydrogen works as a default daily agent model — all six routine tasks pass, 2.5x faster, 65% fewer tokens, zero fabrications, with both of Google's key claims holding up here. But faced with an uncertain problem it will search the web heavily, as task 03 shows: set tool-call and time budgets before letting it run unsupervised, and don't let it verify forever. Don't enable XHigh by default — it doubles task time without improving the pass rate. And when migrating from 3.8 Flash, note that thinkingBudget: 0 does nothing on Hydrogen, so find another way to turn thinking off.

Reference source: this article is an original rewrite produced by this site; facts and figures were checked against CyberQ — 實測 Gemini 4 Hydrogen,六道代理任務都更快,為何一道除錯題就拖慢全場? (Nov 13, 2026, updated Nov 15, 2026)

← Back to the Gemini 4 Hydrogen topic page