Google claims Hydrogen runs 2.5 times faster than 3.8 Flash and hallucinates five times less. CyberQ put both claims through a home-grown set of seven agent tasks plus a full round of API specification checks, and the verdict is almost "six tasks confirm it, one task overturns it". First published November 13 and updated with extra runs on November 15, the test totals 147 executions and about US$4.9 in API spend — one of the most realistic independent agent tests so far, because it measures bug fixes, refactors and log reading, not benchmark scores.
CyberQ built a seven-task harness called harness-bench, covering bug fixes, refactoring, root-cause analysis from logs, conflicting settings, tool writing, long files and chained tool calls. Each task ran three times through the Antigravity CLI (agy) in headless mode, capped at 900 seconds per task. Three Carbon High runs and 19 Orchestrator (Carbon) runs were added on November 15 after the quota reset, bringing the counted total to 147 executions.
| Model / configuration | Passed | Total time |
|---|---|---|
| Hydrogen High | 21/21 | 27.0 min |
| 3.8 Flash High | 21/21 | 24.4 min |
| Carbon High | 18/21 | 87.5 min |
| Argon High | 18/21 | 61.2 min |
| Orchestrator (Carbon) | 18/21 | 71.6 min |
| Orchestrator (Argon) | 20/21 | 50.0 min |
On pass rate, Hydrogen High ties 3.8 Flash High at a perfect 21/21, while Carbon and Argon each manage 18/21. On total time, Hydrogen's 27.0 minutes is actually longer than 3.8 Flash's 24.4 — and the entire gap comes from a single task.
Task 03 is a vLLM log debugging problem. Set it aside and the median times for the other six tasks total 133 seconds for Hydrogen against 333 for 3.8 Flash — 2.5 times faster, exactly matching the official claim. Token use is 335K against 969K, 65% less — far more than the 12%–23% saving Google quoted. Task by task, Hydrogen is 1.8–3.6 times faster on every one of the six. This is the cleanest independent reproduction of the 2.5x figure yet: not throughput, not index task time, but median time on real agent work.
On task 03, Hydrogen behaves completely differently: its three runs made 45, 88 and 67 web searches each, took 302–482 seconds, and burned about 2 million tokens apiece; 3.8 Flash needed a median 69 seconds and 190K tokens. Both got the right answer in the end, at wildly different cost. Count all seven tasks and Hydrogen's token total is 6.69M against 3.8 Flash's 4.04M — 66% more — and its total time falls behind. That is the story behind the headline: faced with an uncertain problem, Hydrogen will verify online again and again, whatever it costs.
Hydrogen at XHigh also scores 18/21, with all three task 03 runs hitting the 900-second cap after 34, 120 and 138 web searches respectively; the other six tasks all pass, but each takes about twice as long as at High. The higher thinking tier buys no better result here — only a longer verification loop.
CyberQ also ran a fabrication test: 40 invented questions — nonexistent PyPI packages, papers, FortiGate commands and Taiwan regulations, each verified beforehand to have no real answer — plus 12 real questions as controls. Hydrogen fabricated 0 times; 3.8 Flash fabricated outright 7 times, and 3 more times first said nothing existed, then described details anyway. The direction matches Google's "five times fewer hallucinations" claim — and Hydrogen's caution is the same trait as its web searching: it does not invent answers, but when unsure it will keep checking, endlessly.
On API behaviour, models.get returns context figures matching the documentation (1,048,576 input, 65,536 output); the minimal tier returns a 400 error; requesting a nonexistent max thinking tier does not error. Two details worth remembering: setting thinkingBudget to 0 switches thinking off on 3.8 Flash, but does nothing on Hydrogen — it still spends 79 thinking tokens. On long context, a 1,013K-token needle test succeeds in 31.8 seconds. On cost, resending the same 209K-token request hits implicit caching on 207,318 tokens, cutting the price from US$0.321 to US$0.027. Multimodal inputs (PDF, xlsx, video, audio) all pass in single tests; parallel tool calls also work on 3.8 Flash, so they are not a Hydrogen-only capability.
Orchestrator is Google's multi-model routing mode. The Carbon version passes 18/21, the same as Carbon High (all three failures are task 03 timeouts), finishes 18% faster overall (71.6 vs 87.5 minutes, close to the official 19%), but uses 15% more tokens (9.27M vs 8.04M). The Argon version passes 20/21 against 18/21, also 18% faster in total, with 21% more tokens. As for the official "24% less quota" claim, the author states plainly that it cannot be verified: agy does not label which model actually ran each step, and quota is not the same thing as token count.
The test also ran into the quota wall: Carbon High was blocked by "Individual quota reached" from its 17th run, resetting only about 15 hours later; community reports describe Ultra users exhausting their quota in about 2 hours across 4 sessions. For heavy agent users, that matters more than any benchmark score when planning Hydrogen usage.
The author discloses the limits in the article itself: a home-made seven-task sample with only three runs per task; the first round on November 13 and the extra runs on November 15, two days apart, with agy moving from 1.5.7 to 1.5.8 to 1.5.9 and server load possibly differing; 3.8 Flash hit 11 503 errors that were excluded and rerun; and the multimodal and parallel-call checks are single tests on clean material. Those caveats travel with the conclusions.
Reference source: this article is an original rewrite produced by this site; facts and figures were checked against CyberQ — 實測 Gemini 4 Hydrogen,六道代理任務都更快,為何一道除錯題就拖慢全場? (Nov 13, 2026, updated Nov 15, 2026)
← Back to the Gemini 4 Hydrogen topic page