ModelBrief · MAI Code 2 · Topic Page
Last updated Nov 20, 2026

MAI Code 2 Benchmarks: Four-Test Average, Blind-Test 64%

ModelBrief · Published 2026-11-20 · Fact-check 2026-11-20 · Reference: Microsoft AI product and Ignite posts, GitHub changelog

Every benchmark figure for MAI Code 2 comes from a single source: Microsoft itself. This article tallies every checkable number — the four-benchmark average, average pass rates, per-task costs, the blind-test preference, Flash's three benchmarks — and labels the methodology and wording of each. The conclusion first: there is no independent score anywhere, and Artificial Analysis, LiveBench and LMArena have not listed MAI Code 2.

The headline figures (all Microsoft-reported)

TestResultWording
Four-benchmark average vs GPT-6.1 Sol98% of the performance at about 40% of the costMicrosoft-chosen benchmarks, 50% promo-price basis; no independent verification
Four-benchmark average vs Claude Opus 5.596% of the performance at about 5% of the costMicrosoft-chosen benchmarks, 50% promo-price basis; no independent verification
Average pass rate (Microsoft blog chart)MAI Code 2 63.2% / GPT-6.1 Sol 64.1% / Opus 5.5 65.3%Average pass rate across the four benchmarks; MAI Code 2 is actually the lowest
Average cost per task (Microsoft blog chart)MAI Code 2 $0.309 / GPT-6.1 Sol $0.761 / Opus 5.5 $6.306MAI Code 2's cost computed at the 50% promotional price
Front-end blind test vs GPT-6 Sol (258 people)Preference for MAI Code 2: 64% (Microsoft has since unified the figure)Microsoft unified the figure at 64% on Nov 18; 67% and 68% were pre-correction values
Flash, three benchmarks vs GPT-6 LunaComparable performance at 65% of the costThe three benchmarks are unnamed; 50% promo-price basis
Flash, three benchmarks vs Claude Haiku 5.5Comparable performance at 20% of the costThe three benchmarks are unnamed; 50% promo-price basis
AT&T custom Flash vs GPT-6.1 SolWins on standard 5G network analysis at 50x lower costSingle-customer anecdote, not a public benchmark

What the four benchmarks are and how they ran

Microsoft says the four benchmarks — DeepSWE 1.1, Terminal Bench 2.1, SWE-Atlas, FrontierCode v1.1 — target "the everyday enterprise coding tasks we see in GitHub Copilot". The methodology comes from the product post's footnotes and deserves a line-by-line read:

First, Microsoft chose the benchmarks. The four tests are Microsoft's picks, not a third party's — and they happen to be MAI Code 2's home ground (everyday enterprise coding), not its weak spots (fundamental science research, very-long-horizon agentic work, which Microsoft itself says the models are not tuned for).

Second, every model ran at max reasoning effort. That is even-handed across models, but the bill at max effort is not the bill of everyday use; the cost comparison uses the max-effort bill.

Third, three of the four ran on Microsoft's internal platform. DeepSWE 1.1, Terminal Bench 2.1 and SWE-Atlas ran on Microsoft's internal evaluation platform MSBench, via the Copilot CLI with no subagents; only the FrontierCode v1.1 scores and costs come from Artificial Analysis. In other words, the test environment for three of the four benchmarks was entirely under Microsoft's control.

Fourth, the per-benchmark scores are in the model card, not the product post. The product post gives only the 98% and 96% averages, but the model card's main table publishes the per-benchmark pass rates (MSBench, max reasoning effort, mean of three runs): MAI Code 2 at 73.2% on DeepSWE 1.1, 84.6% on Terminal-Bench 2.1, 43.1% on SWE Atlas and about 52% on FrontierCode v1.1 (worked backwards from the 63.2% average; Microsoft did not publish that figure separately); Flash at 59.9%, 78.3% and 44.8%. Two caveats: the scores are still Microsoft-reported, and the model card's safety section prints a different set of DeepSWE/SWE Atlas figures (60.2%/27.4%, 80.9%/65.2%) with no explanation of the different setup — cite the main table. With no per-benchmark scores for GPT-6.1 Sol or Opus 5.5, the 98%/96% averages still cannot be checked item by item — and per-benchmark, MAI Code 2's weak spot is SWE Atlas (43.1%).

What "98%" hides: the lowest average pass rate of the three

Microsoft's official blog chart gives the average pass rates: MAI Code 2 at 63.2%, GPT-6.1 Sol at 64.1%, Claude Opus 5.5 at 65.3%. In absolute pass rate across the four benchmarks, MAI Code 2 is the lowest of the three — "98%" is the ratio against GPT-6.1 Sol (63.2 ÷ 64.1 ≈ 98.6%), not an absolute lead. Average cost per task from the same chart: MAI Code 2 at $0.309, GPT-6.1 Sol at $0.761, Claude Opus 5.5 at $6.306 — with MAI Code 2's cost computed at the 50% promotional price, which doubles when the promotion ends. These two sets of numbers are the key to "98% of the performance at about 40% of the cost": on performance it trails slightly, the cost advantage is the real selling point — and half of that advantage comes from the promotional price.

The blind-test figure, clarified: Microsoft has unified it at 64%

The 258-person front-end blind test is the most human number in Microsoft's product post: developers preferred MAI Code 2's front-end app output over GPT-6 Sol's. The number once had a problem: Microsoft's own posts showed three versions — 64%, 67% and 68% — with the product post's main text giving 64% and the other two appearing in the Ignite post and in another statement in the product post. AI-Oriens verified on November 18, 2026 that both posts now read 64% (258 reviewers, preferred on both aesthetics and functionality), and that 67% and 68% were pre-correction values. This site now uses the unified 64% figure, with the old values kept only as a historical note.

Flash's three benchmarks: not even named

MAI Code 2 Flash's claim is thinner: Microsoft says it matches GPT-6 Luna and Claude Haiku 5.5 on three benchmarks for "day-to-day, high-volume routine coding tasks", at 65% and 20% of the cost (50% promo-price basis). But the three benchmarks are unnamed — without knowing which three, there is no way to check what "comparable" means. Microsoft's blog chart separately gives Flash an average pass rate of 61.0%, above GPT-6 Luna's 55.1% and Claude Haiku 5.5's 53.9%, at $0.052 per task against $0.08 and $0.319 (Flash's cost likewise at the 50% promotional price). Those numbers are more positive than "comparable", but they are equally Microsoft-reported, and the benchmarks are unnamed.

AT&T's "50x": one customer, not a benchmark

Microsoft says a custom AT&T version of MAI Code 2 Flash outperformed GPT-6.1 Sol on standard 5G network analysis while costing 50x less. That is a single-customer anecdote: a custom build, one task type, one customer, with no published test method and no statement of which metric "outperformed" refers to. Read it as a marketing case, not as a benchmark.

No independent score: AA, LiveBench and LMArena have not listed it

As of Nov 20, 2026, MAI Code 2 appears on no independent board. Google autocomplete shows "mai code 2 artificial analysis" — that is search interest, people wanting to look up its AA score, not an AA listing. The FrontierCode v1.1 scores and costs come from AA data cited in Microsoft's footnotes, which is equally not an independent AA listing. The first item on this topic's tracking list is the first AA, LiveBench or LMArena listing; until then, every score can only be labelled "Microsoft-reported".

ModelBrief's take: MAI Code 2's benchmark story has two layers. On methodology: it picked its own home ground, ran on its own platform, and gave only the average in the product post (per-benchmark scores are in the model card's main table). On the numbers: its average pass rate is actually the lowest of the three, the cost advantage is the real selling point, and half of that advantage comes from the promotional price. The blind-test figure is a further warning — even Microsoft's own numbers need checking, though Microsoft has since unified it at 64%. Until an independent score appears, every citation must carry the words "Microsoft-reported".