ModelBrief · Muse Ember · Topic Page
Last updated Nov 20, 2026

Muse Ember Benchmarks, All of Them: AA 36 Ties Haiku — and FrontierCode Is Lost

ModelBrief · Published 2026-11-16 · Fact-check 2026-11-16 · Reference: Artificial Analysis, Meta research blog

Muse Ember's benchmark data comes in two layers: the top layer is the Intelligence Index of 36 (max effort), independently published by Artificial Analysis on November 9; the lower layer is six component scores reproduced in charts in Meta's launch post. The two layers carry different weight — this article separates who is speaking first, then takes each score apart, and collects the weak spots at the end.

The independently published layer: AA Index 36, $0.14 per task

AA Intelligence Index v5.0 (November 9, 2026, max effort): Ember 36, level with Claude Haiku 5.5 (max), behind Muse Spark 1.3 (max) at 37. Cost per task: Ember $0.14, Haiku $0.41, Spark $2.39. AA's own characterisation is the "cheapest model scoring 36 or higher"; at-pulse adds that Ember ranks 2nd of 33 comparable models at ~$0.14 per task. This is the only number in Ember's entire benchmark record that an independent board published itself — and it is the source of the headline "ties Haiku at a third of the cost".

AA Intelligence Index v5.0 (max effort) Muse Ember Claude Haiku 5.5 Muse Spark 1.3 AA Index v5.0 36 36 37 Cost per task $0.14 Haiku $0.41 · Spark $2.39 (not drawn to scale; see text)
AA Intelligence Index v5.0 (points; Artificial Analysis, Nov 9, 2026, max effort; orange is Muse Ember, grey is Claude Haiku 5.5, teal is Muse Spark 1.3). The cost-per-task row draws only Ember's $0.14; Haiku's $0.41 and Spark's $2.39 would run off the chart at the same scale — see the text for the figures.

The Meta-reported layer: six component benchmarks, five wins and one loss

Meta's launch-post charts (all vendor-reported, reproducing AA evaluations; rival scores as Meta lists them, not each vendor's published values): AutomationBench-AA 37.8% (Haiku 13.9%, Luna 20.2%), the widest gap — Ember scores nearly 2.7× Haiku; Terminal-Bench 4.0 40.9% (Haiku 39.2%), a win by just 1.7 points; FrontierCode 2.0-AA 27.2% (Haiku 34.9%), a loss by 7.7 points; OSWorld 2.1 69.5% (Haiku 68.9%, Luna 50.5%), a win by just 0.6 points; Agents' Last Exam 61.6 (57.0 / 57.1); PerceptionBench 80.6 (78.6 / 62.8). Ember wins five of six — but two of the wins are under 2 points, and the loss is 7.7 points. Beyond the headline, the real distribution is "a big business-workflow win, a clear coding loss, and narrow wins elsewhere".

Muse Ember six component benchmarks (Meta-reported) Muse Ember Claude Haiku 5.5 GPT-6 Luna AutomationBench-AA 37.8% 13.9% 20.2% Terminal-Bench 4.0 40.9% 39.2% FrontierCode 2.0-AA 27.2% 34.9% OSWorld 2.1 69.5% 68.9% 50.5%
Six component benchmarks (%; Meta research blog, Nov 9, 2026; all vendor-reported, reproducing AA evaluations; orange is Muse Ember, grey is Claude Haiku 5.5, teal is GPT-6 Luna; Luna has no score in Terminal-Bench 4.0 or FrontierCode)

AA-Omniscience 21.1%: the bottom-of-class score under Index 36

AA-Omniscience 21.1%, the lowest among models with similar overall index scores (via alphasignal). Index 36 summarises general intelligence; Omniscience measures a different dimension — Ember's 36 hides a bottom-of-class result. Read together with the FrontierCode loss, Ember's capability profile is complete: level with Haiku on the headline index, with clear weak spots in coding and on Omniscience. When choosing a model, don't just remember 36 — remember 21.1% alongside it.

AA Coding Agent Index 46: Meta's number, not an AA board

Ridge's caution deserves its own paragraph: the AA Coding Agent Index of 46 is a figure from Meta's launch post, and Ridge notes that "Meta's other figures … are its claims". In other words, 46 has the same provenance as the six component benchmarks — Meta. Don't cite it as a second independently published AA score. As of Nov 16, the only Ember results AA published independently are the Index of 36 and, via reporting, Omniscience at 21.1%.

The harness trap: the headline claim is unaudited

Meta's headline claim is "more intelligence per dollar than Claude Haiku 5.5 or GPT-6 Luna at any reasoning setting". Three caveats: first, it is Meta's reading of AA results, not AA's conclusion; second, the unit economics of the cost comparison have not been third-party audited (the dineroit / codema caution); third, Meta's chart notes Ember's costs were estimated at proposed pricing. Most importantly, Meta published benchmark results only at max effort — the "any reasoning setting" claim has no lower-effort data behind it. Simon Willison's hands-on test (Nov 10) found quality does not rise steadily with effort: one request at high failed to return a complete drawing. A single-person, single-task observation, not a benchmark — but it directly questions the "any setting" wording.

Independent evidence: almost nothing beyond AA

As of Nov 16, LiveBench has no Ember run and LMArena has no listing; Simon Willison's hands-on test is the only systematic third-party hands-on record, and it is one person on one task. The model is 7 days old, so the absence means "not yet", not "zero" — but until a score lands, every capability claim about Ember rests on one AA index number plus Meta's retelling. That is the first item on the 8-hour tracking loop's watchlist: first LiveBench / LMArena listings, and AA v5.x rescores.

The weak-spot list: the gaps beyond the six benchmarks

The weak spots, collected: FrontierCode 2.0-AA loses to Haiku 5.5 (27.2% vs 34.9%); AA-Omniscience 21.1% is the lowest in its class; the Terminal-Bench 4.0 and OSWorld 2.1 leads are under 2 points; all six component benchmarks are Meta-reported, not an AA-published board; Meta published max-effort results only, leaving lower-effort quality officially unverified; no LiveBench / LMArena score; parameter count undisclosed. These don't invalidate 36 — but they set Ember's boundaries: business workflows and agent execution are worth trying, code-heavy work should check FrontierCode first, and lower effort levels should not be assumed equal to max.

FAQ

What does AA Index 36 mean?

Level with Claude Haiku 5.5 (max), below Muse Spark 1.3 (max) at 37; cost per task $0.14 vs Haiku $0.41 and Spark $2.39. AA calls it the "cheapest model scoring 36 or higher". It is the only index score AA published independently.

Which board does Ember lose to Haiku on?

FrontierCode 2.0-AA: 27.2% vs 34.9%, a 7.7-point loss. AA-Omniscience 21.1% is also the lowest among models with similar overall scores. Both are weak spots in coding and general-capability dimensions.

Did AA publish the six component benchmarks?

No. They come from charts in Meta's launch post — Meta's retelling of AA evaluations; the AA Coding Agent Index of 46 is likewise Meta's figure. Only the Intelligence Index of 36 was independently published by AA on Nov 9.

Is there a LiveBench or LMArena score?

No. As of Nov 16, LiveBench has no run and LMArena has no listing. The model is 7 days old; the absence means "not yet", not "zero".

ModelBrief's take: Ember's benchmark profile is remarkably consistent: level with Haiku on the headline index, a big win on business workflows, a loss on coding, bottom of its class on Omniscience. The official and independent numbers don't disagree on direction — they disagree on existence: 36 is what AA says, the six component boards are what Meta says AA said, and lower-effort quality is what nobody has measured. The key to reading Ember's scores isn't memorizing 36 — it's remembering 21.1% beside it, and that "any reasoning setting" has no lower-effort data behind it.

Reference: Artificial Analysis — launch evaluation (Intelligence Index v5.0 36, $0.14 per task) · Meta research blog — six component benchmarks and the headline claim · alphasignal (AA-Omniscience 21.1%) · at-pulse (2nd of 33 comparable models) · Simon Willison hands-on test (Nov 10) (rewritten and organized by ModelBrief; facts per the AA evaluation and the Meta launch post)