ModelBrief · Kimi K3.1 Pro · Topic Page
Last updated Nov 20, 2026

Kimi K3.1 Pro: Fireworks SII Test — 14 of 17 Wins, With Trade-offs

ModelBrief · Published 2026-11-18 · Fact-check 2026-11-18 · Reference: Fireworks AI, SII, Sierra, Epiq, Rogo, Factory, Letta, Surge AI

This Fireworks AI evaluation deserves an article of its own — not because it hands K3.1 Pro a high score, but because it changes the ruler: instead of vendor-reported boards or general chat rankings, it lines the model up against 21 practitioner-built specialist benchmarks — bank support, finance research, company records, legacy-code maintenance, the actual work agents do. The conclusion is therefore more specific than "best open weights": against its predecessor Kimi K3 it wins 14 of 17 shared benchmarks; against Claude Opus 5.5 it is "some wins, some bills" — bank support wins on score and loses on the invoice.

Method: 21 practitioner-built benchmarks, four tasks examined in depth

The evaluation uses Fireworks' Specialized Intelligence Index (SII): 21 practitioner-built benchmark boards compared side by side. The research is based on a November 9, 2026 SII snapshot, and the page was updated on November 17, 2026. One version detail to note: the page's Published field reads 2025-11-06, a CMS year error — every model and benchmark in the piece is from November 2026, so cite the November 9 snapshot and the November 17 update. Beyond the board comparison, the article digs into four tasks at the task level — Sierra's τ³-Banking (bank support), Epiq's CorporateBench (company records), Rogo's Big Finance (finance research) and Factory's Legacy-Bench (legacy-code maintenance) — plus supporting analysis from Letta and Surge AI.

Fireworks SII deep-dive comparison on three tasks (Nov 9, 2026 snapshot) Kimi K3.1 Pro Rival (K3 / Opus 5.5) Legacy · Kimi 62.5% Legacy · K3 54.3% τ³-Banking · Kimi 63.3% τ³-Banking · Opus 56.0% CorpBench · Kimi 78.4 CorpBench · Opus 75.4 Fireworks SII, Nov 9, 2026 snapshot; orange is Kimi K3.1 Pro; units per the text All three win on score — but τ³-Banking costs more per fix; see "Against Opus 5.5"
Fireworks SII deep-dive comparison on three tasks (%; Fireworks, Nov 9, 2026 snapshot; orange is Kimi K3.1 Pro)

Against K3: 14 of 17, a big legacy-code win — and memory and contracts go backwards

Against its predecessor K3, K3.1 Pro wins 14 of the 17 shared benchmarks — the post-training gains are real. The cleanest win is Legacy-Bench: Factory's legacy-code maintenance benchmark, 97 tasks, where K3.1 Pro completes 62.5% against K3's 54.3% — 15% more completions at 48% lower model spend, a win on both score and cost. The legal-assistant benchmark LAB also rises from 13.1 to 15.4 (+18%). But the same table holds two clear regressions: Letta's FrontierMemory (memory creation) falls 22% — K3.1 Pro allocates only 50–65% of the source records, while K3 allocated all of them in 2 of 3 runs; on a 22-record task it allocated 1 record once and 0 another time. Worse still is LAB Contracts (contract review): 8.3 down to 3.0, a 64% drop. Post-training pushed code maintenance and general legal assistance forward and pushed memory creation and contract review backwards — before upgrading, check whether your own tasks happen to sit on those two weak spots.

Against Opus 5.5: bank support wins — and the bill goes up

The comparison with Claude Opus 5.5 is the more interesting one, because wins and costs are scored separately. The wins first: τ³-Banking (Sierra's bank support) resolves 63.3% against Opus 5.5's 56.0% — 13% more resolutions; and more importantly, accuracy — the share of account-operation fields changed against instructions falls from 8.5% to 2.6%, down about 69%; across the 12 hardest conversations, Kimi resolves 5 on average, while other models average no more than 2. But the cost is specific: $1.51 per successful resolution against Opus's $0.80, about 89% more expensive, taking more than twice as long and using about 3× the input tokens. CorporateBench (Epiq's company records) is 78.4 vs 75.4, at $0.47 vs $2.80 per task — 83% cheaper, a win on both score and cost; with one caveat: GPT-6.1 Sol scores 82.2 at $0.41 on this one, so Kimi is not the best answer here; and on employee-relations questions Kimi scores only 78, against Astra's 96 and Sol's 93 — a weak sub-category. Big Finance (Rogo's finance research) is 48.2%, level with Opus, at $0.17 vs $0.91 per task — 81% cheaper; but Sol is 52.0% at about $0.05 per task, scoring higher and costing less. Parallel Search scores 74.2 vs 75.4, slightly behind, at $0.27 vs $0.95 per task — 72% cheaper, a small score loss for a large price win. The two clean losses: ORCA (Traversal's production-incident root-cause location) at 32.9%, against Opus 5's 44.3% (Opus 5.5 untested) and Astra's 50.2%; and sudo L7 (Surge AI's production-codebase edits) at a 16.7% pass rate, against Opus 5.5's 45.6% and Sonnet 5.5's 50.6% — though 37% above K3, a relative improvement that is still low in absolute terms.

Surge AI's behavioural observation: it sees the risk — and says nothing, fixes nothing

Surge AI's supporting analysis looks not at scores but at what the model does inside its reasoning: across 256 software tasks, three runs each, in 64% of attempts the model identified a risk in its reasoning yet neither fixed it nor mentioned it in its final reply. That is a behavioural trait worth recording — the model is not blind to the problem, it sees it and leaves it alone. But the analysis carries a qualification that must be stated: it cannot be compared directly across models, because only K3.1 Pro's full reasoning is visible, while the comparison models expose only written notes — the same behaviour may exist in other models, just out of sight.

Routing simulation: splitting the work beats swapping the model — with one condition

The article closes with a routing simulation: 461 tasks, with a router picking the model per task from historical scores and costs for that task type, assuming $20 per point — a simulation assumption, not a real business value. The result: paired with Astra, the pair beats either single model in 98% of resamples; paired with Sol, 89%. Paired with Opus, it beats Opus alone in 99% of resamples and is cheaper in all of them — but beats Kimi alone in only 55% of resamples, meaning the Kimi-plus-Opus combination is worse than Kimi alone about half the time. Two concrete examples: on CorporateBench the pair scores about 100/128, 11 points above Opus alone, with cost falling from $358 to $100; on Legacy-Bench paired with Astra, 93/97 points go to Kimi, 61 tasks for $43 against 56 tasks for $235 with Astra alone — about 82% cheaper. Resampling measures variation across task samples, not run-to-run jitter — a different task batch would move the numbers, but roughly in the same direction. The simulation's message is clear: putting Kimi on the tasks it is good at and handing the rest to a partner beats wholesale replacement by any single model.

Conclusion: a split-the-work model

The whole evaluation condenses to one line: Kimi fits "splitting work by task", not "wholesale replacement". Its strengths are bank support, legacy code and parallel search; its weak spots are contract review, memory creation, employee relations and root-cause location. Before choosing it, hold it against your own task list: where the task sits on a strength, it is far cheaper than Opus 5.5; where it sits on a weak spot, no cheap score helps. And the evaluation's limits must be stated plainly: every result is a benchmark score or a simulation, not a real customer deployment; the $20-per-point figure in the routing simulation is an assumption, not a business price; and for a production router to work at all, it first needs to obtain or predict the task category — without that step, the simulated savings are only numbers on paper.

FAQ

What is SII, and how does it differ from the AA Index?

SII is Fireworks' Specialized Intelligence Index, comparing 21 practitioner-built specialist benchmarks (bank support, finance research, company records, legacy code) — how an agent performs on specific jobs. The AA Index is a general intelligence index covering overall capability. They complement each other: AA says it is "best open weights", SII says which specific tasks it is strong or weak on.

How can it "win 14" and still have obvious weak spots?

Because 17 is the count of shared benchmarks and 14 is the win count; the weak spots sit on individual items — memory creation (FrontierMemory down 22%), contract review (LAB Contracts down 64%) and root-cause location (ORCA 32.9%). Winning the total does not mean winning everywhere; selection depends on whether your tasks fall on the winning side or the losing side.

What does "$20 per point" mean in the routing simulation?

It is a simulation assumption, not a real business value: the router picks models by "score × $20 − cost". A different assumed value would change the pairing results. So the simulation should be read as "splitting work has potential", not as "this pairing will definitely save this much money".

Can this evaluation be used directly as a buying decision?

Not directly: every result is a benchmark score or a simulation, not a real customer deployment, and resampling measures variation across task samples, not run-to-run jitter. It is good for narrowing the shortlist and deciding which tasks to test first; a real buying decision still needs testing on your own tasks.

ModelBrief's take: This evaluation narrows K3.1 Pro's portrait from "best open weights" to "a split-the-work model" — bank support and legacy code are genuine strengths, contracts and memory are genuine weak spots, and the τ³-Banking bill (a score win at ~89% more per successful resolution) is a reminder: in agent selection, score and cost must be counted separately. It does not change the AA 46 conclusion, but it changes the usage: don't ask "should we switch to Kimi" — ask "which tasks go to Kimi".