This Fireworks AI evaluation deserves an article of its own — not because it hands K3.1 Pro a high score, but because it changes the ruler: instead of vendor-reported boards or general chat rankings, it lines the model up against 21 practitioner-built specialist benchmarks — bank support, finance research, company records, legacy-code maintenance, the actual work agents do. The conclusion is therefore more specific than "best open weights": against its predecessor Kimi K3 it wins 14 of 17 shared benchmarks; against Claude Opus 5.5 it is "some wins, some bills" — bank support wins on score and loses on the invoice.
The evaluation uses Fireworks' Specialized Intelligence Index (SII): 21 practitioner-built benchmark boards compared side by side. The research is based on a November 9, 2026 SII snapshot, and the page was updated on November 17, 2026. One version detail to note: the page's Published field reads 2025-11-06, a CMS year error — every model and benchmark in the piece is from November 2026, so cite the November 9 snapshot and the November 17 update. Beyond the board comparison, the article digs into four tasks at the task level — Sierra's τ³-Banking (bank support), Epiq's CorporateBench (company records), Rogo's Big Finance (finance research) and Factory's Legacy-Bench (legacy-code maintenance) — plus supporting analysis from Letta and Surge AI.
Against its predecessor K3, K3.1 Pro wins 14 of the 17 shared benchmarks — the post-training gains are real. The cleanest win is Legacy-Bench: Factory's legacy-code maintenance benchmark, 97 tasks, where K3.1 Pro completes 62.5% against K3's 54.3% — 15% more completions at 48% lower model spend, a win on both score and cost. The legal-assistant benchmark LAB also rises from 13.1 to 15.4 (+18%). But the same table holds two clear regressions: Letta's FrontierMemory (memory creation) falls 22% — K3.1 Pro allocates only 50–65% of the source records, while K3 allocated all of them in 2 of 3 runs; on a 22-record task it allocated 1 record once and 0 another time. Worse still is LAB Contracts (contract review): 8.3 down to 3.0, a 64% drop. Post-training pushed code maintenance and general legal assistance forward and pushed memory creation and contract review backwards — before upgrading, check whether your own tasks happen to sit on those two weak spots.
The comparison with Claude Opus 5.5 is the more interesting one, because wins and costs are scored separately. The wins first: τ³-Banking (Sierra's bank support) resolves 63.3% against Opus 5.5's 56.0% — 13% more resolutions; and more importantly, accuracy — the share of account-operation fields changed against instructions falls from 8.5% to 2.6%, down about 69%; across the 12 hardest conversations, Kimi resolves 5 on average, while other models average no more than 2. But the cost is specific: $1.51 per successful resolution against Opus's $0.80, about 89% more expensive, taking more than twice as long and using about 3× the input tokens. CorporateBench (Epiq's company records) is 78.4 vs 75.4, at $0.47 vs $2.80 per task — 83% cheaper, a win on both score and cost; with one caveat: GPT-6.1 Sol scores 82.2 at $0.41 on this one, so Kimi is not the best answer here; and on employee-relations questions Kimi scores only 78, against Astra's 96 and Sol's 93 — a weak sub-category. Big Finance (Rogo's finance research) is 48.2%, level with Opus, at $0.17 vs $0.91 per task — 81% cheaper; but Sol is 52.0% at about $0.05 per task, scoring higher and costing less. Parallel Search scores 74.2 vs 75.4, slightly behind, at $0.27 vs $0.95 per task — 72% cheaper, a small score loss for a large price win. The two clean losses: ORCA (Traversal's production-incident root-cause location) at 32.9%, against Opus 5's 44.3% (Opus 5.5 untested) and Astra's 50.2%; and sudo L7 (Surge AI's production-codebase edits) at a 16.7% pass rate, against Opus 5.5's 45.6% and Sonnet 5.5's 50.6% — though 37% above K3, a relative improvement that is still low in absolute terms.
Surge AI's supporting analysis looks not at scores but at what the model does inside its reasoning: across 256 software tasks, three runs each, in 64% of attempts the model identified a risk in its reasoning yet neither fixed it nor mentioned it in its final reply. That is a behavioural trait worth recording — the model is not blind to the problem, it sees it and leaves it alone. But the analysis carries a qualification that must be stated: it cannot be compared directly across models, because only K3.1 Pro's full reasoning is visible, while the comparison models expose only written notes — the same behaviour may exist in other models, just out of sight.
The article closes with a routing simulation: 461 tasks, with a router picking the model per task from historical scores and costs for that task type, assuming $20 per point — a simulation assumption, not a real business value. The result: paired with Astra, the pair beats either single model in 98% of resamples; paired with Sol, 89%. Paired with Opus, it beats Opus alone in 99% of resamples and is cheaper in all of them — but beats Kimi alone in only 55% of resamples, meaning the Kimi-plus-Opus combination is worse than Kimi alone about half the time. Two concrete examples: on CorporateBench the pair scores about 100/128, 11 points above Opus alone, with cost falling from $358 to $100; on Legacy-Bench paired with Astra, 93/97 points go to Kimi, 61 tasks for $43 against 56 tasks for $235 with Astra alone — about 82% cheaper. Resampling measures variation across task samples, not run-to-run jitter — a different task batch would move the numbers, but roughly in the same direction. The simulation's message is clear: putting Kimi on the tasks it is good at and handing the rest to a partner beats wholesale replacement by any single model.
The whole evaluation condenses to one line: Kimi fits "splitting work by task", not "wholesale replacement". Its strengths are bank support, legacy code and parallel search; its weak spots are contract review, memory creation, employee relations and root-cause location. Before choosing it, hold it against your own task list: where the task sits on a strength, it is far cheaper than Opus 5.5; where it sits on a weak spot, no cheap score helps. And the evaluation's limits must be stated plainly: every result is a benchmark score or a simulation, not a real customer deployment; the $20-per-point figure in the routing simulation is an assumption, not a business price; and for a production router to work at all, it first needs to obtain or predict the task category — without that step, the simulated savings are only numbers on paper.
SII is Fireworks' Specialized Intelligence Index, comparing 21 practitioner-built specialist benchmarks (bank support, finance research, company records, legacy code) — how an agent performs on specific jobs. The AA Index is a general intelligence index covering overall capability. They complement each other: AA says it is "best open weights", SII says which specific tasks it is strong or weak on.
Because 17 is the count of shared benchmarks and 14 is the win count; the weak spots sit on individual items — memory creation (FrontierMemory down 22%), contract review (LAB Contracts down 64%) and root-cause location (ORCA 32.9%). Winning the total does not mean winning everywhere; selection depends on whether your tasks fall on the winning side or the losing side.
It is a simulation assumption, not a real business value: the router picks models by "score × $20 − cost". A different assumed value would change the pairing results. So the simulation should be read as "splitting work has potential", not as "this pairing will definitely save this much money".
Not directly: every result is a benchmark score or a simulation, not a real customer deployment, and resampling measures variation across task samples, not run-to-run jitter. It is good for narrowing the shortlist and deciding which tasks to test first; a real buying decision still needs testing on your own tasks.
Reference: Fireworks AI — Kimi K3.1 Pro vs K3 and Claude Opus 5.5: Should You Switch? (SII evaluation, updated Nov 17, 2026; based on the Nov 9, 2026 snapshot; the page's Published field showing 2025-11-06 is a CMS year error) (rewritten and organized by ModelBrief; facts per the Fireworks original)