Plotly Data-Analysis Benchmark: 9 Points Above the Predecessor, Same Cost, Twice as Fast

On November 12, weights-release day, Plotly ran Inkling 2 Small through its own data-analysis benchmark (DataAnalyticsBench): 43 real data-analysis questions, answered in Plotly Studio Embedded with multi-turn tool calling, on fabricated wind-farm data (so the answers cannot leak into training sets). It is one of the few third-party measurements from that day — and it tests real workflows, not standard benchmarks. The headline: 39/43 (91%), the best U.S. open-weights score, third among all open-weights models; 9 points above the predecessor at about the same cost and twice the speed.

The score: 39/43, best U.S. open-weights result

Plotly's words: "If you want an open-weight model from a U.S. lab, Inkling 2 Small is now the one we'd pick for data analysis." But mind the sample size: each model ran once, and Plotly says repeat runs moved scores by up to 3 points, so a 1–2 point gap is close to a tie — the 39-vs-38 "U.S. first" is exactly that size of gap.

Where it won: judgment questions, not SQL

By category, Inkling 2 Small was perfect in 4 of 6: ambiguous questions 5/5, impossible requests 5/5, data quality problems 3/3, statistics traps 5/5; analytics questions 19/20; hidden-pattern questions 2/5. Plotly notes the gains over Inkling-Small came on questions that take judgment rather than SQL: the statistics traps are questions where the obvious calculation gives a misleading answer — only 15 of the 49 models on Plotly's board got all five, and among U.S. open-weights models only Ricursive's Ripple-65B Preview has done it too. On planted data-quality problems it caught all three, where Inkling-Small caught one and the first Inkling none. Among open-weights models whose weights are out, it is the only one to get both of those last two categories perfect.

Where it lost: all four misses are real

Plotly hand-checked each miss against the transcript and answer key: all four are genuine failures. On one analytics question it worked out the answer but wrote its final table call out as text instead of running it, so the table never appeared. The other three were hidden-pattern questions, the hardest part of the exam: each points toward something unusual in the data without naming it, and the model has to go looking. It found 2 of 5, tied with Muse Spark Open for the most of any U.S. open-weights model; Kimi K3.1 Pro and MiniMax M3.1 Flash found all five. It did go looking: about 46 queries per hidden-pattern question, far above the board median of about 26 — it searched, but did not find everything.

Cost and speed: cheap and fast, but not the cheapest

At $1.81 a run — about $105 per person per year at Plotly's rollout numbers (1,000 people, 10 questions a day, 250 working days) — the low cost has two causes: only about 5,700 output tokens per question (Muse Spark Open about 7,800, MiniMax about 32,000), and about 96% of input tokens coming from the provider's cache. On speed, a 13-second median per question and about 16 minutes for the whole exam; only 3 of the 49 models on the board are faster (Mercury 3, gpt-oss-120b, Nemotron 3.5 Super VL), and they scored 35, 27 and 30 — no model scoring 39 or more, open or closed, has been faster. But among closed models, Particle-1 scored 40 for $0.35 and GPT-6.1 Terra 40 for $1.31 — if open weights are not a requirement, it is neither the most accurate nor the cheapest.

One number worth noting: local runs need only 356GB?

Plotly says about 356GB of memory at 4-bit, "which fits on one 512 GB Mac Studio." That differs from the official model card's "at least 500GB of aggregated VRAM" — the two measure different things: Plotly is talking about the weights' memory footprint, while the official bar presumably includes runtime overhead (KV cache, activations). This site's topic page follows the official 500GB figure; Plotly's 356GB suggests the weight files themselves are smaller than the official bar, so leave headroom when planning a deployment.