This is a follow-up to the three-part series on running local LLMs against a structured, five-phase (AT1–AT5) Delphi migration benchmark. Part 1 covered the benchmark design, Part 2 the results, Part 3 the practical recommendations. This post covers what changed when a new model showed up — plus one extra test we built specifically for this comparison, clearly marked as such below.
Benchmarks age. A few months after the original series went live, a new model appeared on Hugging Face: KAT-Coder-V2.5-Dev, released by Kwaipilot. What made it worth a dedicated re-test: the model card states it is an architecture-identical fine-tune of qwen3.6:35b-a3b — the model the original series recommended for routing and tool-calling. Same MoE architecture, same parameter count, same VRAM footprint. A same-hardware, same-weights-class fine-tune is as close to a controlled experiment as this kind of benchmarking gets.
A methodology note up front, because it matters for reading the numbers below: re-running the original AT1–AT5 harness against both models surfaced two real bugs in our own scoring scripts (a keyword-extraction regex that misfired on model names containing digits, and an answer-extraction step that returned empty strings when a model skipped the “write prose, then JSON” convention). Fixing them changed how format failures are counted — this pass scores every attempt on a strict raw basis, where a structurally non-compliant response counts as zero rather than being excluded from the average. That is a stricter metric than Part 2 used, and it is not directly comparable to the absolute scores published there. This post therefore compares exactly two models, a3b and KAT-Coder, both scored in this same session with the same fixed pipeline — it is not a re-ranking of the full Part 2 leaderboard.
We also built one additional test beyond the original five phases specifically for this comparison: AT6, a full model-routing test (a router persona picks the right model from a fixed catalog for 30 task descriptions, in English and German). It is new to this session, not part of the original three-part series, and is reported separately below for that reason.
The Numbers: a3b vs. KAT-Coder-V2.5-Dev, Same Pipeline
| Suite | KAT-Coder-V2.5-Dev | qwen3.6:35b-a3b |
|---|---|---|
| AT2 Comprehension | 0.784 | 0.708 |
| AT3 Patch Generation | 0.900 | 0.733 |
| AT4 Routing | 0.827 | 0.839 |
| AT5 Tool-Calling | 0.992 | 0.996 |
| AT1–AT4 combined | 0.837 | 0.760 |
| Throughput (AT2, avg) | ~220 tok/s | ~131 tok/s |
(AT6, bonus test, not part of the original series — see below.)
Tool-calling and routing land within noise of each other — both models are already near ceiling there, so a fine-tune has little room to move the needle. The separation shows up in comprehension and patch generation: AT2 climbs from 0.708 to 0.784, and AT3 patch quality from 0.733 to 0.900 — a large jump, achieved without the format-compliance problems that limited other models in this same raw-scoring pass (see the note above: a model that skips the required output structure scores zero on that attempt here, no exceptions).
The throughput gain is not a rounding error either. At roughly 220 tok/s against a3b’s 131 tok/s on the same AT2 workload, KAT-Coder is close to 70% faster at the same weight class and the same VRAM budget — a direct wall-clock win for batch processing, on top of the quality gain.
A separate real-prompt validation (single ~84k-token file, NumCtx=131072) showed both models holding 100% GPU utilization with no CPU offloading, and near-identical raw tokens/sec (~163–165) on that specific large-single-prompt workload — the AT2 speed gap shows up on the smaller, more numerous prompts typical of interactive comprehension/QA work, not on single giant inputs. Worth keeping in mind: throughput comparisons are workload-shaped.
Bonus Test: AT6 Full Model Routing (New in This Session)
This is not part of the original AT1–AT5 series — we built it specifically to stress-test routing decisions further, and it uses a different task format (a router persona selects from a fixed five-model catalog rather than classifying complexity tiers). Reported here for completeness, not as a series continuation:
| KAT-Coder-V2.5-Dev | qwen3.6:35b-a3b | |
|---|---|---|
| AT6 (60 tasks, EN+DE) | 0.767 | 0.772 |
Essentially tied — consistent with AT4/AT5 above, both models are strong, ceiling-adjacent routers.
A Second Test: The Actual RAG Chatbot, Not Just the Benchmark
The AT1–AT6 numbers measure raw model capability against static Delphi source files. We also had a second, more production-relevant opportunity: our Chat RAG assistant (a legal/software-support chatbot for our main software, built on the same local-Ollama infrastructure) already has an established evaluation harness — a 555-question set (real user-style questions across the full help corpus), judged against ground-truth source text in five outcome classes (grounded, answered-but-broader, honestly-declined, hallucinated, factually-wrong).
We ran the full 555-question set through the identical RAG pipeline (retrieval, generation, anti-hallucination verification — nothing simplified), once with a3b as the answer-generation model and once with KAT-Coder, judged both times by the same judge (Claude Opus 5, high reasoning effort — a deliberately higher bar than either model being evaluated):
| Outcome | a3b (n=555) | KAT-Coder (n=555) |
|---|---|---|
| Grounded | 268 | 275 |
| Answered (correct, broader than source) | 101 | 100 |
| Honestly declined | 132 | 124 |
| Hallucinated | 27 | 25 |
| Factually wrong | 27 | 31 |
| “Good” outcomes | 501/555 = 90.3% | 499/555 = 89.9% |
| Critical outcomes | 54/555 = 9.7% | 56/555 = 10.1% |
The two models are statistically tied on RAG quality. This is worth dwelling on, because an earlier 60-question sample (drawn from the same 555-question set, judged by a weaker model) had shown a clear-looking gap in KAT-Coder’s favor — 91.7% good vs. 86.7% good. The full 555-question run does not confirm that gap: it evaporates, and if anything, a3b answers marginally more questions well. KAT-Coder does hallucinate slightly less (25 vs. 27) but makes more outright factual errors (31 vs. 27), netting out to a wash. The lesson we’re taking from our own mistake here: a 60-question judged sample was enough to produce a confident-looking but wrong directional conclusion, and only the full run caught it. If you’re benchmarking RAG quality on a subsample for cost reasons, treat single-digit-percentage gaps as noise until you can afford the full set.
What This Means in Practice
The static-benchmark advantage (AT2/AT3, and the throughput gain) is real and reproducible: for a pipeline currently using qwen3.6:35b-a3b for comprehension and patch generation, KAT-Coder-V2.5-Dev is worth evaluating as a drop-in replacement — same hardware footprint, meaningfully better AT2/AT3 scores under a strict scoring standard, and faster inference. For routing and tool-calling (AT4/AT5/AT6), the two are interchangeable; pick on other grounds (e.g., whichever is already warm in your model-serving setup).
The RAG-quality advantage, on the other hand, did not survive contact with the full dataset — on the actual production pipeline, judged at a harder bar, KAT-Coder and a3b are statistically tied. If your use case is comprehension or patch-generation work, this data supports switching. If it’s exclusively RAG-chat quality, it does not — and a smaller, cheaper evaluation would have told you the opposite with false confidence.
What this post does not claim: that KAT-Coder is now the best model overall across the full original benchmark field (gemma4, devstral, and the rest were not re-scored in this session under the same strict methodology, so that comparison isn’t available here) — or that it’s a better RAG-chat model, which the full-sample evidence above does not support. The claim is narrower and, we think, more useful: against the specific model the original series recommended for exactly these tasks, an architecture-identical fine-tune measurably improves on it on the static AT2/AT3 benchmark, at meaningfully higher throughput, with no measurable downside or upside on live RAG quality. Three months was long enough for the static-benchmark result to change. If you’re running a3b in production today for comprehension or patching, it’s worth an afternoon to check whether it still should be; if you’re running it purely as a RAG-chat backend, there’s no rush.
No comments:
Post a Comment