This is a follow-up to the three-part series on running local LLMs against a structured, five-phase (AT1–AT5) Delphi migration benchmark. Part 1 covered the benchmark design, Part 2 the results, Part 3 the practical recommendations. This post covers what changed when a new model showed up — plus one extra test we built specifically for this comparison, clearly marked as such below.
Benchmarks age. A few months after the original series went live, a new model appeared on Hugging Face: KAT-Coder-V2.5-Dev, released by Kwaipilot. What made it worth a dedicated re-test: the model card states it is an architecture-identical fine-tune of qwen3.6:35b-a3b — the model the original series recommended for routing and tool-calling. Same MoE architecture, same parameter count, same VRAM footprint. A same-hardware, same-weights-class fine-tune is as close to a controlled experiment as this kind of benchmarking gets.
A methodology note up front, because it matters for reading the numbers below: re-running the original AT1–AT5 harness against both models surfaced two real bugs in our own scoring scripts (a keyword-extraction regex that misfired on model names containing digits, and an answer-extraction step that returned empty strings when a model skipped the “write prose, then JSON” convention). Fixing them changed how format failures are counted — this pass scores every attempt on a strict raw basis, where a structurally non-compliant response counts as zero rather than being excluded from the average. That is a stricter metric than Part 2 used, and it is not directly comparable to the absolute scores published there. This post therefore compares exactly two models, a3b and KAT-Coder, both scored in this same session with the same fixed pipeline — it is not a re-ranking of the full Part 2 leaderboard.
We also built one additional test beyond the original five phases specifically for this comparison: AT6, a full model-routing test (a router persona picks the right model from a fixed catalog for 30 task descriptions, in English and German). It is new to this session, not part of the original three-part series, and is reported separately below for that reason.
The Numbers: a3b vs. KAT-Coder-V2.5-Dev, Same Pipeline
| Suite | KAT-Coder-V2.5-Dev | qwen3.6:35b-a3b |
|---|---|---|
| AT2 Comprehension | 0.784 | 0.708 |
| AT3 Patch Generation | 0.900 | 0.733 |
| AT4 Routing | 0.827 | 0.839 |
| AT5 Tool-Calling | 0.992 | 0.996 |
| AT1–AT4 combined | 0.837 | 0.760 |
| Throughput (AT2, avg) | ~220 tok/s | ~131 tok/s |
(AT6, bonus test, not part of the original series — see below.)
Tool-calling and routing land within noise of each other — both models are already near ceiling there, so a fine-tune has little room to move the needle. The separation shows up in comprehension and patch generation: AT2 climbs from 0.708 to 0.784, and AT3 patch quality from 0.733 to 0.900 — a large jump, achieved without the format-compliance problems that limited other models in this same raw-scoring pass (see the note above: a model that skips the required output structure scores zero on that attempt here, no exceptions).
The throughput gain is not a rounding error either. At roughly 220 tok/s against a3b’s 131 tok/s on the same AT2 workload, KAT-Coder is close to 70% faster at the same weight class and the same VRAM budget — a direct wall-clock win for batch processing, on top of the quality gain.
A separate real-prompt validation (single ~84k-token file, NumCtx=131072) showed both models holding 100% GPU utilization with no CPU offloading, and near-identical raw tokens/sec (~163–165) on that specific large-single-prompt workload — the AT2 speed gap shows up on the smaller, more numerous prompts typical of interactive comprehension/QA work, not on single giant inputs. Worth keeping in mind: throughput comparisons are workload-shaped.
Bonus Test: AT6 Full Model Routing (New in This Session)
This is not part of the original AT1–AT5 series — we built it specifically to stress-test routing decisions further, and it uses a different task format (a router persona selects from a fixed five-model catalog rather than classifying complexity tiers). Reported here for completeness, not as a series continuation:
| KAT-Coder-V2.5-Dev | qwen3.6:35b-a3b | |
|---|---|---|
| AT6 (60 tasks, EN+DE) | 0.767 | 0.772 |
Essentially tied — consistent with AT4/AT5 above, both models are strong, ceiling-adjacent routers.
A Second Test: The Actual RAG Chatbot, Not Just the Benchmark
The AT1–AT6 numbers measure raw model capability against static Delphi source files. We also had a second, more production-relevant opportunity: our Chat RAG assistant (a legal/software-support chatbot for our main software, built on the same local-Ollama infrastructure) already has an established evaluation harness — a 555-question set (real user-style questions across the full help corpus), judged against ground-truth source text in five outcome classes (grounded, answered-but-broader, honestly-declined, hallucinated, factually-wrong).
We ran the full 555-question set through the identical RAG pipeline (retrieval, generation, anti-hallucination verification — nothing simplified), once with a3b as the answer-generation model and once with KAT-Coder, judged both times by the same judge (Claude Opus 5, high reasoning effort — a deliberately higher bar than either model being evaluated):
| Outcome | a3b (n=555) | KAT-Coder (n=555) |
|---|---|---|
| Grounded | 268 | 275 |
| Answered (correct, broader than source) | 101 | 100 |
| Honestly declined | 132 | 124 |
| Hallucinated | 27 | 25 |
| Factually wrong | 27 | 31 |
| “Good” outcomes | 501/555 = 90.3% | 499/555 = 89.9% |
| Critical outcomes | 54/555 = 9.7% | 56/555 = 10.1% |
The two models are statistically tied on RAG quality. This is worth dwelling on, because an earlier 60-question sample (drawn from the same 555-question set, judged by a weaker model) had shown a clear-looking gap in KAT-Coder’s favor — 91.7% good vs. 86.7% good. The full 555-question run does not confirm that gap: it evaporates, and if anything, a3b answers marginally more questions well. KAT-Coder does hallucinate slightly less (25 vs. 27) but makes more outright factual errors (31 vs. 27), netting out to a wash. The lesson we’re taking from our own mistake here: a 60-question judged sample was enough to produce a confident-looking but wrong directional conclusion, and only the full run caught it. If you’re benchmarking RAG quality on a subsample for cost reasons, treat single-digit-percentage gaps as noise until you can afford the full set.
What This Means in Practice
The static-benchmark advantage (AT2/AT3, and the throughput gain) is real and reproducible: for a pipeline currently using qwen3.6:35b-a3b for comprehension and patch generation, KAT-Coder-V2.5-Dev is worth evaluating as a drop-in replacement — same hardware footprint, meaningfully better AT2/AT3 scores under a strict scoring standard, and faster inference. For routing and tool-calling (AT4/AT5/AT6), the two are interchangeable; pick on other grounds (e.g., whichever is already warm in your model-serving setup).
The RAG-quality advantage, on the other hand, did not survive contact with the full dataset — on the actual production pipeline, judged at a harder bar, KAT-Coder and a3b are statistically tied. If your use case is comprehension or patch-generation work, this data supports switching. If it’s exclusively RAG-chat quality, it does not — and a smaller, cheaper evaluation would have told you the opposite with false confidence.
What this post does not claim: that KAT-Coder is now the best model overall across the full original benchmark field (gemma4, devstral, and the rest were not re-scored in this session under the same strict methodology, so that comparison isn’t available here) — or that it’s a better RAG-chat model, which the full-sample evidence above does not support. The claim is narrower and, we think, more useful: against the specific model the original series recommended for exactly these tasks, an architecture-identical fine-tune measurably improves on it on the static AT2/AT3 benchmark, at meaningfully higher throughput, with no measurable downside or upside on live RAG quality. Three months was long enough for the static-benchmark result to change. If you’re running a3b in production today for comprehension or patching, it’s worth an afternoon to check whether it still should be; if you’re running it purely as a RAG-chat backend, there’s no rush.
Note on the Use of AI: This article was created with the "assistance" of generative AI. The content, technical statements, and conclusions have been reviewed and revised by the author. The author bears full responsibility for the publication.
No comments:
Post a Comment