This is another follow-up to the local-LLM benchmark series for Delphi development. Part 1 covered the benchmark design, Part 2 the results, Part 3 the practical recommendations, and the first follow-up compared KAT-Coder-V2.5-Dev against the reigning router/tool-calling model. This post has a new challenger benchmark table — but the real story turned out to be something the benchmark table cannot show at all.
A Routine Follow-Up, Not a Headline Release
The model this time is qwen3.8:27b-q4_K_M — a dense 27B model on the Qwen3.5 hybrid architecture (full attention every fourth layer, SSM/Mamba-style layers otherwise), Q4_K_M quantized, 17.7 GB on disk. Nothing about the release announcement suggested this post: it is simply the next model in the same family we already had two candidates from (qwen3.5, qwen3.6). We re-ran it through the same AT1–AT6 harness used for the KAT-Coder comparison, strict-raw scoring, Opus as judge, purely as a routine lifecycle check.
What came out of that routine check was a new comprehension and patch-generation champion — and, buried in the run logs, a configuration bug that had been silently costing GPU capacity for months without anyone noticing.
The Numbers: qwen3.8 vs. the Field
| Suite | qwen3.8:27b-q4_K_M | qwen3.6:27b | qwen3.6:35b-a3b | KAT-Coder-V2.5-Dev |
|---|---|---|---|---|
| AT1 Secrets (LLM-Judge) | 0.917 | 0.817 | 0.733 | n/a |
| AT2 Comprehension | 0.854 | 0.705 | 0.708 | 0.784 |
| AT3 Patch Generation | 0.983 | 0.767 | 0.733 | 0.900 |
| AT4 Routing | 0.802 | 0.808 | 0.839 | 0.827 |
| AT5 Tool-Calling | 1.000 | 1.000 | 0.996 | 0.992 |
| AT6 Routing v2 | 0.806 | 0.778 | 0.772 | 0.767 |
| AT1–AT4 combined | 0.889 | 0.774 | 0.753 | n/a (AT2–4: 0.837) |
| Throughput (AT2, avg) | ~29 tok/s | ~28 tok/s | ~131 tok/s | ~200 tok/s |
qwen3.8:27b-q4_K_M is the new comprehension and patch-generation champion in this series — and by a wide margin. AT3 patch quality reaches 0.983, beating even KAT-Coder's 0.900. AT2 comprehension jumps to 0.854, well clear of every dense 27B model tested so far. But it pays for that with the field's weakest AT4 routing score, and at ~29 tok/s it is the slowest model in the current lineup — a dense architecture running at roughly a sixth of KAT-Coder's throughput. The pattern from Part 2 holds again: comprehension/patch quality and routing reliability are not the same skill, and speed still tracks architecture (dense vs. MoE) more than raw parameter count.
The Production RAG Test: Another Statistical Tie
As with the KAT-Coder comparison, static benchmark scores are only half the picture. We ran the same 555-question production RAG evaluation (real user-style questions against our help corpus, judged by Claude Opus 5 in five outcome classes) with qwen3.8:27b-q4_K_M as the answer-generation model, against the current production baseline qwen3.6:27b:
| Outcome | qwen3.6:27b (n=555) | qwen3.8:27b-q4_K_M (n=555) |
|---|---|---|
| Grounded | 291 | 279 |
| Answered (correct, broader than source) | 117 | 102 |
| Honestly declined | 116 | 137 |
| Hallucinated | 12 | 10 |
| Factually wrong | 19 | 27 |
| "Good" outcomes | 524/555 = 94.4% | 518/555 = 93.3% |
| Critical outcomes | 5.6% | 6.7% |
A roughly one-point difference — statistically a tie, and the same story we already saw with a3b vs. KAT-Coder in the last post. What is worth noting is the failure pattern: qwen3.8 hallucinates slightly less often (10 vs. 12) but declines to answer more often even when the answer was actually present in the source material, and makes more outright factual errors (27 vs. 19). Extra caution here is not translating into extra accuracy — it is trading one failure mode for a different one, roughly at par. Two follow-ups in a row have now shown a large static-benchmark gap collapse to noise on the real production workload. That is starting to look less like a coincidence and more like a property of these leaderboard-style benchmarks: they measure something real, but not the thing that determines RAG-chat quality in production.
The Real Story: A Silent Context-Length Default
None of the numbers above are the reason this post exists. The reason is something the benchmark run surfaced almost by accident: our production host, WebChatHost, never passes a num_ctx value to Ollama. It never had to — until now.
When no context length is specified, Ollama does not pick something conservative. It loads the architecture's maximum. For qwen3.6:27b, that default happened to fit comfortably inside a 32 GB card, so the missing parameter was invisible — the model just worked, and nobody had reason to look closer. qwen3.8:27b-q4_K_M ships with a 262,144-token architecture maximum. Loaded with no explicit limit, the resulting instance needed 35.9 GB of VRAM on a 32 GB card. Ollama's answer to that mismatch is not to fail — it silently offloads the overflow to system RAM. In this case, roughly 15% of the model ended up running on CPU.
Nothing crashed. No error appeared anywhere in the logs. The only symptom was that the benchmark run, which should have taken a few hours, was tracking toward 13–14 hours of wall-clock time — 4.5 to 7 times slower than it should have been, for a model that scored highest of the entire field on comprehension and patching.
The fix required no code deployment: an Ollama model alias with num_ctx=32768, sized to what the production workload actually needs rather than what the architecture allows. That alone brought the model back to 100% GPU residency and cut the run from an estimated 13–14 hours down to roughly 3.5 hours.
This is not a model weakness, and it is not really a benchmark finding either — it is an infrastructure gotcha: a configuration gap that was harmless for exactly as long as every model we happened to deploy had a context-length default smaller than our VRAM budget. The moment that stopped being true, the missing parameter turned into a silent 4–7x slowdown with zero error signal. If you run Ollama in production and have never explicitly set num_ctx for a deployed model, it is worth checking now, before the next model swap makes the gap visible the expensive way.
What This Means in Practice
For comprehension and patch-generation workloads, qwen3.8:27b-q4_K_M is now the model to beat in this series — but only once its context length is pinned to what the workload actually needs, and only if the slower dense-model throughput is acceptable for your batching strategy. For routing, it is not the right choice; the MoE models (qwen3.6:35b-a3b, KAT-Coder) remain faster and more reliable routers. For RAG-chat quality specifically, the production evaluation again shows no meaningful difference from the existing baseline — a third data point suggesting that if your use case is RAG-chat, chasing the newest static-benchmark leader is not where the actual returns are.
The more durable takeaway is the infrastructure one: an unset num_ctx is not a safe default to rely on by omission. It is a bet — usually a winning one, until a model with a larger architecture maximum quietly loses it for you.
One gap in this comparison: at the time of writing, no qwen3.8:35b-a3b (MoE) variant was available to test. Given how consistently the a3b/MoE builds in this family have combined strong quality with dense-beating throughput, that is the model we would actually expect to end up the new all-round frontrunner — comprehension and patch quality close to the dense 3.8, at something closer to KAT-Coder's speed. Until that build shows up, qwen3.8:27b-q4_K_M is the best we can recommend for comprehension and patching, with the throughput caveat above standing.
Note on the Use of AI: This article was created with the "assistance" of generative AI. The content, technical statements, and conclusions have been reviewed and revised by the author. The author bears full responsibility for the publication.
No comments:
Post a Comment