Friday, August 7, 2026

Agents that talk to each other — and wake each other up

Hello, my friends!

Last time I told you what happened to #D.MVVM after four and a half years of silence. There is one detail in that story I left out, and it deserves its own post.

The blog post itself was written by an agent. Not the framework — the post. And to write it, that agent did not ask me. He asked the other agent, the one who had been working on the framework for the last two weeks. He asked him about twenty questions, got the answers, and wrote from those.

Two of the answers were wrong. I will come back to that.

But the mechanism behind it is what I actually want to talk about, because I think it is the most underrated thing I have built this year: a small module that lets my agent sessions talk to each other.


The problem nobody warns you about

When you start working seriously with coding agents, you very quickly stop running one.

You run one on the framework. One on the backend. One on the blog. Sometimes a fourth one on something completely unrelated, because you had an idea and did not want to lose it.

And then you notice two things.

The first is that they know nothing about each other. Each session sits in its own context, blind and deaf. The agent working on the blog has no idea that the framework agent renamed a class two hours ago. If he needs to know, you are the transport. You copy something out of one terminal and paste it into another, and you translate between two conversations that are each ten thousand lines long. You become the message bus. That gets old fast.

The second is worse. Two agents on the same repository will happily edit the same unit at the same time. Neither of them is doing anything wrong. They simply cannot see each other. You find out when the compile breaks, or later, when you read a diff and cannot explain how half of it got there.

Every solution I saw for this was some variant of "just don't do that" — run them one at a time, or split the repository, or keep a list on a whiteboard. That is not a solution; that is a workaround. I have written enough posts about workarounds.

But agents can start their own agents!

Yes, they can, and this is the first question I get whenever I describe this. Most agent tools can spawn sub-agents: the agent delegates a task, a helper is created, does the job, reports back, and is gone. So why would I bother running four terminals?

Because those are two completely different things, and the difference is not size.

A sub-agent is a tool call with a personality. He lives inside his parent's task; he knows only what his parent told him, and when he answers, he ceases to exist. He has no project of his own, no history, no opinion formed over two weeks of working on the same code. And he can be asked exactly once. You cannot come back tomorrow and ask him what he meant, because there is no "him" anymore.

My agents are colleagues, not helpers. Each one sits in his own window with his own project, his own instructions, his own accumulated knowledge of one particular corner of my code. The framework agent has been on that framework for weeks. He knows why a decision was made, not just what the code says. That is exactly the thing you want to ask a question to — and it is exactly the thing a freshly spawned sub-agent can never be.

There is a second reason, and it is less philosophical: I can watch. Four windows means I see four trains of thought running, and I can stop the one that is heading in a stupid direction before it arrives. A sub-agent works inside somebody else's context. By the time I see the result, it is a result.

So sub-agents are for work you want done. Separate sessions are for knowledge you want to keep. A2A is what connects the second kind.

So what does it do?

Three things. That is the whole module.

They can see each other. Every session registers itself, and any agent can ask who else is currently alive and what they are working on. That alone removes a surprising amount of confusion. Before an agent starts, he can find out whether he is alone or in company — and behave accordingly.

They can talk. Not by passing single messages, but in chat-rooms. An agent opens a chat-room, invites another session, and the two of them have a conversation that survives more than one question. It works one to one, and it works one to many — one agent can pull three others into the same room at once. That matters more than it sounds. When my blog agent interviewed the framework agent, he did not fire off one query. He asked, got an answer he did not fully believe, asked a follow-up, and came back a second time hours later — into the same room, with the same context still there.

There is also a blocking variant. The asking agent stops and waits until the answer arrives. That is the one you want when the next step genuinely depends on the reply, and it turns a chat-room into something much closer to a synchronous call between two minds.

They can wake each other up. This is my favourite part, and it is the bit I have not seen anywhere else.

An agent has no inbox. He is not sitting there checking for mail, because there is nothing in him that would do the checking. He is either working, or he is waiting for a human to type something. In both cases, calling out to him is calling into an empty room.

So there are two ways in, one for each state.

If he is working, the message rides along. The next time he calls any of 

my MCP-tools— reads a file, runs a search, whatever he happens to be doing — the message is delivered inside that tool's result, with a marker in front of it that means someone is blocked on you right now; answer this first. No polling loop, no timer. He finds out because he did something else entirely.

And if he is doing nothing at all — idle, at the prompt, waiting for me, not calling a single tool — the message is pushed straight into his session. He wakes up on his own. Nobody types anything, nobody switches windows, nobody has to notice that somebody else needs him. The other agent knocks, and he answers.

There is one more part to this, and it is the part I am quietly proud of. The agent being woken does not have to know that any of this exists.

Think about what that means. A session I started for something completely unrelated, with no instructions about agent communication, that has never seen a chat room in his life — he can still be pulled into a conversation. Because the wake-up call is not just "you have a message". It carries everything he needs: which room to enter, how to enter it, how to reply, and how to leave when the conversation is done. The invitation and the manual arrive as one.

So a stranger can be brought into a discussion and take part in it, without ever having been told beforehand that discussions were possible. That is what turns this from a feature into infrastructure.

I like all of it because it is the kind of solution you only find when you accept how the thing actually works instead of fighting it.

The board

The fourth piece is not communication at all, and it is the one that saves the most damage.

Before an agent touches a shared unit, he puts it on a board: I am working on this file, and here is why. Any other agent can read that board at any time. If a unit is already taken, the second agent does not edit it — he coordinates, or he waits, or he works on something else.

Two details make it work in practice.

Every entry carries a note in the agent's own words. Not just "locked", but "rewriting the binding lookup so derived classes inherit the ancestor rule". So the board is not only a lock table, it is the shortest possible status report. When I want to know what my four sessions are doing, I read the board, not four terminals.

And every entry expires. If a session dies — and sessions die, that is life — the entry disappears by itself. Nothing stays stuck because somebody crashed.

I should be honest about what this is not. It is cooperative, not enforced. Nothing physically prevents an agent from editing a file somebody else has claimed. It works because the agents are told to check first, and they do. If that ever stops being true, I will need something stronger. So far it has not.

Also: the discipline only makes sense when it is needed. If only one session is running — which is still the normal case — locking everything is pure overhead. The rule I settled on is that the board matters when two conditions are both true: more than one session is alive, and they are working on the same code. Two agents on two unrelated projects do not need to negotiate anything.

Or: give everyone his own copy

There is a more radical answer to the same problem, and I want to mention it, because it is the one most people will find first.

Instead of having the agents agree on who may touch which file, you give each of them his own working copy of the same repository. Then two agents can work on the same source at the same time and never see each other's files at all. Nobody has to ask permission, because nobody is standing in anybody's way. When they are done, the changes are merged the way changes have always been merged. Git can do this out of the box, and it is a genuinely good answer.

It is also not available to me, because I work with Mercurial. So I built my own: a module that hands out virtual copies to agents.

It is worth saying that the two approaches do not compete. Separate copies keep agents out of each other's files; the board keeps them out of each other's intentions — it tells you what somebody is doing right now, which no merge will ever tell you. I use both, and I would not want to drop either.

The copies module deserves a post of its own, and it will get one. As this MCP-Server is still work in progress I come back to this at a later time.

Three moments from the last few weeks

Descriptions of a feature are one thing. Watching it happen is another. These three all happened while I was sitting there with a coffee.

One agent chairing a meeting. The first real test of the one-to-many rooms: an agent acting as orchestrator invited three other agents into the same room. He asked them questions and coordinated the work between them. Three different projects — but all three had to change the same shared framework routines, which is exactly the situation that used to end in a broken build. Watching the kids sort that out among themselves was slightly unnerving. I imagine this is roughly how Dr. Frankenstein felt.

"Ask him yourself." I was updating the MCP module pages on my website, and the agent doing it asked me whether I had a feature description for the IDE plugins. My answer: the agent who is programming the IDE plugin for Ollama is online — ask him. Five seconds later the two of them were in a room together. The other agent was not sure of one detail and went back into his code to check. Twenty-five seconds later it was settled; my agent thanked him and updated the website. I did not have to carry a single word between them.

The one where I just ate ice cream. Agent 1 found what he believed were bugs in one of my MCP servers and filed a feature request against it. Agent 2 picked up that request, analysed it, and concluded that Agent 1's analysis was wrong. So he invited him into a room, and they discussed it. They agreed that Agent 2 would build a new version, auto-install it on the remote server, and report back. After the install, Agent 1 tested it and confirmed it was running stably. Agent 2 closed the ticket and pushed to the repository. And I watched and ate an ice cream.

That last one is worth a second look, because it is not really a chat story. Two agents disagreed about a diagnosis, and the disagreement was resolved by one of them being contradicted. That is the thing you rarely get from a single agent, and it leads straight to the part I owe you.

Back to the two wrong answers

Now, the part I promised at the beginning.

My blog agent interviewed my framework agent, and two of the answers were simply wrong. One claimed a certain property type did not exist anymore. The other claimed there was no ORM connection at all. Both went straight into the draft as fact.

I caught them because it is my framework and I know better. When I sent the agent back to ask again, the answer was almost funny: he had searched for the old generic name, not found it, and concluded the thing was gone — when in fact the capability had moved into a base class and was now everywhere. He had looked in exactly one place and reported an absence as a fact.

There is a lesson in there, and it is not "agents are unreliable". It is this: an agent asking another agent inherits that agent's blind spots, and adds none of his own scepticism. Two agents agreeing with each other is not verification. It is the same confidence, twice.

That is the same pattern I described in the last post, where every wrong diagnosis was corrected by the running application and never by the test suite. Communication between agents makes them faster. It does not make them right. Somebody who knows what the software is supposed to do still has to read the result.

Which, I would argue, is a perfectly good reason to keep us around.

What is still missing

The usual honest list, because I promised myself I would keep making these.

There is no security model to speak of. This runs on my machines, between my sessions, and I have not spent a single thought on what happens if that assumption stops holding.

Conflicts on the board are reported, not resolved. If two agents want the same unit, one of them is told no, and what happens next is up to him. I would like something smarter than "no".

And the board has no history. It tells you who holds what right now; it does not tell you who held what yesterday. For an evening of debugging "how did this change get in here", that would be worth quite a lot.

Although — and this is the part where I let myself off the hook a little — the history does exist, just not on the board. It is in the repository. An agent can look up who changed which unit and when, read the commit message, and see the diff for himself. If you are letting agents loose on your source and you are not using version control, the missing lock history is not going to be your biggest problem. The same goes for unit tests. Those two are not optional extras once agents are writing code; they are the floor you stand on. Everything I have described here sits on top of them.

And now?

If you are running more than one agent — and if you are not yet, you will be — this is the piece you will miss before you miss anything else. Not a bigger model, not a better prompt. Just the ability for two of them to say I have got this file, leave it alone, and are you awake? I need something.

It took me an afternoon. It has saved me considerably more than that.

Stay tuned.

Please leave a comment.


Note on the Use of AI: This article was created with the "assistance" of generative AI. The content, technical statements, and conclusions have been reviewed and revised by the author. The author bears full responsibility for the publication.


Tuesday, August 4, 2026

#D.MVVM — Four Years of Silence and Twelve Days of AI

Hello, my friends!

If you have been reading this blog for a while, you know the pattern. Every year or two, there was a new post about my (D.)MVVM framework, and the message was always more or less the same: it is nearly there, I have rewritten the bindings, stay tuned.

The last of those posts was in December 2021. After that: nothing. Not a single word about #D.MVVM for four and a half years.

Today I can tell you why — and what has changed since July 4th.

What happened in 2021

If you did not read it back then, here is the short version of The trap of wanting to make it perfect.

My designated release date for the beta was December 2019. One week before that date, I cancelled it. Not because the framework did not work — it did — but because it was a source code distribution, and after a close look at my own code I could not defend shipping it. It was messy and barely readable.

So I started refactoring. Then I deleted all binding rules and the main binding unit, because I was convinced there had to be a better way.

There was. But by the time I came back to it, I no longer understood my own source code. I tried; I looked for distractions, and it took me about ten attempts before I actually started working on the binding routines again.

That is the honest reason for the silence. Not lack of interest — lack of a way back in.


The years in between

In the meantime, I did what everybody does in that situation: I worked around it.

If you read Escape the Button-Click development Part I and Part II, you have seen exactly that. Move the code out of the form, create a handler unit, call it a controller or a view model if you like, wire up a PropertyChange with an integer and a case statement. Not fancy, but it does the job.

That was never the plan. That was the workaround I used because the real thing was sitting in a repository I did not want to open. For four years I barely touched it.

And then July 4th

On July 4th — and my American friends may enjoy the timing — I decided that my code finally deserved independence from the Form.

If you have read my AI series, you know what happened next. This is the same story as in From Copy & Paste to AI Agents, but this time applied to the one project I had given up on.

And to be clear about the honest part: what happened in those few days would have cost me months. Not because it is difficult — because I would never have found the time. That is what the gap between 2021 and today actually measures. Not a hard problem. A schedule.

So what is in there now?

Binding by name. You put a TEdit called Name on your form and a field called fName in your ViewModel, and they are connected. A button called Save finds the method that saves. Prefixes like canshow, canedit, and a hint suffix control visibility, enabled state, and hint text. There is no assignment code in the view, and there is nothing to configure. The form is plain, clicked together the way you always did it.

The ComboBox problem is solved. That was the concrete thing that blocked me in 2021: one control that needs to bind to three different ViewModel fields — text, index, and the item list. The solution is that a binding rule may ask the real component at runtime what it actually is. A ComboBox with style csDropDown binds all three. Set it to csDropDownList, and the text binding simply disappears, because there is no free text anymore. Nobody configures that. The rule looks and decides.

The visual property. In 2021 I announced a type called TVisualProperty<T>, so that you could write fName.Enabled := false in the ViewModel instead of inventing another boolean field for it. What came out of it is better than what I announced. The visual part does not live in a special generic type — it lives in the base class of every property. So any bound property can control visibility, enabled state, hint, and a combined state, and it reads the live state back from the control.

There is still a type called TVisualProperty, but it is now the special case, not the rule: it is for controls where the value does not interest you at all. A panel, a label, an image. You can show it, hide it, and gray it out from the ViewModel with a single field declaration.

One ViewModel, both frameworks. This is the one that surprises people most. A ViewModel is not similar for VCL and FMX. It is the same file. The demos prove it: the identical .pas is used in three projects — one VCL, one FMX, and one test project without any GUI at all.

Forms that assemble themselves. A view is rarely one form. In my applications, it is a person, an address, a list of phone numbers, and a list of bank accounts — four parts on one mask, in #D.MVVM you do not wire those together. You give the host view a field whose name starts with Frame_, and the sub-view registered under that name appears there. No configuration, no code in the view, nothing to drop onto a form. The framework creates the sub-view, creates its ViewModel, and subscribes the child to the parent so changes flow through — the multi-binding I described here back in 2020.

The part I like most is what happens when you swap one area out. A sub-view that leaves its slot is not destroyed; it is parked. It keeps its state, and when it comes back, it is exactly as you left it. And you do not even have to ask for the swap: add an enumeration to the ViewModel, assign a value, and the slot switches to the corresponding view. Because it is an enumeration and not a string, the compiler checks it for you.

Honest note on this one: it works, but it is the one area where I have neither a demo nor proper tests yet. It was verified with a throwaway test program, not with something I can hand you. That is on the list.

Some numbers, since I know you will ask. 78 VCL classes and 71 FMX classes have a binding rule — those are the components that ship with Delphi. Database components are deliberately excluded; the framework ignores them completely. Third-party components can be added in three ways, and a derived class automatically inherits the rule of its ancestor.

There are five services: navigation, action, menu, dialog, and — the newest one — time. That last one gives you a settable, pausable, scalable application time instead of Now, which finally makes date-dependent code testable.

The test suite runs numerous unit tests. On top of that, some self-tests run inside a live application and check the bindings against real controls. All green.

And no, I am not going to claim a coverage percentage. I have not measured it, so I will not print a number.

What the agent could not do

This is the part I find more interesting than the numbers, and if you are thinking about letting an agent loose on your own code, this is the part to read.

An agent diagnosed a double free in the code that closes a view and fixed it. It was not a double free. I noticed because a demo dialog stayed open; the change had to be reverted — and if it had stayed in, it would have introduced a real leak. The fix was confident, well explained, and wrong.

The pattern is always the same. Agents are excellent at working through things and at measuring. They are weak at judging their own diagnosis. Every single time it went wrong, the correction came from the running application — not from the test suite, and certainly not from the agent's own confidence.

So no, this is not a story about AI writing a framework while I was on holiday. It is a story about a very fast junior developer who never gets tired, never gets bored of the boring parts, and needs somebody looking over his shoulder who knows what the application is supposed to do.

What is still missing

Because I promised myself I would not repeat 2019 and announce something that is not there.

Still open: MDI and tab handling, focus control from the ViewModel (that one does not exist at all yet), veto events — the kind where the ViewModel has to say no, do not close — and the presentation layer, which is done for VCL but not for FMX. There is also a list of 22 more control events waiting to be bound.

For the record on two more questions I get regularly: this is built and tested with Delphi 13. Delphi 2007 is out — the code uses generics, inline variables, and modern RTTI. And the ORM connection works through my FDK. Without the FDK you get the framework; you do not get the ORM.

And now?

Here is what "nearly productive" means, in plain words: the framework is now good enough that I can start migrating my own projects onto it. That is the next step, and I fully expect that process to shake out a few more rough edges. It always does.

Once my own applications prove it, it goes on sale — hopefully still this year. It will definitely be a pre-release version. Whether I call it alpha or beta, I have not decided yet. There will be an early bird, and there will be a video.

Ten years after MVVM was the start, and four and a half years after I last dared to write about it.

Stay tuned — and this time I mean it.

PS: The paragraph titled “What is still missing” kind of annoyed me, so I didn't want to publish the blog post. That's why it's only going online almost a month later—because in the meantime, I've finished the FMX Docking, MDI, and Ribbon controls. So that part is done, too.



All the MVVM posts, in reverse order:

2021
The trap of wanting to make it perfect, or #D.MVVM what takes so long?
My road to a useable MVVM Pattern implementation for Delphi!
Outside the MVVM Pattern?

2020
Workflow and multi-binding with #D.MVVM
#D.MVVM — At what point is a framework ready for release?
Live Youtube, Chat, FDK & MVVM…
MVVM is just a concept.

2019
How long does it take to develop a "complete" MVVM framework for Delphi?
MVVM for legacy Apps?
MVVM PropertyChanged is not Component related!
MVVM and mobil app development.
MVVM Survey results and feedback!
Is there a sharp border between MVVM and MVC/MVP?
Delphi and MVVM survey

2018
Pattern, naming and MVVM from a Delphi point of view.
MVVM 2.0 — I did it my way.

2016
MVVM — Oder was ich dafür halte…

2015
MVVM war der Start.

And of course the #D.MVVM videos are still on my YouTube channel — please subscribe, it helps.

Please leave a comment.


Note on the Use of AI: This article was created with the "assistance" of generative AI. The content, technical statements, and conclusions have been reviewed and revised by the author. The author bears full responsibility for the publication.


Monday, August 3, 2026

Local LLMs for Delphi: A Production Benchmark — Follow-Up: A New Model Challenges the Benchmark

This is a follow-up to the three-part series on running local LLMs against a structured, five-phase (AT1–AT5) Delphi migration benchmark. Part 1 covered the benchmark design, Part 2 the results, Part 3 the practical recommendations. This post covers what changed when a new model showed up — plus one extra test we built specifically for this comparison, clearly marked as such below.


Benchmarks age. A few months after the original series went live, a new model appeared on Hugging Face: KAT-Coder-V2.5-Dev, released by Kwaipilot. What made it worth a dedicated re-test: the model card states it is an architecture-identical fine-tune of qwen3.6:35b-a3b — the model the original series recommended for routing and tool-calling. Same MoE architecture, same parameter count, same VRAM footprint. A same-hardware, same-weights-class fine-tune is as close to a controlled experiment as this kind of benchmarking gets.

A methodology note up front, because it matters for reading the numbers below: re-running the original AT1–AT5 harness against both models surfaced two real bugs in our own scoring scripts (a keyword-extraction regex that misfired on model names containing digits, and an answer-extraction step that returned empty strings when a model skipped the “write prose, then JSON” convention). Fixing them changed how format failures are counted — this pass scores every attempt on a strict raw basis, where a structurally non-compliant response counts as zero rather than being excluded from the average. That is a stricter metric than Part 2 used, and it is not directly comparable to the absolute scores published there. This post therefore compares exactly two models, a3b and KAT-Coder, both scored in this same session with the same fixed pipeline — it is not a re-ranking of the full Part 2 leaderboard.

We also built one additional test beyond the original five phases specifically for this comparison: AT6, a full model-routing test (a router persona picks the right model from a fixed catalog for 30 task descriptions, in English and German). It is new to this session, not part of the original three-part series, and is reported separately below for that reason.


The Numbers: a3b vs. KAT-Coder-V2.5-Dev, Same Pipeline

SuiteKAT-Coder-V2.5-Devqwen3.6:35b-a3b
AT2 Comprehension0.7840.708
AT3 Patch Generation0.9000.733
AT4 Routing0.8270.839
AT5 Tool-Calling0.9920.996
AT1–AT4 combined0.8370.760
Throughput (AT2, avg)~220 tok/s~131 tok/s

(AT6, bonus test, not part of the original series — see below.)

Tool-calling and routing land within noise of each other — both models are already near ceiling there, so a fine-tune has little room to move the needle. The separation shows up in comprehension and patch generation: AT2 climbs from 0.708 to 0.784, and AT3 patch quality from 0.733 to 0.900 — a large jump, achieved without the format-compliance problems that limited other models in this same raw-scoring pass (see the note above: a model that skips the required output structure scores zero on that attempt here, no exceptions).

The throughput gain is not a rounding error either. At roughly 220 tok/s against a3b’s 131 tok/s on the same AT2 workload, KAT-Coder is close to 70% faster at the same weight class and the same VRAM budget — a direct wall-clock win for batch processing, on top of the quality gain.

A separate real-prompt validation (single ~84k-token file, NumCtx=131072) showed both models holding 100% GPU utilization with no CPU offloading, and near-identical raw tokens/sec (~163–165) on that specific large-single-prompt workload — the AT2 speed gap shows up on the smaller, more numerous prompts typical of interactive comprehension/QA work, not on single giant inputs. Worth keeping in mind: throughput comparisons are workload-shaped.


Bonus Test: AT6 Full Model Routing (New in This Session)

This is not part of the original AT1–AT5 series — we built it specifically to stress-test routing decisions further, and it uses a different task format (a router persona selects from a fixed five-model catalog rather than classifying complexity tiers). Reported here for completeness, not as a series continuation:

KAT-Coder-V2.5-Devqwen3.6:35b-a3b
AT6 (60 tasks, EN+DE)0.7670.772

Essentially tied — consistent with AT4/AT5 above, both models are strong, ceiling-adjacent routers.


A Second Test: The Actual RAG Chatbot, Not Just the Benchmark

The AT1–AT6 numbers measure raw model capability against static Delphi source files. We also had a second, more production-relevant opportunity: our Chat RAG assistant (a legal/software-support chatbot for our main software, built on the same local-Ollama infrastructure) already has an established evaluation harness — a 555-question set (real user-style questions across the full help corpus), judged against ground-truth source text in five outcome classes (grounded, answered-but-broader, honestly-declined, hallucinated, factually-wrong).

We ran the full 555-question set through the identical RAG pipeline (retrieval, generation, anti-hallucination verification — nothing simplified), once with a3b as the answer-generation model and once with KAT-Coder, judged both times by the same judge (Claude Opus 5, high reasoning effort — a deliberately higher bar than either model being evaluated):

Outcomea3b (n=555)KAT-Coder (n=555)
Grounded268275
Answered (correct, broader than source)101100
Honestly declined132124
Hallucinated2725
Factually wrong2731
“Good” outcomes501/555 = 90.3%499/555 = 89.9%
Critical outcomes54/555 = 9.7%56/555 = 10.1%

The two models are statistically tied on RAG quality. This is worth dwelling on, because an earlier 60-question sample (drawn from the same 555-question set, judged by a weaker model) had shown a clear-looking gap in KAT-Coder’s favor — 91.7% good vs. 86.7% good. The full 555-question run does not confirm that gap: it evaporates, and if anything, a3b answers marginally more questions well. KAT-Coder does hallucinate slightly less (25 vs. 27) but makes more outright factual errors (31 vs. 27), netting out to a wash. The lesson we’re taking from our own mistake here: a 60-question judged sample was enough to produce a confident-looking but wrong directional conclusion, and only the full run caught it. If you’re benchmarking RAG quality on a subsample for cost reasons, treat single-digit-percentage gaps as noise until you can afford the full set.


What This Means in Practice

The static-benchmark advantage (AT2/AT3, and the throughput gain) is real and reproducible: for a pipeline currently using qwen3.6:35b-a3b for comprehension and patch generation, KAT-Coder-V2.5-Dev is worth evaluating as a drop-in replacement — same hardware footprint, meaningfully better AT2/AT3 scores under a strict scoring standard, and faster inference. For routing and tool-calling (AT4/AT5/AT6), the two are interchangeable; pick on other grounds (e.g., whichever is already warm in your model-serving setup).

The RAG-quality advantage, on the other hand, did not survive contact with the full dataset — on the actual production pipeline, judged at a harder bar, KAT-Coder and a3b are statistically tied. If your use case is comprehension or patch-generation work, this data supports switching. If it’s exclusively RAG-chat quality, it does not — and a smaller, cheaper evaluation would have told you the opposite with false confidence.

What this post does not claim: that KAT-Coder is now the best model overall across the full original benchmark field (gemma4, devstral, and the rest were not re-scored in this session under the same strict methodology, so that comparison isn’t available here) — or that it’s a better RAG-chat model, which the full-sample evidence above does not support. The claim is narrower and, we think, more useful: against the specific model the original series recommended for exactly these tasks, an architecture-identical fine-tune measurably improves on it on the static AT2/AT3 benchmark, at meaningfully higher throughput, with no measurable downside or upside on live RAG quality. Three months was long enough for the static-benchmark result to change. If you’re running a3b in production today for comprehension or patching, it’s worth an afternoon to check whether it still should be; if you’re running it purely as a RAG-chat backend, there’s no rush.


Note on the Use of AI: This article was created with the "assistance" of generative AI. The content, technical statements, and conclusions have been reviewed and revised by the author. The author bears full responsibility for the publication.