Local LLMs for Delphi: A Production Benchmark — Part 2: What the Numbers Reveal

Part 1 described the setup: eight local models, five benchmark phases (AT1 through AT5), 30 real production Delphi units. The test domain was legacy code migration — but the phases themselves cover tasks that appear in any Delphi LLM pipeline: code analysis, comprehension, patch generation, routing, and tool calling. This post covers what the numbers showed.

The Scoreboard

Local models (Ollama, 32 GB VRAM):

Model	AT1	AT2	AT3	AT4	AT5	Overall
gemma4:26b	0.96	0.62	0.88	0.84	0.99	0.82
qwen3.5:27b	0.86	0.69	0.77	0.85	1.00	0.79
qwen3.6:27b	0.82	0.70	0.70	0.81	1.00	0.76
qwen3.6:35b-a3b	0.79	0.64	0.70	0.84	1.00	0.74
devstral-small-2:24b	0.70	0.58	0.71	0.80	1.00	0.70
qwen2.5-coder:14b	0.68	0.50	0.69	0.79	n/a	0.66

Cloud baseline (Anthropic API):

Model	AT1	AT2	AT3	AT4	AT5	Overall
Claude Sonnet 4.6 ☁	0.983	0.983	0.955	0.994	1.00	0.983

---

Finding 1: Comprehension and Patch Quality Are Not Correlated

gemma4:26b has the lowest AT2 score (0.62) and the highest AT3 score (0.88). A 26-point gap across two phases that test adjacent capabilities is not noise. qwen3.6:27b is the opposite: highest AT2 (0.70), mid-table AT3 (0.70). The two skills are measuring something genuinely different — not just two views of the same underlying "code understanding."

---

Finding 2: AT3 Failure Modes

Format refusal (gemma4:26b)

70% of gemma4's AT3 responses returned as natural-language prose — no JSON block. The 30% that did comply scored 0.88 on content — the best in the benchmark. This is an instruction-following problem, not a capability problem. Fixable with better prompting.

Systematic token corruption (qwen2.5-coder:14b)

Every generated code line was prefixed with L<n>: — e.g. L29: fLogEnabled : Boolean;. Not valid Pascal. Consistent across all 30 units. Content often correct, output systematically malformed.

Structural misplacement (qwen3-coder:30b)

Asked to add a method, the model inserted code into the Initialization block — runs once at startup instead of being callable. Compiles, passes syntax check, fails at runtime. Would not be caught without unit tests.

---

Finding 3: Speed — The MoE Advantage

Model	tok/s	Architecture
gemma4:26b	~170	Dense
qwen3.6:35b-a3b	~131	MoE
qwen2.5-coder:14b	~120	Dense, 14b
qwen3.5:27b / qwen3.6:27b	~28–29	Dense, 27b

For a batch run over 30 units, 170 tok/s vs 28 tok/s is measured in hours of wall clock time.

---

Finding 4: Tool Calling — A Binary Split

Models that returned proper tool_calls objects: devstral, qwen3.5:27b, qwen3.6:27b, qwen3.6:35b-a3b (1.00 each), gemma4:26b (0.98).
Hard fail at the API level: qwen2.5-coder:14b, qwen3-coder:30b (text output only), deepseek-r1:8b (Ollama error: "does not support tools").

This is not a prompting problem — it is an API-level capability. A model either returns tool_calls objects in the response or it does not. For any pipeline that relies on IDE or MCP integration, this is a hard filter applied before any other consideration.

---

Finding 5: German vs English Prompts on AT1

Model	AT1 EN	AT1 DE	Delta
gemma4:26b	0.96	0.83	-0.13
qwen3.5:27b	0.86	0.78	-0.05
qwen2.5-coder:14b	0.68	0.39	-0.31

Every model scores lower in German. The losses range from 5 to 31 points. Recommendation: Use English for any prompt that includes source code or asks the model to name specific identifiers. Use German for everything that faces the developer.

---

Finding 6: Cloud vs Local — Where the Gap Actually Is

Phase	Best local	Sonnet 4.6	Delta
AT1 Code Detective	0.96 (gemma4)	0.983	+0.02 ≈ 0
AT2 Comprehension	0.70	0.983	+0.28 ← main gap
AT3 Patches	0.88	0.955	+0.08
AT4 Routing	1.00	0.994	≈ 0
AT5 Tool Calling	1.00	1.00	0

Local models are essentially on par with Sonnet in code extraction, routing, and tool calling. The comprehension gap (28 points on AT2) is where cloud-scale models are meaningfully better. For GDPR-constrained teams, local-only is viable. For hybrid architectures, route comprehension to Sonnet and keep everything else local.

---

Finding 7: AT4 Routing — Not All Models Are Reliable Routers

Routing accuracy is the capability that matters most in a multi-model pipeline: a misclassified task gets sent to the wrong model, which either wastes compute on an oversized model or produces a failure on an undersized one. The AT4 results show a meaningful spread across the field.

The top scorers — qwen3.5:27b (0.85), gemma4:26b (0.84), and qwen3.6:35b-a3b (0.84) — differ significantly from the problem case: qwen3-coder:30b at 0.71. In a pipeline of 90 routing decisions, a 0.71 accuracy means roughly 26 tasks misclassified. That is not recoverable through prompt engineering — classification drift at that scale corrupts the batching architecture that the pipeline depends on.

Speed is the second dimension here. Routing sits on the critical path of every task — every request passes through the router before any work begins. A model that routes accurately but slowly adds latency multiplied across the entire batch. qwen3.6:35b-a3b combines solid routing accuracy with 131 tok/s, making it the only model that is both reliable and fast enough to use as a production router.

---

VT Pre-Test Findings

Before the five AT phases, every model ran through seven validation tests (VT1–VT7) covering JSON compliance, instruction following, output consistency, basic Delphi understanding, and native tool-call support. All eight models passed every pre-test — this confirmed that the AT results reflect genuine capability differences, not basic reliability failures.

Two VT findings produced results specific enough to be worth documenting alongside the main benchmark.

VT6: Migration Risk Detection

VT6 tested whether models could identify concrete migration traps across three categories of Delphi code: ShortString serialization risks, binary-packed record layouts, and call-chain dependencies. The chart shows significant variation between models, even where their AT1–AT5 scores are similar — VT6 captures a targeted detection capability that the main phases only partially measure.

VT8: Token Budget and Thinking Mode

VT8 exposed a practical deployment trap: when a 300-token output budget is set, and thinking mode is active, all three tested Qwen models consume the entire budget on internal reasoning and return an empty response. The correct answer exists — the model knows it — but there is no token budget left to output it. Setting num_predict ≥ 800 for any thinking-enabled model, the issue is completely.

What the Numbers Add Up To

Seven findings point to a consistent picture. Comprehension and patch generation are genuinely different skills — the best model at one is not the best at the other. Speed varies by a factor of six between the fastest and slowest capable models. Tool calling eliminates three models outright at the API level. Language choice in prompts can cost 31 percentage points on code extraction tasks. Routing accuracy is not guaranteed — one model in the tested pool is too unreliable to use as a router. And the cloud comprehension gap, while real at 28 points, is the only phase where local models fall meaningfully short.

None of these findings argues for a single best model. They argue for a pipeline that routes different task types to different models — which is exactly what Part 3 covers.

---

Part 3 covers the practical takeaways: routing architecture, model selection by task, and the hybrid pipeline decision table.

Local LLMs for Delphi: A Production Benchmark — Part 1: Design and Methodology

Let me save you some time upfront: this is not a post about prompting ChatGPT to help with Delphi. This is about running local, on-premise LLMs — models that never send your source code to any server — through a structured, five-phase benchmark built from 30 real production files.

The test domain is a large legacy codebase undergoing Unicode migration — demanding enough to expose real capability gaps across code analysis, patch generation, routing, and tool calling. The findings apply broadly: if you are integrating local LLMs into any Delphi workflow, this benchmark tells you which models handle which tasks, which ones fail at the API level, and where the architecture decisions actually matter.

The Problem Is Bigger Than It Looks

The codebase in question is large. The production compile base is Delphi 2007, partially migrated to a modern Delphi version. What does legacy here actually mean? ShortStrings used for binary serialization. Records with fixed memory layouts that must not change because data was written to disk in that exact layout twenty years ago. No MVC, no MVVM — a custom control hierarchy that grew organically before those terms meant anything in the Delphi world. The kind of codebase where a seemingly innocent String vs ShortString swap can corrupt a file format three layers down.

A cloud AI service is not an option. The code is proprietary, the clients are sensitive, and the answer to "does your code leave the building?" must always be no. So the question became: can local LLMs running on our own hardware actually help, or are we on our own?

The Hardware Reality

One GPU with 32 GB VRAM. An Ollama server reachable at a local network address. We tested eight models:

Model	Parameters	Architecture
qwen2.5-coder:14b	14b	Dense
devstral-small-2:24b	24b	Dense
qwen3-coder:30b	30b	Dense
qwen3.5:27b	27b	Dense
qwen3.6:27b	27b	Dense
qwen3.6:35b-a3b	35b (MoE)	Mixture-of-Experts
gemma4:26b	26b	Dense
deepseek-r1:8b	8b	Dense

Models That Didn't Make the Cut

Six models failed pre-screening and never entered the main evaluation:

Model	Reason for rejection
granite4:32b-a9b-h	Only 16% GPU utilization — 84% CPU offloading. Unusably slow.
nemotron-3-nano:30b-a3b	Zero valid JSON responses across 120 validation attempts.
llama4:scout	Failed instruction following and basic code output tests.
qwen3-coder-next:latest	Failed instruction following and basic code output tests.
deepseek-r1:32b	Failed validation tests.
deepseek-r1:14b	Failed validation tests.

Why a Five-Phase Benchmark?

What would a real Delphi LLM assistant actually need to do? Whether the task is migration, refactoring, documentation, or IDE integration, the core capability requirements are the same:

• Find hidden problems before you touch anything
• Understand what a unit actually does, not just what it says it does
• Write correct Delphi code, not plausible-looking pseudocode
• Know when a task is trivial and when it needs a more capable model
• Call tools — interacting with an IDE or AST system rather than outputting text

Those five capabilities became five test phases, applied to 30 carefully selected production units.

---

The Production Context

This benchmark was not designed in isolation. It was built to answer a concrete engineering question: which local models can reliably handle which classes of work in a production-grade, on-premise AI pipeline?

That pipeline consists of two components developed in parallel with this benchmark work.

ProxyMCP is a lightweight routing layer that sits between IDE tooling and the model backend. It implements the Model Context Protocol, accepts tool calls from development environments, and forwards them — without transforming the payload — to the backend layer that handles execution.

Enterprise Server is the orchestration layer behind ProxyMCP. It handles model routing, task classification, session management, and audit logging across multiple users and projects. It is designed for on-premise deployment — no data leaves the local network — and built to the requirements of teams that need traceable, reproducible AI-assisted workflows with defined service levels.

The routing architecture described in Part 3 of this series is not a theoretical recommendation. It is the architecture these components implement. The benchmark determined which models fill which roles.

---

Phase AT1: Code Detective Work

Each of the 30 units contains one non-obvious fact — something you can only find by genuinely reading the code. Example:

Unit: AsyncSettings.pas
Question: Which fields of TAsyncSettings are shared across every instance rather than per-instance, making the class effectively a singleton without the type system saying so?
Required: Name the specific identifiers.

Phase AT2: Comprehension (120 Tasks)

Each unit generates four tasks: one unit-level summary and three focused questions. Scored by Claude Opus 4.7 as judge using HIT/PARTIAL/MISS grading.

Phase AT3: Patch Generation

Each unit has one realistic migration task. The model must output a structured JSON patch — insert after line N, replace lines N through M, delete lines N through M. Maximum three operations:

{
  "operations": [
    {
      "op": "insert_after_line",
      "line": 35,
      "content": "      Class procedure WaitForCompletion(aTimeoutMs : Cardinal);"
    }
  ]
}

Phase AT4: Routing (90 Decisions)

30 units × 3 tasks each. Classify as local (trivial), mid (moderate), or top (deep understanding required). A model that consistently over- or under-classifies is useless as a router.

Phase AT5: Tool Calling (73 Tasks)

Can these models use Ollama's native tool_calls API? This is not about prompting a model to output JSON — it is about whether the model returns actual tool_calls objects in the API response. Several otherwise capable models fail here completely.

---

The Judge and the Baseline: Two Roles for Claude Models

This benchmark uses Claude models from Anthropic in two completely separate roles — and it is worth being explicit about this, because the distinction matters for how you read the results.

Model	Role	What it does
Claude Opus 4.7	Judge & Gold Standard Author	Reads each of the 30 source files independently, writes the gold standard answers, then scores every model response as HIT / PARTIAL / MISS. Does not participate as a candidate.
Claude Sonnet 4.6	Cloud Baseline Candidate	Runs the identical five-phase benchmark as the local models — same questions, same tasks, same scoring. Acts as a reference point: how much better does a capable cloud model actually score?

Opus is the most capable model in Anthropic's lineup. Using it as the judge — rather than a fixed rubric or human annotators — means the gold standard is set by the model with the deepest understanding of the source material. Sonnet then competes against local models on the same playing field, graded by the same judge.

Is it a conflict of interest that the judge and one of the candidates come from the same company? It is worth noting. In practice, Opus's judgments on Delphi code analysis were consistent with what a senior developer would recognize as correct — the gold standard answers were not padded to favor any particular model family. And the local models' scores speak for themselves: gemma4:26b reaches 96% on AT1, within two points of Sonnet. A biased judge would not produce numbers like that.

---

A Note on the Cloud Baseline

After completing the local evaluation, we ran the same five-phase benchmark on Claude Sonnet 4.6 via the Anthropic API — not to re-open the DSGVO argument, but to establish a reference point. The short version: local models are essentially at parity with Sonnet on code extraction, routing, and tool calling. The comprehension gap (AT2) is real and significant — 28 percentage points between the best local model and Sonnet. Whether that gap matters depends on whether you need the model to explain code or just to find things in it. Part 2 covers this in detail.

---

Part 2 covers the actual results: which models performed where, the failure modes, and what surprised us. Part 3 covers the practical takeaways — routing architecture and model selection by task type.

Is Ai changing our daily work?

Why do you use AI? That is the question I am currently asked most often.

And the next statement that usually follows right after that question is: I would not even know what to use AI for.

The answer “for everything” is obviously too broad, even though it is true. But let me break it down in more detail. The answer falls into three areas:

1. Daily work: reading and processing Jira tickets, answering emails, and writing letters.

2. Project maintenance: programming new features, fixing bugs, or improving existing parts of the software.

3. New projects: either a helpful tool or a completely new project.

Of course, I can read my emails by hand, and perhaps my spam filter is good enough. But how do I separate customer bug reports from the hundreds of other emails that land in my inbox every day?

But first, let me look at the industry from this perspective.

I assume that you have been programming for quite some time and have managed perfectly well without AI until now. So why should you bother with AI?

The industry and the role of a software developer are changing. And I am not talking about new projects being written in FMX instead of VCL so they become cross-platform. I am also not talking about finally migrating your old database routines to FireDAC.

I am talking about the biggest shift since the move from DOS to Windows. This time, however, it is not a technical software shift. It is a shift in what people expect from you.

I hardly believe that anyone will still program without AI in the future. And even if you do, the company you apply to will expect you to know how to use AI.

I can already hear the comments: “I am my own boss!”

Yes, but the company you have been competing with for years — sure, their software was never as good as yours — is now hiring developers whose entire day consists of sending clever prompts to Codex, Claude, and others. And they will use those tools to turn the software that used to lag behind yours into a product that is superior to yours in every respect.

One or two years ago, that would have been unthinkable. It would have required a lot of money and many developers. Today, it costs only a fraction in token costs.

And that is exactly where the problem lies.

As developers, we are no longer competing only with other developers. If we write code by hand today or search for a bug manually, it simply takes much longer than having an AI generate 1,000 lines of code and a new unit in one go — naturally including the corresponding unit tests for which we supposedly never had time.

We no longer have to spend three hours googling a Windows API call or searching through three different forums. We describe the problem, maybe add a hint about where it should be used, press Return, answer two more questions, and go to lunch.

When we come back, the call has been implemented, the unit test is green, the changelog has been written, and the update has been committed. And this time with a truly detailed explanation — not just the usual “API call done.”

Last year, people said: AI replaces a junior programmer, but costs per year what a junior costs per month.

I think that statement is already outdated — just like last year’s model versions.

But what about code reviews or bug fixes? Should I really review the code written by Claude and others? Even if the unit tests are green?

Everyone has to answer that question for themselves. If the new method controls a nuclear power plant or autonomous driving, then perhaps yes.

Personally, I tend to look more closely at the unit tests to make sure the AI did not simply write the test in such a way that it turns green. I think this is where the wheat is separated from the chaff. Only if the unit tests are good can I trust the code. TDD — test-driven development — is practically mandatory.

And then we arrive at an important point.

The unit tests are green and verified. The commit also looks good. But what if I do not actually understand the code?

Maybe I ask the AI to explain the code to me or add comments. Or maybe I do not care.

But what happens when the AI services are currently unavailable? What if the code contains a bug that the agent does not find?

When Claude was unavailable for several hours, I felt a disturbance in the Force, even though I was not at my PC. Thousands of developers were staring at a screen with an error message and suddenly no longer knew what to do.

So it is not all that simple, and we need to be aware of where we are heading.

I call it the Titanic problem.

Sure, this route is faster. But what happens if an iceberg suddenly appears?

What happens when an entire industry becomes addicted and then the drugs — AI — simply become more and more expensive?

Who will still be able to afford the tokens if the costs multiply?

A good current example — current meaning for about four weeks — is the new Opus 4.7 model. The token prices may have stayed the same, but the model needs 1.3 times more tokens per query than before.

And, of course, the thinking tokens, which conveniently are not shown in the output, have also multiplied by several factors.

The price has not increased yet. But with the same budget, I do not even get half as far as before.

So not everything that glitters is gold.

But hey, it is fun. And you can finally have the things programmed that you never had time for — or simply did not know how to build.

Happy Vibe Coding.