The obvious voice agent architecture was the slower one

If you read anything about building a multi-skill agent, you will be told to split it into specialists behind a router. The reasoning is sound and easy to repeat: a model choosing from four tools makes better choices than a model choosing from sixteen. Give each specialist a narrow job, put a router in front, and accuracy goes up.

I built that. Then I measured it against the thing it was supposed to beat, a single agent holding every tool, and the split lost.

What the numbers said

	specialists	unified
eval score	10 / 10	10 / 10
first turn	4993 ms	3217 ms
turns after	2180 ms	2853 ms
first-turn penalty	2.3x	1.1x

The accuracy argument, the entire reason for splitting, did not show up. Both architectures scored the same on the eval suite. What did show up was a 1.8 second penalty on the first thing the caller says.

That penalty has a simple cause. The router is a model call. Before the agent can look anything up, it spends a full round trip deciding which specialist should handle the request. On the turns after that, the conversation stays with the chosen specialist and the shorter tool menu pays off slightly, which is why specialists are faster from turn two onwards.

Routing is paid once per call, not per turn. That sounds like an argument in its favour until you remember which turn it lands on. Nobody abandons a call on turn three. They abandon it in the first few seconds, listening to silence.

The failure modes I did not predict

The latency was measurable and I expected some cost. What I did not expect was that splitting the agent created two failure modes a single agent cannot have.

A capability reachable from only one specialist. A caller asked to correct the spelling of their own name. That tool lived with one specialist, the caller was talking to another, and instead of being helped they were passed around. With one agent holding every tool, this failure is not expressible.

Two agents handing a request back and forth. When a request sat between two specialists and neither could fully serve it, they took turns deciding it belonged to the other. The caller gave up before they did.

Both of these are structural. They are not bugs I could fix with a better prompt, because they come from the shape of the system rather than the wording inside it.

What did not help

While I had the measurement harness out, I tried the things people usually try.

Switching models. Gemma was the fastest tested. Gemini Flash Lite and GPT-4.1-mini were both slower. The spread between them was much smaller than the spread between architectures.

Shortening prompts. A three word prompt takes the same time as a 1,200 token one. Prompt length is not where the time goes, which surprised me enough that I ran it twice.

Where the time actually goes

The dominant cost on this deployment is network distance to the inference endpoint, roughly 1.5 seconds per model round trip from Tashkent. Tool execution is 4 to 7 milliseconds.

That ratio is worth sitting with. The code I wrote accounts for well under one percent of the time a caller waits. Every optimisation I could make inside my own service is noise against a single round trip. The only changes that matter are the ones that remove a round trip or overlap it with something else, which is why preemptive generation and a lower endpointing delay help and micro optimising a tool handler does not.

What I kept

The unified agent is the default now. The split is still in the codebase and still tested, because the accuracy argument for it is not wrong in principle, it just does not bite at a dozen tools. If the tool count grows well past that, this is worth measuring again.

Both modes share their tool definitions. The unified agent inherits them from the specialists, so a tool added to one appears in the other and a test fails if they drift apart.

The part that generalises

I did not discover that routers are bad. At a different tool count, on a different network, with a different latency budget, the split may well win.

What I would take from this is narrower and more useful: the architecture advice was about accuracy, my actual constraint was latency, and nobody told me that because nobody knew my constraint. The eval suite is what let me see that the accuracy argument had not materialised, and the latency script is what let me see what had. Without both, I would have kept the specialists, because they were more interesting to build and I had already written them.

That is the part worth guarding against. The design you have already built is the one you are most likely to defend.