A cross-linguistic study of everyday talk pinned the average silence between one person finishing a sentence and the next person starting at roughly 200 milliseconds, from Japanese speakers who overlap by a hair to Danish speakers who wait a beat longer. Human turn-taking runs on a clock most software can't match. And that clock is the whole reason a voice agent that answers in 900ms feels like a conversation while one at 1.4 seconds feels like a call to tech support.
Latency is the topic every buyer thinks they understand and most get wrong in the same handful of ways. What follows is a teardown of those assumptions, one at a time, for anyone shipping a voice agent or paying the invoice for one.
Latency Is a Stack of Numbers, Not One
When someone says "our agent responds in 800ms," they almost always mean model latency alone. A real call runs through five or six layers before a caller hears a word, and each one is billed and measured differently. The per-layer view is what the Phony.ai discussion of the Itemised Receipt: Why Your AI Phone Bill Is Lying to You walks through, and it's the frame worth carrying into the rest of this piece.
Add those layers together and the median production system lands somewhere buyers rarely see on a slide deck. One analysis of live voice agents puts real-world median response time at 1,400 to 1,700ms, which is exactly the range where callers start describing the agent as slow or hard of hearing.
Faster Usually Isn't Better If the Agent Cuts You Off
The instinct is to drive every number down. In practice, aggressive endpointing produces an agent that starts answering while the caller is still finishing their sentence, and that's worse than a slow reply. Callers will forgive a brief pause. They won't forgive being talked over, especially while they're reading out an address or a policy number.
Good barge-in design is the flip side of the same problem. When the caller interrupts, the agent has to stop talking within a couple of hundred milliseconds and actually listen, not finish its sentence first. Half the "latency" complaints in the wild are barge-in complaints in disguise: the agent heard the caller but kept going, the caller repeated themselves, and the transcript is now a mess.
The Model Is Rarely the Bottleneck
Teams instinctively blame the language model and go shopping for a faster one. Frontier models have gotten fast enough that this is usually the wrong lever. OpenAI reports GPT-4o can respond to audio in as little as 232ms, well inside human range. If your calls still feel slow, the model isn't what's costing you the second.
The usual culprits are telephony round-trips, an STT pass that waits too long to declare end-of-speech, and a TTS layer that buffers a full sentence before playing anything. Swap in a faster model and you might save 150ms. Fix streaming end-to-end and you save 600.
Transfers and Escalations Belong in the Same Budget
Latency isn't only about how fast the agent talks. It's also about how long the caller waits when the agent hands off. A cold transfer that dumps the caller into a queue with no context is a latency failure even if every turn before it stayed under a second.
A warm handoff, where the agent passes a short summary and the caller's stated reason to the human picking up, adds a second or two to the transfer itself but removes the thirty seconds of "can you tell me again what this is about" on the other side. Design the whole path, not only the turns the agent handles.
Compliance Turns Into a Latency Problem Faster Than Buyers Expect
Every automated call carries a compliance surface: whether the agent has to identify itself as AI, whether recording requires one-party or two-party consent, whether the number was permissioned to dial at all. These vary by jurisdiction and by call type, whether outbound to a cold list, inbound to a published number, or a callback to a form fill, and the rules keep moving.
The latency angle is subtle. A disclosure the agent has to read at the top of the call eats three or four seconds before the caller can say why they rang. A recording notice tacked on after the greeting adds another two.
Buyers who obsess over shaving 100ms off the model turn and then bolt a fifteen-word compliance preamble onto every call have optimized the wrong thing. Write the disclosure tight, put it in the greeting, and measure the whole opening as one budget.
The Invoice Hides Where the Seconds Actually Went
A blended per-minute rate tells you what a call cost. It doesn't tell you which layer ran the clock. Telephony and speech-to-text bill by the second the call is live; the model bills by tokens per turn. A call that felt slow and a call that felt fast can land at the same per-minute number while spending their seconds in completely different places, which is why per-layer receipts matter more than headline pricing.
One reason to care: the layer that dominates cost is usually the layer that dominates latency too. Fix one and you tend to fix the other. Optimizing without the receipt is guessing.
What to Actually Measure
Ship those four numbers on every call and the conversation with your provider changes. Ship a blended average and you'll spend another quarter chasing the wrong second.