Play someone a transcript of a good voice agent and they're impressed. Put them on the phone with the same agent and they hang up, because there was an 1,800ms gap before it answered, and for 1,800ms of silence the human brain concludes the line is dead.
Natural conversation turns over in roughly 200–300ms. Anything past about 800ms reads as broken. That constraint, not reasoning quality, is what makes voice hard.
Where the time actually goes
A naive pipeline stacks four sequential costs:
- End-of-speech detection (waiting to be sure the caller finished): 200–500ms
- Transcription (audio to text): 100–300ms
- Reasoning (the model deciding what to say): 400–1,500ms
- Speech synthesis (text to audio, first byte): 150–400ms
Run those in sequence and you're at two seconds before anyone speaks. Every fix below is about overlapping them or removing one.
Stream everything, wait for nothing
Transcribe while the caller is still talking. Start generating while transcription finalises. Start synthesising the first sentence while the model writes the second. Send audio to the caller as it's produced, not when the response is complete. Done properly this collapses two seconds to under 600ms without changing a single model.
Tune the endpoint, then tune it again
End-of-speech detection is the most underrated dial in voice. Too aggressive and you interrupt people mid-sentence, the single most infuriating agent behaviour. Too conservative and every turn drags.
The trick is that it shouldn't be one setting. After a yes/no question, a short silence is enough. After "tell me what happened", give them room. Adjust the threshold based on what you just asked.
Let it be interrupted
When the caller starts talking over the agent, the agent stops immediately, mid-word if necessary, and the system keeps track of how much of the response was actually heard. Without that bookkeeping, the agent will later reference something it never got to say.
Take slow work off the critical path
A database lookup mid-sentence is dead air. Two approaches, both used together:
- Prefetch. The moment the call connects, pull the caller's record from the number. By the time they've finished saying hello, you have their history.
- Cover the gap. If a lookup genuinely takes a second, say something first: "Let me pull that up." Humans do exactly this, and it converts dead air into normal conversation.
Do not try to be faster than a human at every step. Try to fail the way a human fails, with a filler word, not with silence.
Keep the reasoning small
Voice is the wrong place for a large model doing open-ended thinking. Keep the conversational path on a fast model with a tight prompt and a narrow task. Route the rare complex case to a slower model, and tell the caller you're checking, or transfer to a person.
Design the handoff before launch
The agent should transfer when the caller asks for a human, when it's twice failed to understand, when the topic is outside scope, or when sentiment turns sharply negative. A warm transfer that passes a summary to the human is the difference between "that was efficient" and "I had to explain everything twice".
What to watch once it's live
Track time-to-first-word per turn at p50 and p95, interruption rate, containment rate, transfer rate and average handle time. Then, and this is the part teams skip, listen to ten real calls a week. The metrics tell you the system is healthy; the recordings tell you it's awkward.
Put this to work
Want this built, not just read?
Tell us the workflow that's eating your week. Thirty minutes, no deck.