The difference between a voice agent that feels alive and one that feels like software is roughly three hundred milliseconds. Miss the budget and users start finishing your sentences — for you.
The target: 800 ms end-of-turn to first audible token
Human conversation runs on a turn-taking clock of two hundred to five hundred milliseconds. Anything under eight hundred milliseconds reads as attentive. Anything over a second reads as buffering. That single number is the north star.
The budget, line by line
- Voice activity detection — 80 ms to decide the user stopped talking.
- ASR finalization — 120 ms to lock the last few words.
- Tool routing — 60 ms of parallel dispatch, not sequential thinking.
- LLM first token — 260 ms with streaming, aggressive caching, and small context.
- TTS first byte — 180 ms with a streaming voice.
- Network jitter buffer — 100 ms of margin so nothing pops.
Buying back time
Predictive endpointing
Do not wait for a firm silence to start thinking. As soon as prosody hints the user is winding down, kick off the LLM speculatively on the partial transcript. Cancel if you were wrong.
Warm sockets, warm models
Keep a pool of pre-authenticated streaming sockets to the model, TTS provider, and telephony bridge. TLS handshakes at the moment of truth are unforgivable.
Small context, big memory
The runtime prompt should be short. Long-lived memory lives in a summary refreshed between turns, not in the on-turn context window.
Building intelligent systems for teams that ship. Writing at the intersection of AI, automation, and craft.