Build a voice agent in a weekend: STT, LLM, TTS

A voice agent is an LLM agent with voice I/O. The whole trick is making it feel conversational: under a second from the moment someone stops talking to the first sound of your answer, and the ability to be interrupted mid-sentence like a person. Everything else is plumbing.

This is a realistic weekend plan: Saturday, a working pipeline in the browser or on a phone line. Sunday, make it feel human — streaming, latency budgets, interruption handling. No custom training, no telephony contracts. Just code and a few API keys.

The stack: what each piece does#

[mic / phone] → VAD → streaming STT → LLM (streaming) → streaming TTS → [speaker / phone]
                   │                      │                  │
                   └───── turn logic ─────┴─── barge-in ─────┘

VAD (voice activity detection) decides when someone starts and stops speaking. Silero VAD is the free, standard choice; it runs on CPU and fires in tens of milliseconds. Every serious pipeline runs VAD first, because everything downstream — endpointing, barge-in — depends on knowing exactly when speech happens.

Streaming STT turns audio into text as it arrives, emitting partial transcripts before the speaker finishes. Deepgram's streaming models and AssemblyAI's streaming API are the common hosted picks. The old non-streaming approach (record a chunk, send it off, wait for a transcript) adds seconds and is the first thing to kill.

The LLM sits in the middle and must stream tokens. Pick anything with low time-to-first-token: a small fast model like gpt-4o-mini-class, a self-hosted 7–8B model on vLLM, or a speech-to-speech realtime API (OpenAI's gpt-realtime, Gemini Live) if you want to skip the cascade. For a weekend build, a fast chat model over an SSE stream is the simplest.

Streaming TTS converts tokens to audio before the LLM finishes. ElevenLabs (Flash/Turbo voices), Cartesia, and Deepgram's Aura voices all stream audio in small chunks with time-to-first-audio in the low hundreds of milliseconds. Non-streaming TTS — wait for the full response, synthesize, then play — is the second thing to kill.

Turn logic is the part beginners skip and regret. You need: endpointing (deciding the user is done speaking, usually 150–250ms of trailing silence), a "continue if the sentence ends mid-thought" merge window, and barge-in. Get these wrong and your agent answers halfway through sentences or talks over people.

Saturday morning: the skeleton#

Start with a managed platform to hear an agent fail before you build your own. Vapi, Retell, or ElevenLabs Agents can put a working agent on a web widget or a phone number in under five minutes. Spend an hour talking to it. Notice the two failure modes that matter: it answers too early (endpointing too aggressive) or it keeps talking when you interrupt (no barge-in). You're going to fix both in your own build.

Then scaffold your own pipeline. The fastest route is an open-source framework rather than raw sockets:

• Pipecat (Python, open source): the quickstart CLI scaffolds a Deepgram + OpenAI + Cartesia pipeline you can talk to in a browser in about five minutes. Vendor-neutral — swap any provider later.
• LiveKit Agents (Python/Node): the right pick if your UI runs in a browser and you want WebRTC for the audio leg. Ships a Silero VAD plugin and an open turn-detector model.

Either way, the "hello world" is the same: microphone audio in, a greeting out. Wire it with no streaming at first — full turn, then response. It will feel like a voicemail system, and that's the point.

Saturday afternoon: make it stream#

Here's the key mental model, from a Salesforce AI Research tutorial build that measured each stage:

WITHOUT streaming (turn-based):
[====STT====][======LLM======][====TTS====][play]   → ~2s wait

WITH streaming (overlapping):
[====STT====]
             [tok][tok]["Hello,"][tok][tok]["I can"]
                        |                    |
                  [TTS "Hello,"]       [TTS "I can"]
                        |                    |
                     [play]              [play]        → ~700ms wait

Realtime comes from streaming plus pipelining, not from any single fast model. The Salesforce build measured ~337ms for Deepgram STT, ~337ms for LLM time-to-first-token (a self-hosted Qwen 7B), and ~219ms for ElevenLabs TTS time-to-first-audio — and about 750ms end-to-end once the stages overlapped.

The concrete changes:

1. Stream STT partials, but only act on finals. Use partial transcripts for pre-computation (speculative retrieval, pre-fetching context) and commit to a turn only when the endpointing rule fires.
2. Stream LLM tokens into a sentence chunker. Don't wait for the full response. Flush the first sentence — or even the first clause — to TTS as soon as punctuation appears. Aggressive first-clause flushing is worth 50–150ms.
3. Stream TTS in small frames. If your TTS sends 200ms audio chunks, barge-in will always feel late (more on this Sunday). Some teams drop to ~30ms frames so the audio queue can be flushed instantly.
4. Tune the LLM for voice. Prompt it for short spoken answers: one idea per sentence, no markdown, no lists, no codes or URLs read literally. Voice is linear — a three-paragraph chat answer is unusable aloud.

Log per-turn timing from day one: VAD fire, STT final, LLM first token, TTS first audio, playback start. You can't optimize what you don't measure, and the slow stage is never the one you expect.

Sunday: interruptions and turn-taking#

This is the day that separates an agent from an IVR. Two problems, both cheap to fix, both catastrophic to skip.

Barge-in: stop when the user talks

Human conversation has one rule: when interrupted, you stop. The pipeline version:

• The moment VAD detects user speech while the agent is talking: flush the TTS audio queue, stop playback, cancel the LLM stream.
• Keep recording — never drop mic frames during the interruption.
• Send the interruption to the LLM as first-class input, including what the agent was mid-sentence saying. (The Asterisk voice-agent project documents this well: record only what the caller actually heard, and drop tool calls from the interrupted turn so history stays valid.)
• Handle the easy cases specially: backchannels like "uh-huh" or "yeah" shouldn't stop anything; "what?" should repeat the last sentence rather than re-derive it.

One engineering detail from a 2026 writeup on this exact problem: teams found their TTS streamed in 200ms chunks, so even after VAD fired, two buffered chunks (~400ms) played to completion — users heard the agent finish a fragment before silence. Shrinking TTS frames and flushing the queue on VAD fire moved their barge-in success rate dramatically. The target is a sub-200ms barge-in stop (VAD onset → playback halted).

Endpointing: don't answer halfway through

The mirror problem: knowing when the user finished. Standard approach:

• A short silence threshold (around 150–250ms) commits the turn. Too short and you answer mid-sentence; too long and every reply feels sluggish.
• Add a merge window (a few hundred ms) after the "final" transcript: if more speech arrives, merge it into the same turn. Extend the window automatically when the text ends mid-dictation or on a dangling word.
• Modern turn-detection models (e.g., LiveKit's open turn detector, ~50–160ms per decision) beat pure silence rules, especially for speakers who pause mid-thought.

Test both with real interruptions: script prompts where you cut the agent off at controlled points, in a noisy room. Log the timestamps and look at them. This is your eval set — start with 20 scripted scenarios before you let real users near it.

The weekend build order#

A few parting rules that hold up across every production build writeup:

• Target <800ms end-to-end, <500ms if you can. Nobody measured a single model doing this alone; it's all overlap.
• The metric users feel is interruption success, not average latency. One team improved their barge-in rate and felt faster to users while their p99 latency got slightly worse.
• Go hybrid on providers. Start with hosted STT/TTS for speed of development, bring open-source components in when you can benchmark them, and always keep a fallback for the piece most likely to fail (usually STT in noisy conditions).
• Trace turns, not logs. A per-turn timeline — VAD, STT final, LLM TTFT, TTS first audio, playback — is the debugging tool. Plain text logs won't find a 300ms stall.

By Sunday evening you won't have a product — you'll have something better: a working conversational loop, a latency budget you measured, and interruption handling that doesn't embarrass you. That's the hard part. Tools, RAG, a phone number, polish — just more turns of the same loop.

Takeaway#

Building a voice agent in a weekend is a solved shape: streaming STT, a fast streaming LLM, streaming TTS, VAD-driven turn logic. Spend Saturday on the pipeline and streaming overlap, Sunday on barge-in and endpointing, and measure every stage. The agent that stops when interrupted and answers in under a second will beat a smarter model that can't do either.