AI voice agents are no longer a research prototype — they are production systems answering thousands of calls per day for sales teams, healthcare providers, and customer support desks. After building 60+ voice agent pipelines, we want to share exactly how we architect sub-500ms AI voice agents using LiveKit, GPT-4o Realtime, Deepgram STT, and ElevenLabs TTS.
Why Latency Is Everything
Human conversation tolerates a maximum of about 300–500ms of silence before it feels awkward. Phone-quality voice agents need end-to-end latency — from the moment the user stops speaking to the moment the agent starts responding — below 800ms to feel natural. On a consumer internet connection, the theoretical minimum is already 150–200ms. Every architectural decision you make either eats into that budget or preserves it.
The Core Stack
- LiveKit — WebRTC media server and agent SDK (handles all real-time audio routing)
- OpenAI GPT-4o Realtime API — native audio-in, audio-out model (eliminates STT→LLM→TTS pipeline delay)
- Deepgram Nova-3 — fastest streaming STT on the market (~80ms first token)
- ElevenLabs Turbo v2 — streaming TTS with sub-200ms first audio chunk
- Python FastAPI — agent orchestration, tool calls, CRM webhooks
- Twilio or Telnyx — SIP PSTN bridge to real phone numbers
Architecture: Two Patterns
Pattern A — GPT-4o Realtime (Fastest)
OpenAI's Realtime API accepts raw audio in and streams audio back out. This eliminates the classic STT→LLM→TTS chain and cuts end-to-end latency to 400–600ms on a good connection. LiveKit connects to the Realtime API via a WebSocket. The agent SDK handles VAD (voice activity detection), turn management, and function calling.
from livekit.agents import AutoSubscribe, JobContext, WorkerOptions, cli
from livekit.agents.llm import ChatContext
from livekit.plugins import openai, silero
async def entrypoint(ctx: JobContext):
await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)
agent = VoicePipelineAgent(
vad=silero.VAD.load(),
stt=openai.STT(model="whisper-1"),
llm=openai.LLM(model="gpt-4o-realtime-preview"),
tts=openai.TTS(voice="alloy"),
chat_ctx=ChatContext().append(
role="system",
text="You are a helpful assistant for Samcom Technologies.",
),
)
agent.start(ctx.room)Pattern B — Deepgram + GPT-4o + ElevenLabs (Most Customisable)
When you need fine-grained control — custom voices, precise interruption handling, or complex tool-calling pipelines — the three-stage pipeline gives more flexibility. Deepgram streams transcription as the user speaks (not after). GPT-4o processes the transcript and calls tools. ElevenLabs streams the audio response back.
Deepgram Nova-3 returns the first transcript token in ~80ms. Enable "interim_results: true" to start processing before the user has finished speaking.
Connecting to Real Phone Numbers
LiveKit agents run over WebRTC. To connect to a real phone (PSTN), you need a SIP bridge. Both Twilio and Telnyx support SIP trunking into LiveKit via the LiveKit SIP connector. The agent answers a Twilio phone number, the call is bridged via SIP into LiveKit, and the agent handles the conversation.
CRM Integration: The Last 20%
The voice conversation is only half the value. The other half is what happens with the data. During the call, the agent should be calling tools — creating leads in Salesforce or HubSpot, updating deal stages, booking calendar appointments via Google Calendar API, and sending SMS confirmations via Twilio. All of this happens in real time, during the call, using GPT-4o's native function calling.
Production Checklist
- Use Deepgram endpointing with utterance_end_ms: 1000 to handle natural speech pauses
- Implement answering machine detection (AMD) for outbound campaigns
- Add call recording with PII redaction before storing transcripts
- Set up Redis for conversation history across multi-turn calls
- Monitor P95 latency per call — alert if > 900ms
- Use LiveKit's agent dispatch for horizontal scaling across multiple agent workers
AI voice agents built on LiveKit + GPT-4o Realtime are production-ready today. The key to sub-500ms latency is choosing the right pipeline for your use case, running agents close to your telephony provider, and obsessing over every millisecond from VAD detection to first TTS audio chunk. If you want to talk through your architecture, contact our team for a free technical consultation.