Three platforms now dominate real-time AI phone agent development in 2026: OpenAI Realtime API, Google Gemini Live API, and ElevenLabs Conversational AI. Each takes a fundamentally different approach — different latency profiles, cost structures, customisation depth, and telephony integration story. Over the past year we have deployed AI phone agents on all three platforms across multiple client projects — from appointment booking to lead qualification — and measured what actually matters in production. Here is what we found.
Methodology note: Benchmarks are based on Samcom's internal monitoring data across real client deployments running on AWS us-east-1 with Twilio PSTN bridging. Latency numbers reflect end-of-utterance to first audio byte delivered to the caller. Your numbers will vary based on region, call complexity, and infrastructure choices.
The Three Platforms at a Glance
| Platform | Approach | LLM Backbone | Voice I/O |
|---|---|---|---|
| OpenAI Realtime API | Native audio-in / audio-out WebSocket | GPT-4o | Audio only — no text intermediate step |
| Google Gemini Live API | Native multimodal streaming WebSocket | Gemini 2.0 Flash | Audio + video + text in same stream |
| ElevenLabs Conversational AI | Managed full-stack agent platform | Pluggable (GPT-4o / Claude / Gemini) | ElevenLabs TTS + Deepgram STT built in |
Latency Benchmarks: End-of-Utterance to First Agent Audio
We define latency as the time from the moment the caller stops speaking (VAD end-of-utterance detection) to the moment the first audio chunk of the agent response arrives at the caller. This is the number that determines whether your agent feels human or robotic. Under 600ms feels natural. Over 1,000ms feels like a broken phone line.
| Platform | P50 Latency | P95 Latency | Best Observed | Worst Observed |
|---|---|---|---|---|
| OpenAI Realtime API | 420 ms | 680 ms | 310 ms | 1,100 ms |
| Gemini Live API | 480 ms | 790 ms | 350 ms | 1,350 ms |
| ElevenLabs Conversational AI | 540 ms | 910 ms | 390 ms | 1,600 ms |
OpenAI Realtime API wins on raw latency because GPT-4o natively processes audio tokens — there is no STT step at all. The model hears the voice and generates voice directly, cutting one full network round-trip from the pipeline.
Cost Per Minute Comparison
Cost is where the picture changes significantly. OpenAI Realtime API is the fastest but also the most expensive per minute. Gemini Flash is priced aggressively to capture market share. ElevenLabs sits in the middle but includes managed infrastructure, reducing your DevOps burden.
| Platform | Audio Input | Audio Output | Est. Total per Minute | Notes |
|---|---|---|---|---|
| OpenAI Realtime API | $0.06 / min | $0.24 / min | ~$0.30 / min | GPT-4o Realtime — premium pricing |
| Gemini Live API | $0.012 / min | $0.048 / min | ~$0.06 / min | Gemini 2.0 Flash — 5x cheaper than OpenAI |
| ElevenLabs Conv. AI | $0.08 / min (platform fee) | Included | ~$0.08–$0.12 / min | Includes STT, TTS, LLM, hosting |
At 10,000 minutes/month: OpenAI Realtime = ~$3,000 | Gemini Live = ~$600 | ElevenLabs = ~$1,000. For high-volume outbound campaigns, Gemini Live's cost advantage is decisive. For premium inbound where latency matters more than cost, OpenAI wins.
Architecture Deep Dive
OpenAI Realtime API
The Realtime API runs over a persistent WebSocket. You send PCM16 audio chunks at 24kHz and receive audio deltas back. VAD (voice activity detection) is server-side — OpenAI detects when the user stops speaking and triggers a response. Function calling works natively, letting you call your CRM, calendar, or database APIs mid-conversation in real time.
import asyncio, websockets, json, base64
async def realtime_agent():
url = "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview"
headers = {
"Authorization": f"Bearer {OPENAI_API_KEY}",
"OpenAI-Beta": "realtime=v1"
}
async with websockets.connect(url, extra_headers=headers) as ws:
# Configure session
await ws.send(json.dumps({
"type": "session.update",
"session": {
"modalities": ["audio", "text"],
"voice": "alloy",
"instructions": "You are an appointment booking assistant for Samcom Technologies...",
"tools": [book_appointment_tool],
"turn_detection": {"type": "server_vad", "threshold": 0.5}
}
}))
# Stream audio in, receive audio deltas out
async for message in ws:
event = json.loads(message)
if event["type"] == "response.audio.delta":
audio_chunk = base64.b64decode(event["delta"])
yield audio_chunk # send to Twilio / Telnyx media streamGoogle Gemini Live API
Gemini Live uses a BidiGenerateContent WebSocket (bidirectional streaming). The key differentiator is multimodal input — you can stream audio and video simultaneously, which opens up use cases like screen-sharing support agents. For pure phone agents, this is overhead, but the Gemini 2.0 Flash model's cost advantage makes it compelling for high-volume campaigns.
import asyncio
from google import genai
from google.genai import types
client = genai.Client(api_key=GOOGLE_API_KEY)
config = types.LiveConnectConfig(
response_modalities=["AUDIO"],
speech_config=types.SpeechConfig(
voice_config=types.VoiceConfig(
prebuilt_voice_config=types.PrebuiltVoiceConfig(voice_name="Aoede")
)
),
system_instruction="You are an appointment booking assistant...",
tools=[book_appointment_tool]
)
async with client.aio.live.connect(model="gemini-2.0-flash-live-001", config=config) as session:
# Send caller audio chunks
await session.send(input=types.Blob(data=audio_chunk, mime_type="audio/pcm;rate=16000"))
# Receive agent audio
async for response in session.receive():
if response.data:
yield response.data # PCM16 audio chunksElevenLabs Conversational AI
ElevenLabs Conversational AI is the most opinionated of the three — it is a fully managed platform rather than a raw API. You define your agent in a dashboard (or via API), choose your LLM backend (GPT-4o, Claude 3.5, or Gemini), and ElevenLabs handles STT (Deepgram), TTS (ElevenLabs Flash v2.5), and WebSocket session management. The trade-off: faster to build, less control over the audio pipeline.
from elevenlabs.conversational_ai.conversation import Conversation
from elevenlabs.conversational_ai.default_audio_interface import DefaultAudioInterface
from elevenlabs import ElevenLabs
client = ElevenLabs(api_key=ELEVENLABS_API_KEY)
conversation = Conversation(
client=client,
agent_id=AGENT_ID, # configured in ElevenLabs dashboard
requires_auth=True,
audio_interface=DefaultAudioInterface(),
callback_agent_response=lambda r: print(f"Agent: {r}"),
callback_user_transcript=lambda t: print(f"User: {t}"),
)
conversation.start_session() # handles STT, LLM, TTS automaticallyPSTN / Telephone Integration (The Part That Actually Matters)
For phone agents, connecting to real telephone numbers is non-negotiable. Each platform handles PSTN bridging differently, and this is often the deciding factor in which platform you choose.
| Platform | Twilio Integration | Telnyx Integration | SIP Trunking | WebRTC Browser |
|---|---|---|---|---|
| OpenAI Realtime API | Official Media Streams integration | Via WebSocket proxy | Via LiveKit or Asterisk bridge | Yes (via WebRTC client) |
| Gemini Live API | Via WebSocket proxy (manual) | Via WebSocket proxy (manual) | Via FreeSWITCH / Asterisk bridge | Requires custom implementation |
| ElevenLabs Conv. AI | Native Twilio integration (one-click) | Native Telnyx integration | SIP URI support | SDK-provided widget |
If you need the fastest path to a working phone number with zero infrastructure: ElevenLabs wins. If you need full control over audio routing, recording, and SIP integration with existing PBX infrastructure (Asterisk, FreeSWITCH, 3CX): OpenAI Realtime API via LiveKit or a custom WebSocket bridge gives you the most flexibility.
Customisation & Control
| Capability | OpenAI Realtime | Gemini Live | ElevenLabs Conv. AI |
|---|---|---|---|
| Custom voices | OpenAI built-in voices only | Google built-in voices only | Any ElevenLabs voice (2,000+ cloned voices) |
| Custom LLM | GPT-4o only | Gemini only | GPT-4o, Claude, Gemini, custom |
| Interruption handling | Server VAD built-in | Server VAD built-in | Configurable sensitivity |
| Function calling / tools | Full native tool use | Full native tool use | Via LLM backbone |
| Call transcript access | Via event stream | Via response stream | Webhook + dashboard |
| Custom STT | No — built into GPT-4o audio | No — built into Gemini | No — uses Deepgram |
| Conversation memory | In-session context window | In-session context window | Configurable knowledge base |
| White-label / embedding | Full API control | Full API control | SDK widget or full API |
When to Choose Each Platform
Choose OpenAI Realtime API when:
- Latency is paramount — premium inbound support, healthcare, or financial services where conversation quality justifies the cost
- You need deep integration with existing Asterisk, FreeSWITCH, or Kamailio SIP infrastructure via a custom WebSocket bridge
- Your team has Python/Node.js engineers who can manage a real-time WebSocket pipeline
- You are building on top of LiveKit Agents SDK which has first-class OpenAI Realtime support
Choose Gemini Live API when:
- High-volume outbound campaigns where cost-per-minute is the primary constraint (5x cheaper than OpenAI)
- You need multimodal input — audio + video simultaneously (unique to Gemini)
- You want Google ecosystem integration — Google Calendar, Google Workspace, Google Meet
- You are already on Google Cloud and want unified billing and IAM
Choose ElevenLabs Conversational AI when:
- Speed-to-market is your priority — production phone agent in days, not weeks
- You need premium voice quality with cloned or branded voices (ElevenLabs TTS is the industry leader)
- Non-technical teams need to manage agent scripts and knowledge bases via dashboard
- You want plug-and-play Twilio and Telnyx integration without managing WebSocket infrastructure
Our Production Recommendation (April 2026)
After 50,000+ calls across all three platforms, our recommendation is nuanced:
- For enterprise inbound (support, healthcare, financial services): OpenAI Realtime API via LiveKit Agents. Best latency, deepest control, full SIP integration. Budget $0.25–$0.35 per connected minute.
- For high-volume outbound campaigns (lead qualification, appointment reminders): Gemini Live API. 5x cheaper, good enough latency for outbound, Google ecosystem friendly. Budget $0.05–$0.08 per minute.
- For fast-to-market SMB deployments: ElevenLabs Conversational AI. Best voice quality, easiest setup, dashboard management. Budget $0.08–$0.14 per minute.
- For hybrid / custom requirements: Build your own pipeline — Deepgram STT → your LLM of choice → ElevenLabs TTS — orchestrated via LiveKit Agents or Pipecat. This gives you full control over every component and the ability to swap any layer independently.
The best AI voice platform is the one that matches your scale, budget, and integration requirements — not the one with the most impressive demo. Platform selection should come after you have defined your call volume, acceptable latency budget, required integrations, and total cost ceiling.
The AI phone agent market has matured dramatically in 2026. All three platforms — OpenAI Realtime, Gemini Live, and ElevenLabs Conversational AI — are production-grade. The choice comes down to your specific constraints: latency budget, cost per minute, voice quality requirements, and how deeply you need to integrate with existing telephony infrastructure. Our AI voice team has built production systems on all three platforms across 30+ countries. If the open question is the orchestration layer rather than the model, VAPI vs LiveKit covers pricing, latency and SIP for both. If you are evaluating which platform to build on, or need help architecting a production-grade AI phone agent pipeline, contact us for a free technical consultation — we will help you choose the right stack and build it right the first time.