AI Summary: This article covers the WebSocket vs SIP decision for backend and API integration engineers building AI voice agents in 2026. The core constraint is audio format: SIP carries G.711 PCM at 8,000 samples per second over RTP, while WebSocket-based media streaming exposes raw linear PCM chunks that your server processes directly. Engineers must decide which transport owns the media path before writing a single line of agent logic. Teler’s programmable voice API supports both paths, streaming raw audio/l16 at 8 kHz or 16 kHz over WebSocket and offering SIP trunking with UDP/TCP/TLS transport for teams that already run SIP infrastructure.
Scope: This walkthrough covers real-time media transport choices for AI voice agents connecting to PSTN calls, verified against Teler’s WebSocket media streaming documentation and SIP trunking specification as documented in September 2026. It does not cover WebRTC peer-to-peer sessions or browser-native media endpoints.
Quick Answer: WebSocket streams raw audio chunks directly to your server so your STT engine processes them with no intermediary; SIP routes media over RTP and requires a SIP stack to handle dialog state, codec negotiation, and NAT traversal. In 2026, most AI voice agent teams start with WebSocket transport because it eliminates SIP infrastructure overhead while still reaching PSTN callers through a provider like Teler.
The WebSocket vs SIP decision for AI voice agents is fundamentally a question of where audio terminates and who owns the media loop. WebSocket delivers raw PCM frames to your application server; SIP delivers encoded RTP packets to a media server or softswitch that you operate.
What is real-time media transport for voice agents?
Real-time media transport for voice agents is the layer that moves raw audio between a telephone network and the AI pipeline that processes it. It sits between the carrier edge and your speech-to-text engine, and it determines the codec, sample rate, framing, and delivery guarantee your agent receives. It does not include the STT model, the LLM, or the TTS synthesizer. Those components are your responsibility. What the transport layer supplies is a continuous, low-latency audio stream and a mechanism to inject synthesized audio back into the live call. A developer building on this layer must supply their own STT integration, LLM orchestration, TTS rendering, and interruption logic on top of whatever the transport exposes.
Why is the WebSocket vs SIP choice harder than it looks?
The two protocols solve different problems at different layers, and conflating them is the first mistake most teams make. WebSocket (defined in RFC 7118) is a persistent, message-oriented TCP transport. SIP (RFC 3261) is a signaling protocol that sets up sessions; the actual audio travels separately over RTP (RFC 3550). Choosing between them is not choosing between two audio transports. It is choosing between two architectural models for who owns the media path.
The engineering constraint that makes this choice consequential is the audio format mismatch. Standard narrowband telephony runs ITU-T G.711 PCM at 8,000 samples per second with 8-bit companded encoding. A WebSocket media stream from a provider like Teler exposes audio/l16 at 8,000 Hz, which is linear PCM, not companded. Your STT engine may expect a different sample rate entirely. If you are streaming to the OpenAI Realtime API, for example, it expects 24 kHz audio, which means a resampling hop sits between the carrier and the model. That hop adds latency and, if implemented carelessly, adds distortion. The SIP path has the same underlying codec problem but surfaces it differently: your media server handles codec negotiation during SDP exchange, so the mismatch is resolved at the infrastructure layer rather than in your application code.
What does the audio path look like end to end?
Understanding the full audio path from carrier to model and back is the prerequisite for making the WebSocket vs SIP decision correctly. The table below maps each hop for both transport models.
| Hop | WebSocket path | SIP/RTP path |
|---|---|---|
| Carrier to edge | PSTN, G.711 PCM, 8 kHz | PSTN, G.711 PCM, 8 kHz |
| Edge to your server | WebSocket, audio/l16, 8 kHz or 16 kHz, Base64-encoded chunks | RTP over UDP, codec per SDP negotiation, your media server receives it |
| Your server to STT | Your code resamples if needed, streams to STT engine | Media server transcodes, forwards to STT engine |
| TTS back to caller | Your code sends audio chunks over the same WebSocket | Media server injects audio into the RTP stream |
| Who owns media state | Your application server | Your SIP stack or media server |
| NAT traversal | TCP, handled by WebSocket upgrade | STUN/TURN or SBC required for RTP |
On the WebSocket path, Teler sends a start message first, containing stream metadata including encoding: "audio/l16", sample_rate: 8000, and channels: 1. Subsequent audio messages carry Base64-encoded PCM chunks. You can flush queued playback at any time by sending {"type": "clear"}, which is how barge-in interruption works at the transport layer. The SIP path delivers no such application-layer control primitive; interruption handling is your media server’s responsibility.
What do you need before you start?
- A Teler account with an API key, passed as the
x-api-keyheader on all authenticated requests - A provisioned Teler phone number in your target country (50+ countries available via API)
- A publicly reachable HTTPS endpoint to serve your call flow JSON (the Flow URL)
- A WebSocket server capable of handling persistent connections and binary or text frames, or a SIP stack if you are taking the SIP path
- Python (
pip install teler) or Node.js (npm i @frejun/teler) SDK installed, or direct WebSocket library of your choice - An STT engine endpoint, such as Deepgram, AssemblyAI, or the OpenAI Realtime API, with its own credentials
- If streaming to OpenAI Realtime at 24 kHz: a resampling library, because Teler’s stream delivers 8 kHz or 16 kHz audio and the Realtime API requires 24 kHz input
How do you implement WebSocket vs SIP media transport for AI voice agents?
Step 1: Define the call flow
Teler fetches a call flow JSON object from your Flow URL when a call arrives. The flow tells Teler to open a media stream and hand you a ws_url. Your server connects to that URL to begin receiving audio. The flow is retrieved over HTTPS POST, so your endpoint must be reachable from Teler’s edge nodes.
{
"actions": [
{
"type": "stream",
"sample_rate": "8k",
"chunk_size": 400
}
]
}The chunk_size field accepts values from 20 to 2000 milliseconds in multiples of 20; the default is 400. Teler’s documentation warns that chunks that are too small produce choppy or distorted audio, and recommends sending at least 500 milliseconds of audio back per response chunk.
Step 2: Open the WebSocket and handle the start message
Once Teler opens the WebSocket, the first message is always a start event containing stream metadata. Read it before processing any audio frames. The stream_id field identifies the session for logging and debugging.
import asyncio
import websockets
import json
import base64
async def handle_call(websocket):
async for raw_message in websocket:
message = json.loads(raw_message)
if message["type"] == "start":
stream_id = message["stream_id"]
encoding = message["encoding"] # "audio/l16"
sample_rate = message["sample_rate"] # 8000
# initialise your STT session here
elif message["type"] == "audio":
pcm_bytes = base64.b64decode(message["data"])
# forward pcm_bytes to your STT engineThe encoding value is always "audio/l16" and the sample_rate is the integer 8000 when you configure the stream at "8k". Confirm both before passing audio to your STT engine, because a mismatch here is the most common source of garbled transcription.
Step 3: Inject TTS audio back into the call
Send synthesized audio back over the same WebSocket connection as audio messages. To cancel queued playback when the caller interrupts, send a clear message before the next audio chunk. This is the barge-in mechanism at the transport layer.
async def send_audio(websocket, pcm_bytes):
encoded = base64.b64encode(pcm_bytes).decode("utf-8")
await websocket.send(json.dumps({
"type": "audio",
"data": encoded
}))
async def clear_queue(websocket):
await websocket.send(json.dumps({"type": "clear"}))Send audio in chunks of at least 500 milliseconds. Smaller chunks increase round-trip overhead and risk audible gaps between frames.
Step 4: SIP trunking path for teams with existing infrastructure
If your team already operates a SIP stack, PBX, or media server, Teler supports SIP trunking with UDP/TCP/TLS transport and credential-based or IP-based authentication. You configure an IP access control list on the Teler side and point your SIP trunk at Teler’s carrier network. The media path then terminates at your media server rather than at a WebSocket handler. See the Teler documentation for SIP trunk configuration details. The trade-off is that you own SIP dialog state, RTP jitter buffering, codec transcoding, and NAT traversal, which is significant operational surface area if you are starting from scratch.
A working reference integration connecting Teler’s WebSocket transport to the Gemini Live API is available at frejun-tech/teler-gemini-bridge on GitHub. It demonstrates the full session lifecycle from call initiation through audio streaming and response injection. For more on building real-time assistants on top of this transport layer, see How To Build Real-Time Assistants Using An AI Voice Agent API.
Most teams discover their latency problem is not the model. It is the resampling hop they added between the carrier and the STT engine, often without realising the carrier delivers 8 kHz G.711 and the model expects 24 kHz linear PCM. Fix the format mismatch first, then tune chunk size.
– Subhash Kalluri, Founder & CEO, FreJun
What breaks in production with WebSocket vs SIP transport?
Both transport paths have well-documented failure modes. The table below covers the ones that appear most often in production deployments, sourced from GitHub issues, community threads, and practitioner write-ups.
| Symptom | Root cause | Fix |
|---|---|---|
| Choppy or robotic playback | Audio chunks sent before enough buffer accumulates, or chunk size too small | Buffer at least 500 ms before playback; increase chunk_size in the stream flow config |
| Garbled audio under packet loss | WebSocket over TCP has no programmatic loss recovery at the application layer; WiFi or cellular loss hits the stream directly | No application-layer fix; consider a transport with built-in loss concealment for lossy networks |
| 4 to 5 second delay before first response | Waiting for the full stream to connect before sending chunks; silence-timer endpointing adds additional delay | Stream audio in chunks inside the WebSocket loop; reduce silence threshold; keep the initial prompt short |
| STT socket dies silently after 60 to 70 seconds of silence | No keepalive or reconnect logic on the STT WebSocket path | Add explicit ping-based keepalive and reconnect handling on the STT connection |
| Audio corruption in edge runtimes | Binary WebSocket framing behaves differently in Cloudflare Workers versus Node.js | Use the Node.js WebSocket stack; avoid Workers for binary audio paths until the runtime issue is resolved |
The failure that catches the most teams is the silent STT socket death. The WebSocket to Teler stays alive, the call continues, but the STT connection has quietly closed after a period of silence. The agent stops transcribing but keeps the call open, producing a session that appears healthy in your monitoring but delivers no responses to the caller. Add explicit keepalive pings and reconnect logic on every downstream WebSocket connection, not just the one facing the telephony provider.
One honest limitation of Teler’s WebSocket transport: the documentation does not specify WebSocket close codes, timeout values, or maximum session duration. Teams building long-running sessions, such as multi-hour contact center calls, should instrument their own session health checks and test reconnection behaviour before going to production. The media streaming documentation covers message types and stream parameters but does not currently document keepalive semantics.
How do WebSocket and SIP compare for AI voice agent architecture?
The WebSocket vs SIP decision maps to a set of concrete trade-offs. The table below covers the axes that matter most for AI voice agent deployments in 2026.
| Dimension | WebSocket media streaming | SIP/RTP trunking |
|---|---|---|
| Where media terminates | Your application server | Your SIP stack or media server |
| Audio format at your boundary | audio/l16, 8 kHz or 16 kHz, Base64 chunks | G.711 or negotiated codec over RTP |
| NAT traversal | TCP, handled by WebSocket upgrade | STUN/TURN or Session Border Controller required |
| Interruption handling | Application-layer clear message, your code owns it | Media server owns it, more infrastructure to configure |
| Operational burden | Low: no SIP stack, no RTP jitter buffer, no SBC | High: SIP dialog state, codec negotiation, NAT traversal, jitter buffering |
| Interop with existing PBX | Requires a bridge or gateway | Native: SIP trunk connects directly |
| Model agnosticism | Full: your server routes audio to any STT or model | Full: your media server routes audio to any STT or model |
| Right choice when | Starting from scratch, no existing SIP infrastructure | Existing SIP PBX, contact center platform, or carrier SIP trunk |
For most teams building AI voice agents in 2026 without existing telephony infrastructure, WebSocket media streaming is the faster path to a working agent. The operational surface area is smaller, the audio format is explicit and easy to inspect, and interruption handling is a single JSON message rather than a media server configuration. Choose the SIP path when you already operate SIP infrastructure and need to integrate the AI agent into an existing call flow without replacing the carrier connectivity layer. For teams evaluating how to scale this further, How To Scale AI Voice Agent APIs For Contact Center Traffic covers the architectural patterns that apply at higher call volumes.
WebRTC runs on UDP, was built for low-latency real-time media, handles NAT traversal, and produces noticeably better interruption handling and voice quality than WebSocket transport.
– Luong Hong Thuan, author, luonghongthuan.com
Frequently asked questions
Can I use WebSocket transport to reach regular phone numbers?
Yes. WebSocket is the application-layer transport between your server and the telephony provider’s edge. The provider, such as Teler, handles PSTN connectivity and phone number provisioning separately. Your WebSocket handler receives audio from real phone calls without touching SIP or RTP directly. Teler provisions numbers in 50+ countries via API.
What audio format does Teler’s WebSocket stream deliver?
Teler sends audio/l16 linear PCM at 8,000 Hz, mono, with audio chunks Base64-encoded inside JSON messages. You can configure the stream at "16k" for 16,000 Hz if your STT engine benefits from the higher rate. The start message always confirms the encoding and sample rate before audio frames arrive.
Why does my agent have a 4 to 5 second delay at the start of every call?
The most common cause is waiting for the entire stream connection to establish before sending audio chunks, compounded by a silence-timer endpointer that waits for the caller to stop speaking before forwarding audio to the STT engine. Stream audio in chunks inside the WebSocket loop from the first frame, reduce the silence threshold, and keep the agent’s opening prompt short to cut perceived latency.
Does WebSocket transport handle packet loss as well as RTP?
No. WebSocket runs over TCP, which retransmits lost packets but adds latency when retransmission occurs. RTP over UDP drops lost packets and relies on jitter buffers and loss concealment at the media server. For most broadband connections the difference is negligible, but on lossy mobile or WiFi networks, TCP retransmission can cause audible gaps that a UDP-based transport with loss concealment would mask.
What do I need to build myself on the WebSocket path that SIP handles automatically?
On the WebSocket path your application code owns interruption handling via the clear message, STT session management including keepalive and reconnect, resampling if your model expects a different sample rate than the stream delivers, and any DTMF parsing your agent needs. SIP infrastructure handles codec negotiation and some of these concerns at the media server layer, but at the cost of operating that infrastructure.
What happens to a WebSocket session if the STT connection drops silently?
The call continues and audio keeps arriving over the Teler WebSocket, but your agent stops transcribing. The session appears healthy in basic monitoring. This is one of the most dangerous failure modes in production: add explicit keepalive pings and reconnect logic on every downstream WebSocket, not just the telephony-facing one. The Teler WebSocket and the STT WebSocket are separate connections with independent lifetimes.
Can I switch between WebSocket and SIP without changing my AI model integration?
Yes, because both paths deliver audio to your server and your model integration sits above the transport layer. The audio format and framing differ between paths, so you need an adapter at the boundary, but the STT, LLM, and TTS components do not change. Teler’s model-agnostic architecture supports both transport options against the same carrier network.
Is SIP over WebSocket a third option worth considering?
SIP over WebSocket (RFC 7118) tunnels SIP signaling over a WebSocket connection for browser-friendly clients, but audio still travels over the SIP media path. It reduces one browser integration hurdle but does not eliminate SIP dialog state, RTP handling, or NAT traversal complexity. It is worth considering only when a browser client must interoperate with existing SIP infrastructure.
What the WebSocket vs SIP decision means for your build
The WebSocket vs SIP choice for AI voice agents in 2026 is a decision about operational ownership, not audio quality. WebSocket media streaming gives your application server direct access to raw PCM frames with minimal infrastructure, while SIP trunking routes media through a stack you operate and maintain. The one decision that matters most is whether you already have SIP infrastructure: if you do, the SIP path integrates without replacing it; if you do not, WebSocket is the faster path to a working agent.
Choose SIP when carrier-grade telephony interop with an existing PBX is a hard requirement. For teams building AI voice agents on top of Teler’s programmable voice API, both paths are available, and the WebSocket transport’s explicit audio format and application-layer interruption control make it the lower-friction starting point for most new deployments. Check current Teler pricing before committing to a transport architecture, since telephony costs vary by channel and usage volume.
