Twilio Media Streams: How They Work and the Alternatives

AI Summary: This article covers how Twilio Media Streams works at the protocol level, who it suits, and what alternatives exist, written for backend engineers evaluating real-time audio streaming for AI voice pipelines in 2026. Twilio delivers raw audio as base64-encoded μ-law at 8 kHz over a secure WebSocket; audio sent back to Twilio must match that format exactly, with no file header bytes. Engineers must supply their own codec conversion, VAD, and downstream model integration. eler is a programmable voice API that exposes a similar WebSocket media path, but delivers 16-bit linear PCM at 8k or 16k instead of μ-law, which removes the μ-law decode step. Like Twilio’s clear, Teler’s clear and interrupt messages flush queued audio; detecting the interruption is still up to you.

Scope: This walkthrough covers Twilio Media Streams as documented in the official Twilio Voice docs and changelog, verified against the state of the API as of September 2026, including the <Connect><Stream> DTMF support that reached general availability on 2026-09-14. It does not cover Twilio Voice Intelligence or Twilio Flex media routing.

Quick Answer: Twilio Media Streams forks live call audio to a developer-controlled WebSocket server using the <Stream> or <Connect><Stream> TwiML verbs. Audio arrives as base64-encoded μ-law at 8 kHz. Audio sent back must be audio/x-mulaw at 8000 Hz with no file header bytes. In 2026, the bidirectional variant also surfaces DTMF events.

Twilio Media Streams gives developers raw WebSocket access to live call audio, but the 8 kHz μ-law format and the protocol lifecycle it imposes mean most AI voice pipelines require a codec conversion step before the audio reaches a speech model.

What is Twilio Media Streams?

Twilio Media Streams is a Twilio Voice feature that forks the raw audio of a live PSTN (Public Switched Telephone Network) or SIP call to a developer-controlled WebSocket server in near real time. It sits at the media layer of the Twilio platform, below call control and above the carrier, and it does not perform transcription, speech recognition, or any AI inference itself. In unidirectional mode the WebSocket server receives audio but cannot send audio back to the call; in bidirectional mode, started with <Connect><Stream>, the server can inject audio into the call but Twilio only delivers the inbound track from the caller. The developer must supply the WebSocket server, the codec conversion pipeline, any STT or LLM integration, and the TTS synthesis layer if the agent needs to speak.

Why is real-time audio streaming over WebSockets hard to get right?

The core constraint is format mismatch: telephony audio arrives as narrowband μ-law (ITU-T G.711) at 8 kHz, while most modern speech and language models expect 16 kHz or 24 kHz linear PCM. Every hop that converts between these formats adds latency and, if done carelessly, introduces distortion. The resampling step is often where latency budgets collapse before the model is even involved.

Twilio Media Streams compounds this by delivering audio in a queue-based WebSocket message flow. If the downstream processing falls behind, frames accumulate and the agent ends up responding to audio that is several seconds old. A dev.classmethod.jp post from December 2025 documented a case where backlog grew to tens of seconds under sustained load, requiring an explicit load-shedding layer to keep delay near 320 ms. That is not a Twilio bug; it is the expected behaviour of any unbounded queue feeding a slow consumer.

How does the Twilio Media Streams audio path work end to end?

The path from caller to model and back involves several distinct hops, each with its own format, owner, and latency contribution. The table below maps each hop so you can identify where conversion is required.

The two hops you own, the conversion from 8 kHz μ-law to your model’s expected format and the reverse path back to Twilio, are where most production problems originate. The ITU-T G.711 specification defines the μ-law encoding; the RTP specification (RFC 3550) governs how that audio travels over the carrier network before Twilio receives it.

What do you need before you start?

• A Twilio account with a provisioned phone number and Programmable Voice enabled
• A publicly reachable WebSocket endpoint over wss:// on TCP port 443; Twilio requires this and will not connect to ws:// or non-standard ports
• Your firewall configured to accept inbound WebSocket connections from any public IP address (Twilio does not publish a fixed egress IP range for Media Streams)
• Your application configured to validate the X-Twilio-Signature header on every incoming request
• A μ-law-to-PCM conversion library; Python developers commonly use audioop (stdlib through Python 3.12; on 3.13+ install the audioop-lts package) or pywav for file output.
• A downstream STT or LLM endpoint that accepts your converted audio format

How do you implement Twilio Media Streams for an AI voice pipeline?

Step 1: Configure TwiML to start the stream

For a unidirectional stream, use the <Stream> verb inside <Start>. For a bidirectional stream where your server injects audio back into the call, use <Connect><Stream>. The bidirectional variant blocks subsequent TwiML execution while the WebSocket is open.

<!-- Unidirectional: Twilio sends audio to your server -->
<Response>
  <Start>
    <Stream url="wss://your-server.example.com/media" />
  </Start>
  <Say>Streaming has started.</Say>
</Response>

<!-- Bidirectional: your server can also send audio back -->
<Response>
  <Connect>
    <Stream url="wss://your-server.example.com/media" />
  </Connect>
</Response>

Watch the verb choice: <Connect><Stream> blocks the TwiML queue, so any <Say> or <Play> after it will not execute until the WebSocket closes or the call ends.

Step 2: Handle the WebSocket message lifecycle

Twilio sends JSON messages over the WebSocket. The message types you must handle are connected, start, media, dtmf, stop, and mark. And media message carries the base64-encoded μ-law audio in the media.payload field. The clear message goes the other direction: your server sends it to Twilio to flush buffered audio and return any pending mark acknowledgements.

import asyncio, base64, audioop, json
import websockets

async def handle_stream(websocket):
    ratecv_state = None  # carried across frames
    async for raw in websocket:
        msg = json.loads(raw)
        event = msg.get("event")

        if event == "connected":
            print("WebSocket connected")

        elif event == "start":
            stream_sid = msg["start"]["streamSid"]
            print(f"Stream started: {stream_sid}")

        elif event == "media":
            # Decode base64 mulaw payload
            mulaw_bytes = base64.b64decode(msg["media"]["payload"])
            # Convert 8 kHz mulaw to 16 kHz linear PCM for downstream STT
            pcm_8k = audioop.ulaw2lin(mulaw_bytes, 2)
            pcm_16k, ratecv_state = audioop.ratecv(
            pcm_8k, 2, 1, 8000, 16000, ratecv_state)
            # Forward pcm_16k to your STT engine here

        elif event == "stop":
            print("Stream stopped")

async def main():
    async with websockets.serve(handle_stream, "0.0.0.0", 8765):
        await asyncio.Future()

asyncio.run(main())

The audioop.ratecv call returns a tuple; index 0 is the converted bytes and index 1 is the resampler state, which you must carry across frames to avoid clicks at chunk boundaries.

Step 3: Send audio back to the call (bidirectional only)

When your TTS engine returns audio, convert it to μ-law at 8000 Hz, base64-encode it, and send it as a media event. The payload must not include any WAV or file header bytes; raw μ-law samples only. Twilio’s docs state the payload can be of any size, but keeping chunks to a few hundred milliseconds avoids perceptible gaps if the WebSocket stalls.

import audioop, base64, json

async def send_audio_to_twilio(websocket, stream_sid, pcm_24k_bytes, state=None):
    # Resample 24 kHz PCM (from TTS) down to 8 kHz; pass the returned
    # state back in on the next call to avoid clicks at chunk boundaries
    pcm_8k, state = audioop.ratecv(pcm_24k_bytes, 2, 1, 24000, 8000, state)
    # Convert linear PCM to mulaw
    mulaw_bytes = audioop.lin2ulaw(pcm_8k, 2)
    payload = base64.b64encode(mulaw_bytes).decode("utf-8")

    await websocket.send(json.dumps({
        "event": "media",
        "streamSid": stream_sid,
        "media": {"payload": payload}
    }))

async def clear_twilio_buffer(websocket, stream_sid):
    """Call this when the caller interrupts the agent."""
    await websocket.send(json.dumps({
        "event": "clear",
        "streamSid": stream_sid
    }))

The clear event is the barge-in mechanism: send it when your VAD (voice activity detection) detects the caller speaking over the agent, and Twilio will discard any audio it has buffered but not yet played.

The official reference implementation is in the twilio/media-streams GitHub repository, which includes working examples in Node.js, Python, Java, and Ruby. The Python quickstart is also documented in the Twilio Flask tutorial. For the full WebSocket message schema, see the Teler media streaming docs alongside the Twilio reference for a side-by-side comparison of message types.

Most teams discover their latency problem is not the model; it is the resampling hop they added between the carrier and the STT engine. An 8 kHz μ-law stream converted to 24 kHz PCM in a blocking call on the same thread as the WebSocket handler will accumulate backlog within seconds under any real call volume. Move the conversion off the receive loop and measure queue depth, not just round-trip time.

– Subhash Kalluri, Founder & CEO, FreJun

What breaks in production with Twilio Media Streams?

The failure modes below come from documented reports with source URLs. Frequency is stated honestly; a single Stack Overflow report is not a common problem.

The latency snowball is the one that catches most teams. It is invisible during single-call testing because a single call rarely saturates the worker. Under concurrent load, the queue grows faster than it drains, and the agent starts responding to audio from several seconds ago. The fix is not a faster model; it is a bounded queue with explicit frame-dropping logic on the receive side.

One honest limitation of Teler worth stating here: Teler’s configurable chunk size (a multiple of 20 ms, 400 ms by default) gives you more control over frame granularity than Twilio’s fixed delivery cadence but it does not eliminate the queue-depth problem. If your downstream model is slow, you still need load-shedding logic regardless of which telephony layer you use.

What are the alternatives to Twilio Media Streams?

Four distinct architectural approaches exist for getting live call audio into an AI pipeline. The right choice depends on where you want the media to terminate and how much infrastructure you want to own.

How do the Twilio Media Streams alternatives compare?

Which Twilio Media Streams alternative fits your architecture?

If you are already on Twilio and your pipeline handles the μ-law conversion correctly, Twilio Media Streams is a reasonable starting point. Choose a different approach when you need a configurable sample rate without writing your own resampler,when you want linear PCM instead of μ-law, or when you need phone number provisioning across more than a handful of countries without managing multiple carrier relationships. The SIPREC path is the right choice for standards-based call recording workflows where portability across vendors matters more than low integration latency. For more on how real-time audio streaming integrates with AI pipelines, see How To Handle Real-Time Audio Streams With Voice API Integration and Media Streaming Architecture Explained: From API Call To Real-Time Conversation.

Frequently asked questions

What audio format does Twilio Media Streams deliver?

Twilio delivers audio as base64-encoded μ-law (G.711) at 8 kHz in the media.payload field of each WebSocket message. Audio you send back to Twilio must also be audio/x-mulaw at 8000 Hz with no file header bytes included in the payload. Most modern STT and LLM APIs expect 16 kHz or 24 kHz linear PCM, so a conversion step is required on both the inbound and outbound paths.

Why does my saved audio sound staticky?

The most common cause is treating the μ-law payload as if it were standard PCM WAV data. μ-law encoding maps 8-bit values to a non-linear amplitude scale; writing those bytes into a WAV container configured for linear PCM produces distorted output. Use a μ-law-aware writer or decode the payload with audioop.ulaw2lin before writing. A reported fix used pywav.WavWrite(..., 8000, 8, 7) to specify mono, 8 kHz, 8-bit, μ-law format explicitly.

When should I use WebSocket media streaming versus a SIP trunk into a media server?

Use WebSocket media streaming when you want the telephony platform to handle carrier connectivity and call control while your server owns the AI processing path. Use a SIP trunk into a media server when you need full media ownership, advanced routing logic, or integration with existing PBX infrastructure. The SIP path adds RTP jitter buffer complexity and NAT traversal work; the WebSocket path adds codec conversion work. Neither is universally simpler. The SIP specification (RFC 3261) governs the SIP-side behaviour.

What happens to latency under concurrent call load?

Latency compounds if your WebSocket handler processes frames synchronously or if the downstream model is slow. Frames accumulate in the receive queue faster than they drain, and the agent responds to increasingly stale audio. The fix is a bounded queue with explicit frame-dropping once delay exceeds your budget, not a faster model. Teler’s configurable chunk size (a multiple of 20 ms, 400 ms by default) gives you finer control over frame granularity, but you still need load-shedding logic if your model cannot keep up.

Does Twilio Media Streams support DTMF?

Yes, as of 2026-09-14, DTMF support for <Connect><Stream> bidirectional connections reached general availability. Twilio surfaces DTMF keypresses as dtmf events in the WebSocket message stream. The Twilio docs do not document sending DTMF tones from your server back to the call through the media stream; that path is not supported in the retrieved documentation.

What must I build myself that Twilio Media Streams does not provide?

You must supply the public WebSocket server, μ-law-to-PCM codec conversion, resampling to your model’s expected sample rate, VAD or barge-in detection, STT and LLM integration, TTS synthesis, and the reverse conversion back to μ-law for audio injection. Twilio handles carrier connectivity, call control, and the WebSocket transport. Teler delivers 16-bit linear PCM at 8k or 16k, so there’s no μ-law decode step, and offers clear and interrupt messages to flush playback. As with Twilio, VAD and barge-in detection stay in your code. See the Teler developer docs for the full message reference.

Does Twilio Media Streams lock me into a specific STT or LLM provider?

No. Twilio explicitly documents use with Google Speech-to-Text, Google Dialogflow, Amazon Lex, Amazon Transcribe, and Gridspace Sift, among others. The WebSocket transport is model-agnostic; you forward the converted audio to whichever provider your pipeline uses. The lock-in is to Twilio for carrier access and call control, not to any downstream AI provider.

How do I handle barge-in when the caller interrupts the agent?

Send a clear event from your server to Twilio over the bidirectional WebSocket. Twilio will discard any audio it has buffered but not yet played and return any pending mark acknowledgements. Your VAD layer must detect the caller speaking and trigger the clear send; Twilio does not perform server-side VAD on your behalf. For a deeper look at interruption handling patterns, see How Can Companies Enable Real-Time Interruptions While Building Voice Bots.

What Twilio Media Streams tells you about building with real-time audio in 2026

Twilio Media Streams gives you direct access to live call audio over a well-documented WebSocket protocol, and for teams already on Twilio it is the lowest-friction path to feeding a call into an AI pipeline. The decision that matters most is whether you want to own the codec conversion and VAD layers yourself or offload them to the telephony infrastructure. If you want to own them, Twilio Media Streams is a reasonable foundation; if you want linear PCM at a selectable sample rate with no μ-law conversion, a purpose-built layer like Teler is worth evaluating. Choose Twilio Media Streams when your team has the bandwidth to build and maintain the conversion pipeline; choose a higher-abstraction alternative when that engineering time is better spent on the AI layer itself.

About the Author: Subhash Kalluri is the Founder & CEO of FreJun, which he has been building since 2019. With over 8 years in voice communication and SaaS, he works on the problem of getting real-time AI onto live phone calls: raw media streaming, interruption handling, and carrier connectivity. That work became FreJun Teler, FreJun’s programmable voice API for developers building AI phone agents. Connect with him on LinkedIn.