OpenAI Agents SDK: Adding Real Phone Calls

Scope: This walkthrough covers streaming audio between the OpenAI Agents SDK’s Realtime layer and a live PSTN call over a WebSocket media bridge, verified against the SDK documentation and transport guides published in September 2026. It does not cover browser WebRTC sessions or Azure OpenAI endpoint configuration.

Quick Answer: The OpenAI Agents SDK does not provide phone numbers or carrier access. To put it on a real call in 2026, you connect a telephony layer that streams pcm16 audio over WebSocket to a RealtimeSession, resample between the carrier’s 8 kHz audio and the Realtime API’s 24 kHz expectation, and handle call control separately.

The OpenAI Agents SDK enables server-side voice agents through a long-lived WebSocket connection to the Realtime API, but adding real phone calls requires a separate telephony layer that handles carrier audio, resampling, and call control before the SDK ever sees a single audio frame.

What is the OpenAI Agents SDK?

The OpenAI Agents SDK is a lightweight Python and JavaScript framework for building multi-agent and realtime voice applications on top of OpenAI’s Realtime API. It occupies the application layer: it wraps the raw WebSocket event flow in higher-level abstractions such as RealtimeAgent, RealtimeSession, and transport helpers, but it does not own the network transport or the media path below those abstractions. Specifically, the SDK ships with no PSTN connectivity, no phone number provisioning, and no carrier integration. A developer who wants to put a RealtimeAgent on a real phone call must supply the telephony bridge, the audio resampling pipeline, and the call control logic themselves. The SDK’s documented transport options are server-side WebSocket, browser WebRTC, a SIP bridge via OpenAIRealtimeSIP, and a Twilio extension transport, but none of these include the carrier-side infrastructure that delivers the actual phone call.

Why is adding real phone calls to the OpenAI Agents SDK harder than it looks?

The core problem is a format mismatch at every boundary. The Realtime API expects pcm16 audio, and the SDK’s Python quickstart confirms audio.input.format: "pcm16" and audio.output.format: "pcm16". PSTN carriers, by contrast, deliver narrowband audio at 8 kHz, defined by ITU-T G.711. The Realtime API operates at 24 kHz. That gap means every audio frame crossing the boundary needs resampling in both directions, and resampling adds latency. If you do it naively in a synchronous loop, you will hear it.

Beyond the codec mismatch, a phone call introduces turn-taking constraints that a browser voice session does not. The caller can interrupt at any moment. The model may be mid-sentence when the interruption arrives. Your bridge must flush the telephony layer’s queued audio the moment the interruption is detected, within the latency budget a human caller will tolerate before assuming the line is dead. The SDK handles the detection: with semantic_vad and interrupt_response: True it stops the model and emits an audio_interrupted event, which your bridge uses to send the clear message to Teler.

What does the full audio path look like for an OpenAI Agents SDK phone call?

The path from a caller’s handset to the model and back involves four distinct hops, each with its own format, owner, and failure surface. The table below maps them.

The resampling step sits between hop 2 and hop 3. Teler streams 16-bit PCM at 8 kHz, so your server must upsample to 24 kHz before writing frames to the RealtimeSession, and downsample the model’s 24 kHz output before injecting it back into the call.

What do you need before writing the first line of code?

• An OpenAI API key with Realtime API access, stored as OPENAI_API_KEY
• The OpenAI Agents SDK installed: pip install openai-agents for Python or npm install @openai/agents for JavaScript/TypeScript
• A Teler account with at least one provisioned phone number and a WebSocket media stream endpoint configured; see the Teler API documentation for number provisioning steps
• A resampling library: scipy and numpy in Python (pip install scipy numpy), or a native Node.js resampler if you are on the JavaScript SDK
• An async Python runtime (3.10 or later) or Node.js 18 or later
• Network reachability from your server to wss://api.openai.com on port 443, and from Teler’s infrastructure to your WebSocket endpoint on a public or VPN-accessible address
• A basic understanding of SIP (Session Initiation Protocol, RFC 3261) and RTP (Real-time Transport Protocol, RFC 3550), since the failure modes in production almost always trace back to those layers

How do you wire the OpenAI Agents SDK to a live phone call?

The implementation has four steps: create the agent, open the Realtime session, connect the telephony WebSocket, and handle audio in both directions. Each step is independent enough to test in isolation, which matters when you are debugging a live audio path.

Step 1: Create the RealtimeAgent

A RealtimeAgent is a specialised agent instance for use inside a RealtimeSession. Unlike a standard Agent, it does not accept a model parameter directly; the model is set at the session level. The SDK defaults to gpt-realtime-2.1 as of the September 2026 release notes.

import asyncio
from agents.realtime import RealtimeAgent, RealtimeRunner, RealtimeSession

agent = RealtimeAgent(
    name="phone_agent",
    instructions="You are a helpful assistant on a phone call. Keep responses concise.",
)

The instructions field shapes turn length; verbose instructions produce long model turns that increase the chance of mid-sentence interruption from the caller.

Step 2: Open the RealtimeSession with turn detection

The session holds the long-lived WebSocket connection to the Realtime API. Set semantic_vad for turn detection so the model responds to natural speech pauses rather than fixed silence thresholds, which matter more on a phone call where background noise is common.

async def run_call(teler_ws):
    runner = RealtimeRunner(
        starting_agent=agent,
        config={
            "model_settings": {
                "audio": {
                    "input": {
                        "format": "pcm16",
                        "turn_detection": {"type": "semantic_vad", "interrupt_response": True},
                    },
                    "output": {"format": "pcm16"},
                },
            },
        },
    )

    session = await runner.run()
    async with session:
        output_task = asyncio.create_task(stream_output(session, teler_ws))
        try:
            await handle_call(session, teler_ws)  # ends when Teler closes the socket
        finally:
            output_task.cancel()

The async with block keeps the WebSocket open for the call’s duration; exiting it closes the connection, so your telephony teardown logic must happen before the block exits.

Step 3: Accept the Teler WebSocket and resample inbound audio

Teler delivers call audio to your server over a WebSocket. Each message carries a chunk of 16-bit PCM audio at 8 kHz, wrapped in JSON with the audio base64-encoded. Before forwarding frames to the RealtimeSession, upsample to 24 kHz. The chunk size Teler sends is configurable from 20 to 2000 milliseconds in multiples of 20; the default is 400 ms. Teler’s specification requires you to send back at least 500 ms of audio per write to avoid choppy playback.

import base64
import json
import logging

import numpy as np
from scipy import signal

logger = logging.getLogger(name)

class StreamResampler:
“””
High-quality audio resampling for real-time communication between 16kHz/24kHz and Teler (8kHz).
“””

def __init__(self):
    self.gemini_input_sample_rate = 16000  #  expects 16kHz input
    self.gemini_output_sample_rate = 24000  # outputs 24kHz
    self.teler_input_sample_rate = 16000  # Teler streams to us at 16kHz
    self.teler_output_sample_rate = 8000   # Teler requires 8kHz from us

def downsample(self, audio_data: bytes, source_rate: int) -> bytes:
    """
    Downsample audio to Teler's required 8kHz.

    Args:
        audio_data: Raw PCM audio data as bytes
        source_rate: Source sample rate

    Returns:
        Downsampled audio data as bytes (pcm_s16le format at 8kHz)
    """
    try:
        # Convert bytes to numpy array (pcm_s16le format)
        audio_array = np.frombuffer(audio_data, dtype=np.int16)

        # Calculate downsampling factor
        factor = source_rate // self.teler_output_sample_rate

        # Use scipy.decimate for high-quality downsampling
        downsampled_array = signal.decimate(
            audio_array,
            q=factor,
            n=8,
            ftype='iir'
        )

        # Convert back to 16-bit PCM
        downsampled_int16 = downsampled_array.astype(np.int16)

        return downsampled_int16.tobytes()

    except Exception as e:
        logger.error(f"Error downsampling audio: {e}")
        return audio_data

def upsample(self, audio_data: bytes, target_rate: int) -> bytes:
    """
    Upsample audio from Teler's 16kHz input to target rate.

    Args:
        audio_data: Raw PCM audio data as bytes
        target_rate: Target sample rate

    Returns:
        Upsampled audio data as bytes
    """
    try:
        # Convert bytes to numpy array (pcm_s16le format)
        audio_array = np.frombuffer(audio_data, dtype=np.int16)

        # Calculate upsampling factor
        factor = target_rate / self.teler_input_sample_rate

        # Use scipy.resample for upsampling
        upsampled_array = signal.resample(
            audio_array,
            int(len(audio_array) * factor)
        )

        # Convert back to 16-bit PCM
        upsampled_int16 = upsampled_array.astype(np.int16)

        return upsampled_int16.tobytes()

    except Exception as e:
        logger.error(f"Error upsampling audio: {e}")
        return audio_data

resampler = StreamResampler()

async def handle_call(session: RealtimeSession, teler_ws):
# The loop ends by itself when Teler closes the socket at the end of the call
async for message in teler_ws:
data = json.loads(message)
if data[“type”] == “audio”:
raw = base64.b64decode(data[“data”][“audio_b64”]) # 16-bit PCM, 8 kHz
await session.send_audio(resampler.upsample(raw, 24000)) # 8 kHz -> 24 kHz
“StreamResampler downsamples with scipy.signal.decimate and upsamples with scipy.signal.resample

Step 4: Stream model audio back to the caller and handle interruptions

The SDK emits a RealtimeAudio event each time the model generates audio. Downsample from 24 kHz to 8 kHz and write the result to the Teler WebSocket as an audio message. When the caller interrupts, the SDK emits an audio_interrupted event. Send a {“type”: “clear”} message to Teler at that point to flush its queued playback. If your own logic needs to stop the model, call session.interrupt().

async def stream_output(session: RealtimeSession, teler_ws):
                pcm_8k = downsampler.process(event.audio.data)  # 16-bit PCM, 24 kHz -> 8 kHz
    chunk_id = 1

    async for event in session:
        if event.type == "audio":
            pcm_8k = downsampler.process(event.audio.data)  # 16-bit PCM, 24 kHz -> 8 kHz
            await teler_ws.send(json.dumps({
                "type": "audio",
                "audio_b64": base64.b64encode(pcm_8k).decode(),
                "chunk_id": str(chunk_id),
            }))
            chunk_id += 1
        elif event.type == "audio_interrupted":
            # Caller spoke over the agent: flush Teler's queued playback
            await teler_ws.send(json.dumps({"type": "clear"}))

The full reference implementation for the Realtime API voice-agent pattern is available in the openai/openai-realtime-agents repository. It is a demo, not a production telephony bridge, but it shows the event loop structure the SDK expects. For the Teler-side WebSocket protocol, see the Teler media streaming documentation.

Most teams discover their latency problem is not the model. It is the resampling hop they added between the carrier and the session, running synchronously on the same thread that handles WebSocket reads. Moving that conversion off the event loop cuts perceived lag more reliably than switching to a faster model.

– Subhash Kalluri, Founder & CEO, FreJun

What breaks in production when using the OpenAI Agents SDK for phone calls?

The failure modes below come from reported issues in the SDK’s GitHub repository and the OpenAI developer community. Each was reported once; none should be described as common, but each is worth knowing before you hit production.

The SIP audio failure (caller hears nothing while transcripts arrive) is the one that catches most teams, because the session looks healthy from the SDK’s perspective. The model is responding, events are flowing, but no RTP audio reaches the PSTN leg. The root cause in that thread was never resolved, which means the debugging path is to verify RTP port reachability, confirm the codec negotiated in the SDP matches what your bridge is sending, and check whether the telephony provider is receiving the RTP stream at all before assuming the SDK is at fault. RFC 4566 defines the SDP negotiation that controls codec selection; a mismatch there is silent at the application layer.

One honest limitation of Teler’s WebSocket transport: the 500 ms minimum send-back requirement means you cannot stream audio back in very small chunks even if your resampler produces them faster. Teams that try to send 20 ms frames to reduce perceived latency will hear choppy playback on the caller’s end. Buffer to at least 500 ms before each write.

Which transport approach fits your deployment?

The OpenAI Agents SDK documents four transport paths for voice. The right one depends on where the media terminates and how much of the telephony stack you want to own.

For most teams building AI phone agents in 2026, the server-side WebSocket path with a programmable telephony layer is the right starting point. It keeps the media path visible, the resampling logic under your control, and the model choice open. The Twilio extension transport is the right choice if your deployment is already Twilio-based and you want to avoid writing a custom bridge. The SIP path makes sense when you need carrier-grade call control and already operate SIP infrastructure; the W3C WebRTC specification and RFC 4566 are useful references for understanding the codec negotiation that determines whether audio actually flows. If you are building a browser voice agent rather than a phone agent, WebRTC is the correct path and the PSTN resampling problem does not apply.

Frequently asked questions

Does the OpenAI Agents SDK support DTMF tones from a phone keypad?

The SDK’s documented event types do not include DTMF handling. DTMF (Dual-Tone Multi-Frequency) signals are defined in RFC 4733 and travel as RTP telephone-event payloads. Your telephony layer must detect them and convert them to a signal your application can act on before passing audio to the RealtimeSession. Teler exposes DTMF as a call control event with a configurable tone duration of 40 to 1000 ms.

What happens when a SIP call connects but the caller hears silence?

This is the most-reported SIP failure mode in the OpenAI community forums. The session appears healthy at the SDK level because transcripts and events flow normally, but no audio reaches the PSTN leg. Start by verifying RTP port reachability between your media server and the carrier, then confirm the codec negotiated in the SDP matches what your bridge is actually sending. A mismatch between G.711 ulaw and alaw is silent at the application layer and audible only as silence on the caller’s end. The root cause in the reported thread was never resolved by OpenAI.

Should I use the SIP transport or the WebSocket transport for phone calls?

Use the WebSocket transport with an external telephony bridge if you want maximum control over the media path and do not already operate SIP infrastructure. The OpenAIRealtimeSIP transport attaches a RealtimeSession to an existing SIP-initiated call by callId, which is the right choice if you already own a SIP media server and need carrier-grade call control. The WebSocket path is easier to debug because every audio frame passes through your server and you can log or inspect it.

What audio format does the OpenAI Realtime API require, and how does that affect carrier audio?

The Realtime API requires pcm16 audio. The SDK quickstart confirms audio.input.format: "pcm16" and audio.output.format: "pcm16". PSTN carriers deliver G.711 audio at 8 kHz, while the Realtime API operates at 24 kHz. That means every inbound carrier frame must be upsampled by a factor of 3, and every outbound model frame must be downsampled by the same factor before it reaches the caller. Teler streams 16-bit PCM at 8 kHz.

What happens if a call runs longer than expected under load?

One reported failure shows every inbound SIP call terminating at exactly 30.16 seconds regardless of configuration, with the sideband WebSocket receiving no events. The root cause was unresolved as of August 2026. Under load, the more common issue is resampling latency accumulating in the event loop, which causes audio chunks to arrive out of order. Running the resampling step in a thread pool executor and keeping the WebSocket read loop non-blocking are the two changes that most reliably prevent this.

What does the developer have to build that the OpenAI Agents SDK does not provide?

The SDK does not provide carrier connectivity, phone number provisioning, audio resampling, jitter buffer management, DTMF handling, call recording, or call transfer logic. A developer building a production phone agent must supply all of these, either by integrating a programmable telephony layer like Teler or by building on a CPaaS (Communications Platform as a Service) that exposes raw media streams. Teler handles carrier connectivity, number provisioning in 50 or more countries, and WebSocket media streaming, leaving the resampling and session logic to the application layer.

Can the OpenAI Agents SDK handle call transfers to a human agent?

The SDK supports handoffs between RealtimeAgent instances, but transferring a call to a human agent on a different phone number requires a cold transfer at the telephony layer. Teler exposes a cold transfer call control action with a configurable timeout of 1 to 600 seconds. The SDK-level handoff and the telephony-level transfer are separate operations; you must coordinate them in your application logic, since the SDK has no awareness of the underlying phone call.

Is the handoff feature in the OpenAI Agents SDK reliable for multi-agent phone flows?

Not fully, as of mid-2026. A reported GitHub issue (openai-agents-python #1197) shows that after one handoff, subsequent turns stay pinned to the handed-off agent rather than returning to the original agent. This is unresolved. For production multi-agent phone flows, test the handoff chain thoroughly before deploying, and consider designing flows where handoffs are one-directional rather than bidirectional until the issue is addressed upstream.

What can you build now with the OpenAI Agents SDK and a telephony layer?

With the OpenAI Agents SDK connected to a programmable telephony layer, you can build production AI phone agents that handle inbound calls, run outbound campaigns, and transfer to human agents, all in 2026. The decision that matters most is where you place the resampling step: on the event loop thread and you will hear the latency; in a thread pool executor and you will not. If your deployment is already Twilio-based, the Twilio extension transport reduces the bridge work at the cost of provider coupling. For teams that want carrier flexibility and model-agnostic infrastructure, the server-side WebSocket path with Teler’s programmable voice API is the more maintainable architecture. For teams evaluating SIP trunking providers in the US or comparing voice API providers in the US, the transport decision in this guide applies regardless of which provider you choose.

About the Author: Subhash Kalluri is the Founder & CEO of FreJun, which he has been building since 2019. With over 8 years in voice communication and SaaS, he works on the problem of getting real-time AI onto live phone calls: raw media streaming, interruption handling, and carrier connectivity. That work became FreJun Teler, FreJun’s programmable voice API for developers building AI phone agents. Connect with him on LinkedIn.