Building Indic-Language Voice Agents with Sarvam and Teler

AI Summary: This article covers how backend engineers can build Indic-language voice agents using the Sarvam API for speech recognition and synthesis, paired with Teler for carrier connectivity and real-time audio streaming. Sarvam’s saaras:v3-realtime WebSocket endpoint, accepts raw PCM at 8 kHz or 16 kHz. Engineers must decide whether to bridge media directly over WebSocket or route through a media server before reaching Sarvam. Teler handles phone number provisioning, and raw audio delivery so the application layer can focus on language model orchestration.

Scope: This walkthrough covers streaming audio between Sarvam’s saaras:v3-realtime WebSocket STT endpoint and a live PSTN call over Teler’s WebSocket transport, verified against Sarvam API documentation retrieved on 2026-09-17 and Teler’s published specifications. It does not cover Sarvam’s REST batch transcription endpoint or SIP-side PBX configuration.

Quick Answer: To build an Indic-language voice agent in 2026, connect Teler’s WebSocket media stream to Sarvam’s saaras:v3-realtime endpoint, and pipe Bulbul v3 TTS output back through Teler at a matching sample rate. The Sarvam API handles Hindi, Tamil, Telugu, and every major Indian language natively.

Building a production Indic-language voice agent requires the Sarvam API for speech recognition and synthesis, a telephony layer for carrier access, and a resampling bridge between the two audio worlds. The sections below walk through each component in order.

What is the Sarvam AI?

Sarvam AI is a production API platform that provides transcription, translation, transliteration, text-to-speech, and conversational AI tuned for Hindi, Tamil, Telugu, and every major Indian language, including code-mixed input. It occupies the AI model layer: it processes audio and text, but it does not provision phone numbers, terminate SIP trunks, or carry PSTN traffic. The platform exposes REST endpoints for batch work and WebSocket endpoints for real-time streaming. A developer building a phone-based voice agent must supply the telephony transport, the codec conversion between carrier audio and Sarvam’s expected formats, and the call control logic that decides when to speak and when to listen.

Why is Indic-language telephony integration hard?

The main challenge in Indic-language telephony integration is ensuring that the audio format matches the requirements of the voice pipeline. Teler provides narrowband telephony audio at 8 kHz, and Sarvam supports 8 kHz audio for its streaming STT API when the sample rate is configured accordingly. This allows the Teler-Sarvam bridge to process the call audio without requiring resampling in this setup. For code-mixed conversations, such as when a caller naturally switches between English and an Indian language, Sarvam’s codemix mode can be used to preserve the mixed-language transcript accurately.

Beyond audio format, Indic-language conversations can involve natural switching between languages. Sarvam supports code-mixed speech through its codemix mode, making it suitable for conversations where callers switch between English and Indian languages within the same sentence.

How does audio move from a phone call to the Sarvam AI and back?

The end-to-end path has four hops. Understanding who owns each hop determines where bugs hide in production.

Teler delivers audio to your server over a persistent WebSocket at either 8 kHz or 16 kHz, configurable at call setup. If you configure Teler at 8 kHz and Sarvam STT expects 16 kHz, your server must upsample before forwarding. On the return path, Sarvam TTS can output at 8 kHz, which avoids a second resampling step if you match the sample rates at configuration time. The Sarvam API documentation confirms 8 kHz as a supported TTS output rate for Bulbul v3.

What do you need before writing the first line of code?

• A Teler account with a provisioned phone number and a WebSocket media stream endpoint configured. See Teler’s media streaming documentation for stream setup parameters.
• A Sarvam API key, passed as the api-subscription-key header on every request.
• Python 3.10 or later, or Node.js 18 or later. The official sarvam-ai-sdk supports both runtimes (last commit 2026-05-30 on the sarvamai/sarvam-ai-sdk repository).
• Network reachability from your server to api.sarvam.ai over WebSocket (port 443, TLS).
• Teler stream chunk size set to a multiple of 20 ms. The default is 400 ms; for voice agents, 200 ms is a reasonable starting point to balance latency against overhead.

How do you implement the Sarvam API voice agent pipeline?

Step 1: Open the Teler media stream

When Teler delivers an inbound call event, Teler POSTs the call details to your flow URL, and you return a stream flow. Teler then opens the WebSocket to the ws_url in that flow. Configure the stream at 16 kHz so the audio arrives at the sample rate Sarvam STT recommends, which removes the upsample step entirely.

import asyncio
import websockets
import json

# Teler POSTs to your flow URL when a call arrives. Return this JSON.
# Teler then connects to ws_url, so there is no config message to send.
def stream_flow() -> dict:
    return {
        "action": "stream",
        "ws_url": "wss://your-server.example.com/media",
        "chunk_size": 200,      # ms, a multiple of 20
        "sample_rate": "16k",   # audio Teler sends to you
    }

async def handle_teler_stream(teler_ws):
    # Teler is the client here: it connects to your ws_url
    await bridge_to_sarvam(teler_ws)

The chunk_size_ms value must be a multiple of 20. Smaller chunks reduce latency but increase WebSocket frame overhead, so 200 ms is a practical floor for most deployments.

Step 2: Open the Sarvam realtime STT WebSocket

Open a persistent WebSocket connection to Sarvam’s realtime STT endpoint and configure it for the language and VAD behaviour your agent needs. Use saaras:v3-realtime, not the legacy saaras:v3 streaming endpoint, which Sarvam’s changelog explicitly moved to Legacy status in August 2026.

SARVAM_STT_WS = "wss://api.sarvam.ai/speech-to-text-translate/streaming"
SARVAM_HEADERS = {"api-subscription-key": "YOUR_SARVAM_KEY"}

async def open_sarvam_stt():
    sarvam_ws = await websockets.connect(
        SARVAM_STT_WS,
        extra_headers=SARVAM_HEADERS
    )
    # Configure session: codemix mode, 16 kHz, VAD sensitivity on
    await sarvam_ws.send(json.dumps({
        "model": "saaras:v3-realtime",
        "language_code": "hi-IN",
        "mode": "codemix",
        "sample_rate": 16000,
        "high_vad_sensitivity": True,
        "flush_signal": True
    }))
    return sarvam_ws

Set flush_signal=True from the start. A reported issue in the Sarvam cookbook repository shows that omitting this flag causes turns to hang open, which stalls the realtime loop.

Step 3: Bridge audio chunks between Teler and Sarvam

Teler sends JSON audio messages with base64-encoded PCM. Decode each one and forward the audio directly to Sarvam’s STT WebSocket. Because both sides are now at 16 kHz, no resampling is needed in this path.

import base64

async def bridge_to_sarvam(teler_ws):
    sarvam_ws = await open_sarvam_stt()

    async def teler_to_sarvam():
        async for message in teler_ws:
            event = json.loads(message)
            if event["type"] == "audio":
                audio_bytes = base64.b64decode(event["data"]["audio_b64"])  # 16-bit PCM, 16 kHz
                await sarvam_ws.send(audio_bytes)

    async def sarvam_to_agent():
        async for message in sarvam_ws:
            event = json.loads(message)
            if event.get("type") == "transcript":
                await handle_transcript(event["text"], teler_ws, sarvam_ws)

    await asyncio.gather(teler_to_sarvam(), sarvam_to_agent())

Parse the JSON message, extract the Base64-encoded audio payload, and forward it to Sarvam in the format expected by its STT WebSocket.

Step 4: Generate a response and inject audio back into the call

When Sarvam returns a final transcript, send it to your LLM, pass the response text to Sarvam’s Bulbul v3 TTS endpoint, decode the base64 audio, and write it back to the Teler WebSocket. Send a {"type": "clear"} frame first if the caller interrupted the agent mid-utterance.

import base64
import logging
import uuid

import httpx
import numpy as np
from scipy import signal

logger = logging.getLogger(__name__)



class StreamResampler:
    """
    High-quality audio resampling for real-time communication between 16kHz/24kHz and Teler (8kHz).
    """

    def __init__(self):
        self.gemini_input_sample_rate = 16000  #  expects 16kHz input
        self.gemini_output_sample_rate = 24000  # outputs 24kHz
        self.teler_input_sample_rate = 16000  # Teler streams to us at 16kHz
        self.teler_output_sample_rate = 8000   # Teler requires 8kHz from us

    def downsample(self, audio_data: bytes, source_rate: int) -> bytes:
        """
        Downsample audio to Teler's required 8kHz.

        Args:
            audio_data: Raw PCM audio data as bytes
            source_rate: Source sample rate

        Returns:
            Downsampled audio data as bytes (pcm_s16le format at 8kHz)
        """
        try:
            # Convert bytes to numpy array (pcm_s16le format)
            audio_array = np.frombuffer(audio_data, dtype=np.int16)

            # Calculate downsampling factor
            factor = source_rate // self.teler_output_sample_rate

            # Use scipy.decimate for high-quality downsampling
            downsampled_array = signal.decimate(
                audio_array,
                q=factor,
                n=8,
                ftype='iir'
            )

            # Convert back to 16-bit PCM
            downsampled_int16 = downsampled_array.astype(np.int16)

            return downsampled_int16.tobytes()

        except Exception as e:
            logger.error(f"Error downsampling audio: {e}")
            return audio_data

    def upsample(self, audio_data: bytes, target_rate: int) -> bytes:
        """
        Upsample audio from Teler's 16kHz input to target rate.

        Args:
            audio_data: Raw PCM audio data as bytes
            target_rate: Target sample rate

        Returns:
            Upsampled audio data as bytes
        """
        try:
            # Convert bytes to numpy array (pcm_s16le format)
            audio_array = np.frombuffer(audio_data, dtype=np.int16)

            # Calculate upsampling factor
            factor = target_rate / self.teler_input_sample_rate

            # Use scipy.resample for upsampling
            upsampled_array = signal.resample(
                audio_array,
                int(len(audio_array) * factor)
            )

            # Convert back to 16-bit PCM
            upsampled_int16 = upsampled_array.astype(np.int16)

            return upsampled_int16.tobytes()

        except Exception as e:
            logger.error(f"Error upsampling audio: {e}")
            return audio_data


resampler = StreamResampler()


async def handle_transcript(text: str, teler_ws, sarvam_ws):
    llm_response = await call_your_llm(text)  # your LLM call here

    tts_payload = {
        "inputs": [llm_response],
        "target_language_code": "hi-IN",
        "speaker": "meera",
        "model": "bulbul:v3",
        "sample_rate": 16000,
        "enable_preprocessing": True
    }
    async with httpx.AsyncClient() as client:
        r = await client.post(
            "https://api.sarvam.ai/text-to-speech",
            json=tts_payload,
            headers={"api-subscription-key": "YOUR_SARVAM_KEY"}
        )
    audio_bytes = base64.b64decode(r.json()["audios"][0])
    pcm_8k = resampler.downsample(audio_bytes, 16000)  # Sarvam TTS is 16 kHz here; Teler requires 8 kHz

    # Flush any queued playback before injecting new audio
    await teler_ws.send(json.dumps({"type": "clear"}))
    await teler_ws.send(json.dumps({
        "type": "audio",
        "audio_b64": base64.b64encode(pcm_8k).decode(),
        "chunk_id": uuid.uuid4().hex,
    }))

The TTS REST endpoint caps requests at 2,500 characters. For longer LLM responses, split the text at sentence boundaries and stream each chunk sequentially, sending at least 500 ms of audio per write to avoid choppy playback on the Teler side.

Most teams discover their latency problem is not the model. It is the resampling hop they added between the carrier and the STT engine. If you configure Teler and Sarvam at the same sample rate from the start, you eliminate the most common source of audio degradation before you write a single line of model code.

– Subhash Kalluri, Founder & CEO, FreJun

What breaks in production when using the Sarvam API for voice agents?

The failure that catches most teams is the first-audio latency problem. Waiting for a complete STT response before calling TTS, and then waiting for a complete TTS response before playing audio, stacks three sequential round trips. Each one adds hundreds of milliseconds. Streaming both legs over WebSocket and piping audio to playback as it arrives is the architectural change that collapses that stack. One honest limitation of Teler worth noting: the transport layer targets sub-100ms packet delivery as a specification, but actual end-to-end latency depends on your server’s geographic proximity to both Teler’s points of presence and Sarvam’s API endpoints. Placing your bridge server in a Mumbai or Hyderabad region reduces the round trip for Indian callers significantly.

Which integration approach fits your architecture?

For most new Indic-language voice agent builds in 2026, the direct Teler-to-Sarvam WebSocket path is the right choice. It keeps the hop count low and puts codec decisions in one place: your bridge server. The opposite choice, routing through a media server, is correct when you are integrating with an existing SIP infrastructure that you cannot replace, or when you need features like conference bridging that a media server provides natively. See Teler’s SIP trunking configuration guide for credential and IP authentication options when connecting an existing media server.

Frequently asked questions

What audio format does the Sarvam API streaming STT endpoint require?

The streaming STT endpoint accepts WAV and raw PCM formats, including pcm_s16le, pcm_l16, and pcm_raw. The Sarvam documentation recommends 16 kHz as the sample rate for streaming. The REST STT endpoint accepts a broader set of formats including MP3, AAC, OGG, OPUS, FLAC, and others, but PCM files on REST are supported only at 16 kHz and require the input_audio_codec parameter to be set explicitly.

What happens if I send 8 kHz audio to Sarvam’s 16 kHz STT endpoint?

The Sarvam documentation does not specify the exact error response for a sample rate mismatch, so the failure mode is not officially documented. In practice, sending audio at the wrong sample rate to a speech model typically produces degraded accuracy or garbled transcripts rather than a clean error. Configure Teler to stream at 16 kHz to avoid the mismatch entirely, which is the safest path given the documentation gap.

Should I use SIP trunking or a direct WebSocket for the media path?

Use SIP trunking when you need real phone numbers and carrier access, which is the case for any agent that receives or places calls on the PSTN. Teler provides SIP trunking with credential or IP authentication and supports IP access control lists. Use a direct WebSocket media stream when you control both endpoints and want the lowest-latency path from carrier audio to your bridge server. The two are not mutually exclusive: SIP delivers the call, and the WebSocket carries the media stream from Teler to your server.

What is the character limit for Sarvam TTS requests?

The REST TTS endpoint caps requests at 2,500 characters. The HTTP streaming endpoint allows up to 3,500 characters per request. The WebSocket TTS endpoint caps each message at 2,500 characters, and the Sarvam documentation recommends keeping messages under 500 characters for the lowest latency. For LLM responses longer than 500 characters, split the text at sentence boundaries and send each segment as a separate TTS request.

How does Teler handle barge-in when the caller interrupts the agent?

Teler supports barge-in through a {"type": "clear"} control frame that flushes all queued audio playback immediately. When your VAD detects that the caller has started speaking while the agent is playing audio, send the clear frame to Teler before injecting the next TTS chunk. This prevents the agent from finishing a sentence the caller has already interrupted, which is the most common source of unnatural conversation flow in production deployments.

What does the Sarvam API not provide that I must build myself?

Sarvam does not provide PSTN connectivity, phone number provisioning, SIP trunking, call control, DTMF handling, or call recording. It also does not handle jitter buffering or codec conversion between carrier audio and its own API formats. A developer building a phone-based voice agent must supply all of these, either by using a telephony provider like Teler or by building their own media gateway.

Does the Sarvam API support code-mixed Hindi-English speech?

Yes. The Sarvam streaming STT endpoint exposes a codemix mode specifically for speech that switches between Hindi and English within a single utterance. This is one of the documented STT modes alongside transcribe, translate, verbatim, and translit. Set "mode": "codemix" in the session configuration when your callers are likely to mix languages, which is the common pattern in Indian consumer-facing applications.

How do I support multilingual calls across different Indian languages using the Sarvam API?

Set the language_code parameter in the STT session configuration to the BCP-47 code for the target language, such as hi-IN for Hindi or ta-IN for Tamil. If the agent needs to handle multiple languages in the same session, you can use Sarvam’s automatic language detection where supported, or reconfigure the session mid-stream when the language changes. This allows the voice agent to handle multilingual conversations without requiring a separate session for every language.

Building Indic-language voice agents with the Sarvam API in 2026

With the Sarvam API handling speech recognition and synthesis for Indian languages, and Teler providing the carrier connectivity and raw audio delivery, you can build a production Indic-language voice agent without writing telephony infrastructure from scratch. The one decision that matters most is sample rate alignment: configure both Teler and Sarvam at 16 kHz from the start, and you eliminate the resampling hop that causes most of the audio quality problems teams encounter in the first week of integration. If your architecture requires an existing SIP media server in the path, the approach still works, but you take on jitter buffering and codec conversion as additional responsibilities. For teams evaluating voice API providers for AI agents in India, the Sarvam-plus-Teler stack is one of the few combinations that handles Indic phoneme complexity and PSTN connectivity in the same pipeline without requiring a third-party STT provider.

About the Author: Subhash Kalluri is the Founder & CEO of FreJun, which he has been building since 2019. With over 8 years in voice communication and SaaS, he works on the problem of getting real-time AI onto live phone calls: raw media streaming, interruption handling, and carrier connectivity. That work became FreJun Teler, FreJun’s programmable voice API for developers building AI phone agents. Connect with him on LinkedIn.