Open-Source Voice AI Agents: What You Still Have to Build

Open-source voice AI agent frameworks like Pipecat, LiveKit, and Bolna give teams a strong starting point, but telephony, compliance, and conversation quality gaps remain. This guide covers what US businesses...
AI voice agent open source stack: STT, LLM and TTS in place, with the phone line piece from Teler.
In this article

AI Summary: This article covers what US teams building an ai voice agent open source system still need to handle after choosing a framework, written for founders, operations leads, and product managers evaluating the real scope of the work. A 2026 analysis of 10,000 calls found that 34 percent had disruption in the first eight seconds once callers recognized they were speaking to an AI (Reddit, August 2026). Readers must decide which gaps to close themselves and which to hand to a managed telephony layer. FreJun Teler connects AI systems to live phone calls so teams can focus on conversation design rather than carrier infrastructure.

Scope: This article covers open-source voice AI agent frameworks available to US-based teams in 2026, including the telephony, compliance, and conversation-quality work that frameworks do not handle out of the box. Regulatory claims reflect FCC and FTC rules checked against primary sources as of September 2026.

Quick Answer: An ai voice agent open source framework gives you the orchestration code, but you still need to connect phone numbers, choose speech and language models, wire in your calendar or customer records system, and meet FCC and FTC consent rules before your first outbound call in 2026. The framework is the engine; the rest of the car is yours to build.

An ai voice agent open source setup handles the logic of a live phone conversation, but the phone network, the compliance layer, and the business systems it must talk to are all things a team must supply separately.

What is an open-source voice AI agent?

An open-source voice AI agent is a self-hostable software framework that listens to a caller in real time, decides what to say or do next, and speaks a reply, all without a human operator in the loop. A working setup has five parts: a connection to the phone network, a speech-to-text engine (software that turns spoken words into written text), a language model (the AI that decides what to say), a text-to-speech engine (software that turns written text back into spoken audio), and an orchestration layer that ties them together. Frameworks like Pipecat, LiveKit Agents, and Bolna provide the orchestration layer and connectors to common speech and language services, but they do not provide the phone numbers, the carrier connections, or the business-system integrations a real deployment needs.

The most common mix-up is treating a voice agent as a smarter version of an IVR (interactive voice response, the press-1-for-billing menus most callers know) or a text chatbot. The difference is that a voice agent holds a live, open-ended spoken conversation rather than playing fixed recordings or exchanging typed messages. If the system cannot handle a sentence it has never seen before, it is not a voice agent.

Why are teams building ai voice agent open source systems now?

The cost of the underlying speech and language models has dropped sharply since 2024, and several open-source frameworks have reached a level of maturity where a small team can get a working prototype in days rather than months. Pipecat had over 13,000 GitHub stars as of July 2026, LiveKit Agents over 11,000, and both projects show active maintenance in 2026. That momentum has pulled more teams into the space, including teams that have never built telephony software before, which is where the gaps start to show.

The shift is also practical. A voice agent that answers after-hours calls, qualifies inbound leads, or sends appointment reminders can replace work that previously required a staffed phone line. For businesses with predictable, repeatable call types, the economics are straightforward once the setup is done. The setup, however, is larger than most teams expect.

How does an ai voice agent open source call actually work?

Every call follows the same sequence, regardless of which framework a team uses. The table below shows what happens at each step, who or what handles it, and where things typically break.

StepWhat happensWho handles itWhat can go wrong
1. Call connectsThe caller dials in or the system dials out; audio starts flowingTelephony provider and phone networkPoor line quality, dropped connection, number not provisioned
2. Speech to textThe caller’s words are converted to written text in real timeSpeech-to-text engine (chosen by the team)Names, accents, and background noise cause misreads
3. DecisionThe language model reads the transcript and decides what to say or look upLanguage model (chosen by the team)Slow lookups cause noticeable pauses; wrong tool results mislead the agent
4. Text to speechThe reply is converted to spoken audioText-to-speech engine (chosen by the team)Unnatural phrasing, wrong emphasis, or a voice that sounds robotic
5. Call continues or endsThe conversation loops back to step 2, transfers to a human, or closesOrchestration layer and human-transfer hookNo clear handoff path; caller gets stuck or hangs up

The table shows that the framework handles step 3 and parts of steps 2 and 4, but the telephony connection in step 1 and the human-transfer hook in step 5 are almost always the team’s responsibility to wire up.

Two things callers notice most are the pause after they stop speaking (while the system transcribes and thinks) and what happens when they try to interrupt. A well-built agent can handle interruptions mid-sentence, just like a person, so the caller does not have to wait for the agent to finish before speaking. Frameworks support this feature, but teams still need to tune it. A 2026 analysis of 10,000 calls found that roughly 28 percent had interruption-management problems where filler words like “yeah” triggered full interruptions and caused the agent to lose context (Reddit, August 2026).

What does it cost to run an ai voice agent open source system?

Cost has two distinct shapes, and confusing them is the most common budgeting mistake. The table below shows the main cost lines.

Cost lineWho charges itWhat makes it go up
Phone capacity (simultaneous calls)Telephony providerMore calls happening at the same time
Inbound per-minute usageTelephony providerLonger calls, higher inbound volume
Outbound per-minute usageTelephony providerLonger calls, higher outbound volume
Phone number rentalTelephony providerMore numbers, more markets
Call recording and storageTelephony or cloud providerMore recordings, longer retention
Speech-to-text usageAI model providerMore audio minutes processed
Language model usageAI model providerMore conversation turns, longer calls
Text-to-speech usageAI model providerMore characters spoken per call
Setup and ongoing maintenanceInternal team or implementation partnerMore integrations, more complex workflows

The distinction that surprises most teams is that phone capacity is charged by how many calls run at the same time, while AI model costs are charged by how much is said. A business with 50 short calls per day has a very different cost profile from one with 5 long calls running simultaneously, even if the total minutes are similar.

Hidden costs that rarely appear in early estimates include script and prompt rework after the first real calls, transcript review to catch misheard names or addresses, handling calls the agent could not complete, delays in provisioning phone numbers in certain markets, and the time spent on compliance registration before outbound calling can start. These are real costs that show up in the second month, not the first.

For current telephony pricing, check current Teler pricing. For AI model costs, check the pricing pages of the speech and language providers your team selects, since those are paid directly to those providers and vary by model and volume.

Teams budget for the AI models and forget that phone capacity is charged by how many calls run at once, not by how many calls happen in a day. The first busy Monday can double a line they never modelled.

– Subhash Kalluri, Founder & CEO, FreJun

What does it take to set up an ai voice agent open source system?

Setup has two sides: what the business supplies and what the platform or framework supplies. Mixing these up is how projects stall.

The business supplies the phone number or numbers, the script or conversation design, access to the systems the agent must query (calendar, customer records, order system, ticketing system), someone to review early calls and flag errors, and the consent records required before outbound calling starts. None of these come from the framework.

The framework supplies the orchestration logic, connectors to speech and language services, and tools for testing conversation flows. A managed telephony layer like FreJun Teler adds the phone network connection, carrier routing, phone numbers across markets, call controls (transfer, hold, hang up, recording), and the ability for callers to interrupt the agent mid-sentence. The Teler developer documentation covers how these pieces connect.

Systems the agent must connect to

A voice agent that cannot look anything up is a very expensive hold message. Most deployments need at least one of these connections: a calendar or scheduling system (for appointment booking or reminders), a customer records system or CRM (customer relationship management software, the database that holds contact and account history), an order or inventory system (for status checks or changes), and a ticketing or support system (for logging issues or checking case status). Each connection takes time to build and test, and each one is a point where the agent can pause noticeably if the lookup is slow.

On the build-versus-buy question: building the full stack in-house is the right answer when a team has specific data-handling requirements that rule out third-party telephony, when the conversation design is genuinely novel and no existing framework fits, or when the team already has telephony engineers on staff. For most businesses, the open-source framework handles the orchestration and a managed telephony layer handles the phone network, because building carrier connectivity from scratch takes months and requires ongoing maintenance that has nothing to do with the AI. If you’re still deciding where to begin, our guide on where to start with open-source voice agents breaks down the first decisions in more detail.

What rules apply to ai voice agent open source deployments in the US?

US voice AI deployments face federal rules from two agencies. The table below covers the confirmed rules. This is descriptive information, not legal advice.

What the rule requiresWho it applies toWho is responsibleSource
Prior express written consent before making autodialed or prerecorded telemarketing calls to wireless numbers or prerecorded calls to residential linesAny business making outbound calls of this typeThe business placing the call, not the platformFCC TCPA Order, Federal Register 2012-06-11
An interactive opt-out mechanism must be available during every prerecorded telemarketing callAny business making prerecorded telemarketing callsThe business placing the callFCC TCPA Order, Federal Register 2012-06-11
No calls to numbers on the National Do-Not-Call Registry for telemarketing purposesAny business making telemarketing callsThe business placing the callFCC Telemarketing guidance
Permitted calling hours: no telemarketing calls before 8 a.m. or after 9 p.m. local time at the called party’s locationTelemarketers under the FTC Telemarketing Sales Rule (TSR)The business placing the callFTC Telemarketing Sales Rule, 16 CFR Part 310
TSR robocall restrictions apply to calls using AI-generated or voice-cloned audioAny business using AI voice in outbound telemarketingThe business placing the callFTC press release, March 2024
Specific disclosures, prohibition on misrepresentation, and honoring do-not-call requestsTelemarketers under the TSRThe business placing the callFTC: Complying with the Telemarketing Sales Rule

The key point in this table is that every obligation sits with the business placing the call, not with the telephony platform or the framework. For a broader primer on consent, recording, and disclosure requirements beyond this US-specific table, see our guide on compliance basics for AI voice applications. Using a managed service does not transfer the consent or registration requirement to that service. Inbound calls (where the customer calls you) carry fewer restrictions than outbound calls, but recording consent rules still apply in many US states.

What goes wrong after launch?

A 2026 analysis of 10,000 voice AI calls identified four recurring problems. The table below shows each one, why it happens, and what to do about it.

ProblemHow often it appearedWhy it happensWhat to do
Caller disruption in the first eight seconds once they recognize an AIAbout 34% of calls (Reddit, August 2026)Callers change behavior when they realize they are not speaking to a personDesign the opening to set expectations clearly and move quickly to a useful question
Interruption handling failures, including filler words treated as full interruptionsAbout 28% of calls (Reddit, August 2026)The system is not tuned to distinguish “yeah” from a real turn-changeTune the interruption sensitivity and test with real callers before launch
Noticeable silence during knowledge-base or tool lookupsAbout 22% of calls (Reddit, August 2026)External system lookups take longer than the caller expectsAdd a brief spoken acknowledgment while the lookup runs; optimize slow integrations
No clear path to a human when the agent cannot helpAbout 11% of calls (Reddit, August 2026)The handoff logic was not built or testedBuild and test the human-transfer path before going live, not after

These four problems account for the majority of calls that end badly, and all four are fixable before launch if the team tests with real callers rather than only internal testers. One honest limitation that applies to any platform, including Teler: the quality of the conversation depends almost entirely on the script, the language model, and the tuning the team puts in. The infrastructure can carry a bad conversation just as reliably as a good one.

When should you not automate this call?

An ai voice agent open source system is not the right answer in every situation. Be specific about when to keep a human on the phone.

Do not automate when call volume is too low to justify the setup time. If a business takes 20 inbound calls per week and each one is different, the engineering and tuning cost will not pay back for years. A human answering those calls is cheaper and better.

Do not automate conversations that are emotionally sensitive by nature: bereavement, mental health, serious medical news, or any situation where a caller is in distress. An agent that cannot read emotional tone will make these calls worse, not better.

Do not automate regulated interactions that require a licensed person. Financial advice, legal counsel, and certain healthcare conversations have rules about who can give them. An AI agent cannot hold a license.

Do not automate when your customer base has made clear they reject it. Some industries and some customer segments have a strong preference for human contact. Forcing automation on those callers increases churn, not efficiency.

Do not automate outbound telemarketing calls until the consent records, do-not-call scrubbing, and opt-out mechanism are fully in place. The FCC and FTC rules described above apply from the first call, not after a grace period.

Frequently asked questions

What is the best way to get started with an ai voice agent open source framework?

Start with a single, narrow call type that has a predictable script and a clear success measure, such as appointment reminders or after-hours FAQ calls. Pick one open-source framework (Pipecat and LiveKit Agents are the most actively maintained in 2026), connect it to a managed telephony layer for the phone network, and run at least 50 real test calls before going live. Expand to more complex call types only after the first one is stable. For a full walkthrough of this process, see our open-source voice agents starter guide for developers.

How much does it cost to run a voice AI agent?

Cost has two parts: telephony charges (phone capacity, per-minute usage, and number rental, paid to a telephony provider) and AI model charges (speech-to-text, language model, and text-to-speech, paid directly to AI providers). Both scale with call volume, but telephony capacity scales with simultaneous calls while AI costs scale with total minutes spoken. For current figures, check current Teler pricing and the pricing pages of your chosen AI providers.

What US laws apply to outbound voice AI calls?

Two federal frameworks apply. The TCPA (Telephone Consumer Protection Act), enforced by the FCC, requires prior express written consent before making autodialed or prerecorded telemarketing calls to wireless numbers. The FTC’s Telemarketing Sales Rule (TSR) sets calling-hour limits, requires disclosures, and prohibits calls to numbers on the National Do-Not-Call Registry. The FTC confirmed in March 2024 that TSR robocall restrictions apply to AI-generated voice calls. Both obligations sit with the business placing the call, not the platform.

Will callers know they are speaking to an AI, and does it matter?

Most callers will figure it out within the first eight seconds, and a 2026 analysis of 10,000 calls found that about 34 percent changed their behavior once they did (Reddit, August 2026). It matters because callers who feel deceived are more likely to hang up or complain. The better approach is to set expectations clearly at the start of the call. Some US states also have disclosure requirements for AI-generated voice in certain contexts, so check state law before going live.

Should we build the full stack ourselves or use a managed telephony layer?

Build the full stack yourself only if you have telephony engineers on staff, specific data-handling requirements that rule out third-party providers, or a genuinely novel use case that no existing framework fits. For most teams, the open-source framework handles orchestration and a managed layer like FreJun Teler handles the phone network, carrier routing, and phone numbers. Building carrier connectivity from scratch takes months and requires ongoing maintenance unrelated to the AI work. For a deeper look at what a production-grade telephony layer needs to handle at scale, see our guide on enterprise-grade voice infrastructure.

What happens when the agent does not understand a caller?

A well-designed agent asks a clarifying question once, then offers to transfer to a human if the answer is still unclear. The worst outcome is a loop where the agent keeps asking the same question. Build the human-transfer path before launch, test it explicitly, and make sure the transfer carries the call transcript and context so the human agent does not ask the caller to repeat everything they just said.

How long does setup actually take?

A narrow, single-use-case deployment with a managed telephony layer can reach a testable state in two to four weeks. Getting to a production-ready state, including real-caller testing, script tuning, integration with business systems, and compliance checks for outbound calling, typically takes six to twelve weeks for a small team. More complex deployments with multiple call types and deep system integrations take longer. The compliance registration step for outbound calling can add time if number provisioning in certain markets is delayed.

How does Teler fit into an open-source voice AI setup?

FreJun Teler is the layer between the open-source framework and the phone network. It handles carrier connections, phone numbers in 50-plus countries, call controls (transfer, hold, hang up, recording), and the ability for callers to interrupt the agent mid-sentence. It works with any speech-to-text, language model, or text-to-speech provider the team chooses, with no lock-in. The Teler documentation covers integration with common open-source frameworks. Teams can try it with prototyping credits and no credit card required.

Where is voice AI agent technology heading in the next two years?

The clearest trend is that the boundary between the framework and the telephony layer is narrowing. Platforms like LiveKit now bundle phone numbers, carrier connections, and observability alongside the orchestration code, which reduces the number of vendors a team needs to manage. Language models are also getting faster, which shortens the pause callers notice between speaking and hearing a reply. What is not changing is the compliance layer: consent, do-not-call, and disclosure obligations are set by law and will not be automated away.

What to decide before your first ai voice agent open source call

The decision a team faces is not which open-source framework to pick. It is how much of the surrounding infrastructure to build versus buy, and whether the call type they want to automate is actually a good fit for automation. The factor that should drive that decision is the ratio of setup cost to call volume: high volume, repeatable calls justify the investment; low volume, variable calls do not. If the call type involves sensitive topics, licensed advice, or a customer base that rejects automation, keep a human on the phone. For everything else, an ai voice agent open source framework paired with a managed telephony layer is a practical path to a working system in 2026.

About the Author: Subhash Kalluri is the Founder & CEO of FreJun, which he has been building since 2019. With over 8 years in voice communication and SaaS, he works on the problem of getting real-time AI onto live phone calls: raw media streaming, interruption handling, and carrier connectivity. That work became FreJun Teler, FreJun’s programmable voice API for developers building AI phone agents. Connect with him on LinkedIn.

Get started with FreJun today

In this article

Make every voice interaction count

FreJun is a real-time voice platform for automating calls and connecting AI bots.

Popular Articles
Stacked bar splitting a 150 ms media streaming latency budget across capture, encode, network, jitter buffer and playback

Quick Answer: Low-latency media streaming for voice AI means keeping mic-to-speaker delay under roughly 200–400 ms. Developers do this with...

Voice recognition SDK turning noisy audio into clean speech through FreJun Teler.

Quick Answer: A noise-resilient voice recognition SDK keeps speech recognition accurate in noisy environments by combining voice activity detection, real-time...

Voice API routing a live caller through FreJun Teler to an AI agent for a better caller experience.

Quick Answer: Voice APIs help businesses elevate live caller experiences by letting software place, receive, and control calls in real...