Build Your Own AI IVR Agent

A phone, a conversation and a specific task. Connect telephony with your local AI models to build an assistant that listens, asks for missing information and prepares a request with the caller's confirmation.

Imagine calling a support service and saying, “I want to report a problem with a device.” The assistant asks which item you mean, accepts a correction and summarizes the information before continuing. Conversation becomes the input to an application you can design around your own service.

An AI IVR agent combines telephony, speech recognition, language interpretation and speech synthesis. You can use this pattern to collect requests, retrieve authorized information or prepare work that a person will continue. The key is connecting those components to a clear procedure.

This guide walks through that construction: choosing the first task, connecting the call to your models, controlling confirmations and measuring the wait. We draw on a lab implementation with Asterisk, Pi and Qwen, with a coding prompt for building your agent and charts to help identify improvements.

We start with models already running locally. The goal is a prototype that collects and confirms a test request. You can then design its connection to your management application. This guide covers the architecture and application rules; it does not deliver a configured phone system or a production deployment.

1. Choose a task your agent can complete

Start by describing what a call should accomplish. In our example, the result is a proposal containing a category, an affected item and a description, confirmed by the caller.

This small contract lets you test the logic before connecting a phone. Enter a request as text, check which field is missing and introduce a correction. If that path does not work, adding voice will introduce more variables to diagnose.

To add a real lookup or write operation later, define an adapter with the exact operations the agent needs. Authentication, authorization and result verification belong in that integration.

2. Organize the components: telephony, voice and agent logic

Separate responsibilities so you can check each one independently.

These models identify the stack used in our tests, not a requirement for your agent. You can choose other locally running ASR, language and TTS models with compatible interfaces; check their input formats, output contracts and capabilities before connecting them. The measurements below do not establish the performance of those alternatives.

These names identify the configuration reviewed; the results do not compare models or isolate the performance of their weights. Qwen separately documents its speech recognition and speech synthesis families. Pi provides the agent runtime; application code establishes its restrictions.

Caller → Asterisk → voice detection → ASR → text
                                           ↓
                                  Pi + language model
                                           ↓
                                    intent and fields
                                           ↓
                                      state machine
                                     ↙             ↘
                              ask / correct     confirm data
                                     ↑             ↓
                                     └─ response ← adapter
                                            ↓
                              prepared prompt or TTS → caller

In this integration pattern, Pi receives the state and utterance and returns a bounded structure. It has no general-purpose file, shell or browsing tools. The program decides what to ask next and when an operation is allowed.

Model instructions do not replace application rules. A skill can describe a procedure; the application must enforce it. In the interpreter reviewed, automatic skill loading is disabled and a specific instruction is paired with a structured-output tool.

3. Connect the call to the conversation loop

Once the basic logic is defined, bring it into a call. In the reviewed architecture, Asterisk handles telephony and the application exchanges audio through a WebSocket channel. The integration follows this sequence:

1. Open one session per call. Store its fields, state and current revision. Each caller needs a separate context.

2. Wait for audio readiness. A call arriving and a media connection becoming available are separate events. Play the greeting when the channel can receive it.

3. Detect an utterance and transcribe it. Voice activity detection delimits the audio sent to ASR. Send the recognized text to the interpreter with the current question and state.

4. Turn the result into a transition. Pi returns intent and fields; the state machine decides whether to collect information, correct it or request confirmation.

5. Play the response and listen again. Use prepared audio for fixed questions and TTS for variable content. Cancel pending work and timers when the caller hangs up.

Debug each connection with a known input: audio to text, text to fields, then text to speech. Integrate these stages with the call manager once all three work. Their individual timings help locate delays without automatically blaming the language model.

4. Design a dialogue that accepts corrections

This fictional example is unrelated to the use case behind the measurements. Devices A and B illustrate the design; they do not reproduce a call or represent an organization, industry or location.

Assistant: I am an automated assistant. What issue would you like to report?

Caller: I want to report a problem with a device.

Assistant: Which device is affected?

Caller: Example device A. It does not respond when I turn it on.

Assistant: I will prepare an issue report: example device A does not respond when switched on. Is that correct?

Caller: No, I mean example device B.

Assistant: I have changed the affected item to example device B. Do you confirm these details?

Caller: Yes, correct.

Assistant: The test request is prepared. It has not been submitted to the management system.

A correction must invalidate the previous confirmation. “Yes” applies only to the current summary; silence must not count as consent. If a required field is missing, the application asks again.

An integration that creates records must verify the backend result before announcing creation and return a reference supplied by that system. Our preparation test does not demonstrate that write operation. Keeping the distinction clear prevents a convincing conversation from becoming a false promise.

The same principle applies to lookups. The assistant explains what the authorized read returns. The model should not invent a missing resolution date or grant access simply because someone knows a reference.

5. Copy this prompt to build your own agent

If speech recognition, the language model and speech synthesis are already running locally, ask a coding LLM to build the application that connects them. Specify the dialogue, each call's state and the conditions for taking action.

Fill the bracketed fields with your service information and paste the following prompt into your coding assistant. Do not include passwords or personal data. Leave unknown fields open so the assistant can clarify them before implementing that connection.

Act as a software engineer specializing in voice agents.
Build the code for an IVR agent for my project. I want a functional,
testable application, not just an explanation.

STARTING POINT
I already have three model services running locally:
- ASR: [URL, model, protocol, input audio format and response format].
- LLM: [URL, model, protocol and structured-output support].
- TTS: [URL, model, voice, response format and audio sample rate].
The environment variable names used to authenticate are:
[variable names, without their secret values].
Do not install, download, train or start models.
Call the existing services directly. Do not introduce intermediary
layers or design infrastructure or deployments.

MY AGENT
- Task: [what the agent must be able to complete].
- Conversation language: [language].
- Required fields: [fields and validation rules].
- Allowed actions: [closed list of operations].
- Help option: [what to offer if the task cannot be completed].
If I need a starting point, use a generic issue report with a category,
affected item and description. Test data must be fictional, without
real organizations, industries or locations.

PROJECT ENVIRONMENT
Respect the existing repository's language and conventions.
For a new project, use Node.js and TypeScript. Use Pi as a restricted
interpreter if compatible with my LLM endpoint. Check compatibility;
do not invent SDK functions, methods or supported capabilities.
If essential API information is missing, ask for its contract or a
sample response without sensitive data. Meanwhile, continue with
modules that do not depend on that information.

AGENT LOGIC
1. Separate sessions, state machine, interpretation, voice and actions.
   Each call needs its own fields, bounded history, timers and tasks.
2. The LLM returns only intent, fields and a proposed interpretation.
   Validate its output against a schema. Give it no general-purpose
   shell, file or network-request tools.
3. The application chooses the next step: collect information, clarify,
   present the summary, confirm or cancel. Ask only for missing data.
   Never invent values or fill uncertain fields by assumption.
4. Bind explicit confirmation to the exact revision of the summary
   the caller heard. A correction invalidates confirmation and requires
   a new summary. Silence and stale confirmations authorize nothing.
5. Begin with a simulated adapter: return a proposal and state that it
   has not been submitted. Define an interface for later real actions
   with validation, authorization and idempotency. Do not connect real
   business services without their contracts.
6. Announce success only after the relevant operation confirms it.
   If its outcome is uncertain, explain that uncertainty and avoid
   automatically repeating a write.

VOICE AND TELEPHONY LOOP
- Connect incoming audio, utterance detection, ASR, interpretation,
  the state machine and the TTS response.
- Normalize audio to the formats my services actually support. Do not
  assume a codec or sample rate without checking.
- End an utterance both when silence is received and when audio blocks
  stop arriving.
- When the caller interrupts, stop queued audio, cancel processing
  where possible and discard stale results.
- Reuse fixed prompts. Never place responses containing caller data
  in a shared prompt catalog.
- Make waiting time, clarification attempts and call duration limits
  configurable. Release the session and cancel its tasks on closure.
- Prepare an adapter for an already available Asterisk installation:
  [connection contract and configuration variable names].
  Wait for the audio channel to be ready before playing the greeting.
  If the connection is unavailable, retain a simulated transport and
  identify real telephony as pending verification.
- If human assistance is configured, distinguish a transfer attempt,
  an answer and an audio connection. Never invent destinations or
  mark a transfer complete merely because it was initiated.

DELIVERY AND VERIFICATION
Deliver clear modules, complete files, environment-based configuration
without secrets, and a README for running the agent against my local
services. Implement the text flow first, then voice, then telephony.

Include tests for missing fields, corrections, stale confirmation,
silence, interruptions, model failure, uncertain action outcomes
and isolation between sessions.

Measure ASR, interpretation, actions and TTS separately. When a real
call is available, also measure from the end of the utterance to the
first audible part of the useful response. Do not confuse generating
audio with playing it. Do not log recordings, transcripts, credentials
or personal data by default.

Run the checks the environment allows and state which ones you ran.
Distinguish simulated services, real local models and real telephony.
Never invent latency figures, successful outcomes or passing tests.

The prompt asks for the agent itself: the models already exist and are reached through their interfaces. Adapt the task and fields while retaining confirmation, isolation and action-control rules. Generated code needs review and testing against your services before it handles calls.

6. Make the agent listen, wait and handle interruptions

1. Prepare messages that never change

Greetings, menus and fixed questions can be generated before calls arrive. The reference pattern loads a fixed-prompt catalog and checks both text correspondence and integrity. Responses containing variable data follow a different path.

This removes the need to generate those fixed messages during a call. It is a design property, not a measured saving in seconds: this test contains no A/B comparison of the greeting. Audio containing personal data must not enter the shared catalog.

2. Detect silence that never arrives

Waiting only for silent audio samples can leave a turn open if transport stops sending packets during a pause. The reviewed detector combines sample analysis with a timer for missing new blocks.

Test a pause containing silent blocks and another containing no blocks at all. Both need a defined outcome.

3. Cancel stale work when someone interrupts

Barge-in involves more than reducing volume. Flush queued audio, cancel processing where possible and discard results from an utterance that has been superseded.

Asterisk's WebSocket driver documents FLUSH_MEDIA for discarding queued audio. The application must also identify which generation of work is still current. Asterisk documentation.

If the caller corrects an item identifier while the previous response is being generated, that response must not reappear when inference finishes. A revision counter and cancellation signals can enforce this rule.

4. Make help options explicit

The flow retains keypad options and a request for human assistance. Exhausting clarification attempts offers help; it must not be confused with a completed transfer.

Dialing the destination proves only that a transfer was attempted. Success requires an answer and an audio connection. This in-memory test does not validate a human transfer, and we do not count one as achieved.

5. Separate acknowledging a wait from responding faster

A short waiting message can reassure callers that the call is still active. Prepared questions reduce dependence on inference. Neither decision eliminates the time needed for a variable response.

Distinguish the first audible acknowledgment from the first useful answer so you do not optimize a metric that misses the caller's actual need.

7. Measure response time and improve with evidence

Once the flow works, record each stage's duration and check the data it retains. The following lab measurements illustrate how to turn testing into decisions about your agent.

What we tested and what we did not

Chart scope. Twelve utterances from a synthetic caller across two scripts, evaluated on September 17, 2026. The models are real; telephony transport and test persistence are replaced in memory. These are not twelve human calls or a load test.

The evaluator uses the project's call manager, voice detection, recognition client and interpreter. It feeds synthetic audio in 20-millisecond PCM frames. Models and the lookup integration remain real, while telephone transport and test storage use in-memory implementations.

The scripts contain four and eight utterances respectively. The reviewed sample contains 13 recognition segments: 12 completed and one canceled when superseded. Timing charts include the 12 completed segments; the canceled one is declared and excluded. For the split turn, we show the completed segment, not the total duration of all attempts.

We measure ASR request duration and Pi interpretation duration from the client. They can include transport, queues and service processing. These are neither pure GPU timings nor the full wait between a caller finishing speaking and hearing the answer.

This sample has no complete measurement of TTS, first audible response, jitter, packet loss, energy consumption, cost per call or concurrency. It also has no group of human callers with different accents. We do not replace those missing measurements with estimates.

The run reports do not sufficiently capture hardware, load and effective server configuration. We therefore do not attribute the figures to a particular GPU or context limit, or present them as a hardware comparison.

Where the wait accumulates

Figure 1. Client measurements by stage. Separate bars, not a sum representing total latency. Small sequential sample; one completed segment per turn.

Recognition has the higher median, although interpretation also contributes substantially in some turns. The next optimization effort should measure both paths separately: audio duration, queueing, processing, request length and generation.

We do not publish a p95 as though it characterized a service. A percentile can be calculated from twelve observations across two scripts, but it would be a weak operational signal. The full distribution, median and range better describe what we observed.

Nor do we claim a percentage improvement over the initial test. Segment handling and audio verification changed during review, so the two runs do not form a controlled speed comparison.

Real-time factor helps interpret ASR timing

Real-time factor, or RTF, divides recognition time by the duration of the submitted audio:

RTF = ASR request duration / audio segment duration

An RTF of 1 means the request takes as long as the segment. It is a reference point, not a standalone measure of conversational fluency.

Figure 2. Median RTF across the twelve completed requests is 3.37. Submitted segments last 1.22–4.98 seconds. The denominator is the actual ASR input duration, not the edited conversation audio.

Every segment took longer to recognize than its own duration. This identifies an investigation priority for this particular path. It does not establish the model's performance on any server or reveal how many simultaneous calls a GPU can handle.

Check dialogue and recognition separately

Figure 3. Per-turn checks. These are different criteria applied to the same sample, not independent production reliability percentages.

Ten utterances met the evaluator's word error rate threshold of WER ≤ 20%. Eleven retained the critical terms specified in the script. All twelve passed checks for interpreted fields, dialogue state, response content, confirmation and technical audio validity.

WER compares the transcript with reference text by counting substitutions, deletions and insertions. This evaluation normalizes case, accents and some numerical expressions. Its results should not be compared directly with a benchmark using different normalization or a different corpus.

A turn can fail text matching while retaining enough context to reach the expected state. That explains the difference between completing the flow and meeting every criterion; it does not make recognition errors irrelevant. A misrecognized identifier or reference can require repetition even when the rest of the dialogue looks correct.

Output speech was also transcribed back into text automatically. After the verifier was reviewed, responses in all twelve turns fell within its threshold. Verification uses the same recognition system: it is not an independent human listening assessment and does not certify the exact pronunciation of every proper name.

Both scenario reports retain a failing strict overall result. A language-model judge finding the dialogue reasonable must not erase that result. Each signal answers a different question.

8. Validate the whole path before opening it to callers

To reproduce this method, start with a bounded procedure, required fields, allowed actions and verifiable results. Write scripts covering confirmation, correction, silence, interruption, cancellation and backend failure.

1. Version the test. Fix the code, models, parameters, audio and normalization. Record the effective server configuration, not just the intended one.

2. Measure independent stages. Record ASR, interpretation, adapter, TTS and playback start and finish times. Include cancellations and errors.

3. Retain failures. Excluding a canceled request from the completed-request distribution is legitimate if declared. Removing it from the accounting is not.

4. Check data and actions. Compare fields, current confirmation, executed operation and spoken outcome. Saving a proposal is not creating a real request.

5. Move to real telephony. Repeat with people, noise and SIP/RTP transport, measuring from the end of speech to the first audible part of the useful answer.

6. Increase load gradually. Measure distributions, errors and queues under concurrency, with limits agreed before testing. Shared models must not mean shared call state.

The charts show timings, audio durations and quality checks. They contain no recordings, recognized text, addresses, record references, session identifiers or internal infrastructure.

Your first agent: a useful task and a clear next step

This pattern is a product hypothesis for repeatable tasks: preparing generic issue reports, retrieving authorized status information or collecting details before a person takes over. Its commercial appeal is completing those steps with less friction; this sample does not quantify staffing savings, satisfaction or reduced abandonment.

For a team considering building or buying such a system, I would request three demonstrations: a correction invalidating prior confirmation, a failure that does not announce nonexistent success, and a complete waiting-time measurement under the expected load.

Your first version can handle one task: collect the fields, accept a correction and return a confirmed proposal. Then add an authorized read, connect a real action and test the complete telephone path. Each stage adds a capability you can verify before expanding the service.

What would you give your first phone agent: collecting requests, checking status or preparing a handoff to a person?