By Yuanshan Zhang, Lin Si, and Xinyuan Huang
Connecting the call is only the beginning. To complete a task, an AI agent must know when to listen, when to wait, when to press a key, and how to tell whether the phone system actually received its selection.
The hardest part may happen before a human ever answers
Imagine that you need to call a clinic to reschedule an appointment.
The call connects, but a person does not answer. The system plays a welcome message, asks you to choose a language, and then offers options for appointments, billing, prescriptions, and other requests. At the next level, it may ask for a member ID—or ask you to stay on the line until a representative becomes available.
For a person, this is merely a slightly tedious phone call. We listen to the menu, remember the right number, press the key, and then naturally continue the conversation when the hold music ends and someone answers.
For an AI agent, however, this is not simply a matter of hearing a number and pressing it.
The agent must first determine what it is interacting with: a live person, voicemail, a hold message, or an automated phone menu. It must know whether the menu has finished playing and whether it should speak, stay silent, or send a keypad tone. After pressing a key, it must keep listening to determine whether the call has moved to a new menu, entered a transfer, reached a person, or simply started replaying the same prompt.
A connected call is not necessarily a completed task.
That is why CALL-E began building IVR（Interactive Voice Response）navigation.
Teaching AI to use the phone system—not just speak through it
Traditional IVR systems—the familiar “press 1 for billing, press 2 to change an appointment”—still handle a large share of real-world calls. Clinics, insurers, pharmacies, airlines, public agencies, and many local businesses rely on them to route callers.
If an AI agent cannot get through those menus, even the most natural conversation skills never get a chance to help the user complete the task.
To solve this, CALL-E added DTMF (Dual-Tone Multi-Frequency) keypad support to its Calling Subagent, allowing it to send telephone keypad tones when an IVR menu clearly requests them. DTMF is an automated tool phones use to send keypad inputs—such as 1, 2, or #—to automated menus. Once this capability reached real telephone networks, we continued iterating on speech segmentation, menu understanding, keypad transmission, and outcome verification.
CALL-E’s IVR behavior can be summarized as a repeating loop:
- Listen to the latest information from the call.
- Determine whether the current state is a live conversation, an automated menu, a transfer queue, or something else.
- Choose among waiting, speaking, and sending a keypad tone.
- Observe the phone system’s response and decide what to do next.
When a complete menu clearly requests keypad input, CALL-E can choose the option that best matches the user’s goal. If the menu offers a clear route to a representative, it can prioritize reaching a person.
At the same time, CALL-E will not guess an account number, PIN, member ID, or other information the user has not provided. If the information required for the next step is unavailable, it will try to explain the purpose of the call or request a representative rather than inventing an input.
That sounds straightforward. But once the capability encountered real telephone networks, we quickly learned that making a tool available solves only the first layer of the problem.
Pressing the right key is only half the job
After launch, we separated an IVR interaction into three independent questions:
- Did the agent invoke the keypad tool when it should have?
- Did the selected key match both the user’s goal and the current menu?
- After the key was sent, did the receiving phone system recognize it and advance to the next step?
These questions are easy to conflate, but they have very different causes and require different solutions.
Sometimes the agent did not press a key when it should have
In some early calls, the agent interpreted “press 1” as a request for a spoken answer. It said the number aloud or continued waiting for the system to accept speech. In other cases, after several keypad attempts appeared to have no effect, the agent remained silent without explaining the task again or requesting a transfer to a person.
We therefore made the boundary for keypad use explicit: the agent sends a key only when the latest menu clearly requests keypad input, the menu is complete, and the correct selection can be determined. During the turn in which it sends a key, it does not speak at the same time, preventing speech and DTMF from interfering with each other.
Sometimes the menu the agent sees is incomplete
Real IVR menus are rarely a single clean, uninterrupted recording. A system may say, “Please listen carefully to the following options,” pause, and only then begin listing departments and numbers. Transfers, background noise, and language changes can introduce additional boundaries in the audio.
Telephone audio is transmitted and recognized as it arrives. The system uses voice activity detection (VAD) to decide whether the other side has finished speaking, and automatic speech recognition (ASR) to convert the audio into text. The problem is that VAD largely relies on sound and silence to divide speech into turns. A natural pause in an IVR recording can therefore be mistaken for the end of an utterance.
Instead of receiving the whole menu, the agent may first see only a fragment such as:
Press 3 for Hypoluxo in Congress in Boynton Beach. Press 4 for …
The rest of the option does not arrive until the next segment.
Consider a location directory that is playing the following menu:
For Deerfield Beach, press 1; for West Boynton Beach, press 2; for Hypoluxo, press 3; and for other locations, press…
To a person, this is obviously an unfinished list. On a real call, however, a long menu and the pauses between options may cause the audio to be divided into multiple segments. If voice activity detection decides too early that the system has finished speaking, the agent may temporarily receive only “for other locations, press…,” without the number. If it treats that fragment as a complete instruction, it may guess a key, start speaking too early, or incorrectly conclude that the menu does not contain a relevant option.
An even simpler example is a system that first says:
For service in English, please press…
and adds, after a brief pause:
three.
To a human listener, these audio segments form one sentence. In a real-time system, they may arrive as two separate inputs. Reliable behavior means recognizing that the action is present but its parameter is missing. Instead of guessing from the word “press,” the agent stays silent and waits until it hears the complete instruction, “press 3,” before sending the key.
Long prompts create another boundary. When an automated recording exceeds the maximum duration allowed for a single speech turn, the system may be forced to split it. We later increased that limit significantly, reducing the number of long menus cut off by a hard duration boundary. But a longer limit does not prevent VAD from ending a segment at a shorter pause. This remains a tradeoff between responsiveness and completeness: waiting longer provides more context, but can make ordinary human conversations feel sluggish.
These failures taught us that waiting is not inaction. It is a necessary IVR navigation behavior.
We therefore did not rely on speech-segmentation settings alone. We added a second guardrail at the agent’s decision layer: DTMF is allowed only when the agent has both a complete option and the explicit key associated with it. If the latest input ends with “please press,” “please select,” or an introduction to options that have not yet played, the agent treats the menu as incomplete, preserves the IVR state, and keeps listening. A brief silence does not mean the menu is over.
This prevents the agent from guessing 0, *, or another key that was never offered. It is still, however, a recovery strategy built on an incomplete transcript. If the remaining audio is never recognized correctly, the agent can wait or look for a route to a representative, but it cannot reconstruct a missing digit out of thin air. We are continuing to explore more reliable ways to combine menu fragments and reason about audio state.
Sometimes the agent chose correctly, but the key did not make it through the network
One real call illustrates the problem well.
CALL-E needed to contact an automotive service business to schedule a tire installation and vehicle diagnostic. After entering the IVR, the agent selected the correct options based on the menu it had heard. Twice, however, the receiving system responded with “invalid selection.” The call never reached the scheduling branch or a representative.
From the agent’s perspective, the decision was correct. From the task’s perspective, the menu was never passed.
This exposed an easily overlooked fact about IVR navigation:
Choosing the correct key and successfully delivering that key through the telephone network are two different problems.
A successful tool response tells us only that CALL-E initiated a keypad action. It does not prove that the receiving system accepted it. In a telephone network, DTMF is not a text message that says “the user pressed 1.” A key is encoded as a sequence of telephone events—with duration and end markers—using RTP telephone-event, then passed through the calling platform and network to the receiving IVR.
The timing of that sequence matters. If an event is too short, if multiple packets are emitted in a burst, or if consecutive digits are sent without enough separation, the receiving system may miss a key or interpret several keys as an invalid input. In other words, choosing the right key does not guarantee that it was sent in a form every IVR can recognize.
In an early group of failed calls, we repeatedly saw CALL-E produce a DTMF action while the receiving system still returned invalid or replayed the same menu. That pattern narrowed the problem from “did the model press a key?” to “did the receiving IVR recognize it?”
We then changed the transport-layer sending model. Event packets were paced according to real time, individual key events were given a longer effective duration, end markers were repeated, and distinct gaps were added between consecutive digits. Multiple keypad requests within the same call were serialized to prevent two sequences from interleaving. After these changes, the concentrated pattern of unrecognized inputs declined significantly.
Even so, improving the transport layer does not guarantee that every IVR will accept every input. We therefore distinguish three separate states: the system accepted the keypad request, the telephone layer completed transmission, and the receiving IVR confirmed progress. The first two cannot substitute for the third.
After sending a key, the agent must keep listening for a new menu, a transfer, or a live person. If it hears invalid or receives the same menu again, it treats the action as a navigation failure and changes strategy instead of blindly repeating it. Our observability follows the same path, allowing us to distinguish rejected requests, interrupted transmissions, and keys that were sent but not recognized—rather than collapsing them all into a generic “keypad failure.”
Real calls revealed the problem layer by layer
During the first production observation period, the tool itself was already triggering reliably, yet some invocations still failed to advance beyond the menu. Other calls successfully sent a key but then failed to interpret the resulting hold or transfer state.
As our evaluation criteria matured, we began measuring three dimensions separately:
- Tool invocation: When keypad input was clearly required, did the agent act?
- Key accuracy: Did the key sent by the agent match the user’s goal and the options in the latest menu?
- Post-invocation progression: Did the receiving system stop replaying the same menu and move to a new menu, a transfer, a live person, or another meaningful next step?
This separation showed that similar-looking failures to “get through the menu” could have completely different causes.
An inaccurate key, for example, may come from two very different sources. At the decision layer, the agent might choose the wrong option, send a key the menu never offered, or repeat a key from the previous menu after the call has already moved on. At the perception layer, ASR may fail to recognize a number, or VAD may end the segment too early at a pause, causing the agent to act on only half of the menu.
A post-invocation failure, by contrast, typically appears as an invalid-input response, a replay of the same menu, or no observable state change after the key. Diagnosing it requires two kinds of evidence. The agent’s decision record answers whether the chosen key was correct; the keypad event and the receiving system’s next prompt answer whether that key was received. Only when the first is correct and the second fails should the telephone transport path become the focus of the investigation. A failed task alone is not proof that the agent never sent a key.
We also learned that evaluation criteria must remain aligned with product policy. If the product is designed to prioritize a clearly offered route to a representative, an evaluator should not mark the decision incorrect merely because the agent did not choose the department whose name most closely resembled the user’s request.
Improving a voice agent means improving not only the agent, but also the way we measure it.
How we made CALL-E more reliable
Every class of production failure prompted us to tighten a decision boundary.
First, we made the distinctions among IVR, live humans, voicemail, and transfer queues more explicit. A recorded greeting is not necessarily voicemail. “Please stay on the line” does not mean a person has answered. Sending a key means only that the agent initiated a navigation action; it does not mean the call has reached a person.
Second, we addressed premature actions at both the audio and decision layers. At the audio layer, we increased the maximum duration of a speech turn to reduce hard splits in long menus. At the decision layer, a complete menu became a prerequisite for keypad input. Even if VAD still creates an early endpoint at a pause, the agent should recognize an incomplete fragment and wait for the rest instead of guessing.
Third, every key is bound to the latest complete menu. The agent cannot send an option that was never offered, invent missing identifying information, or keep sending 3 merely because the original goal mentioned it after the call has moved to a different menu. It also does not speak and send DTMF in the same turn.
Finally, we made the response after a key part of the state transition. Navigation advances only when the receiving system enters a new menu, begins a transfer, or connects to a person. If the system repeats the menu or reports invalid input, the agent stops repeating the same action and instead waits for a complete prompt, looks for a route to a representative, or ends safely when no viable path remains. This makes transport, perception, and decision failures independently observable.
We built continuous evaluation around these behaviors. Real calls reveal problems, structured metrics distinguish among missing actions, inaccurate keys, and post-invocation failures, and regression cases tell us whether a change breaks a different scenario.
This was not a one-time feature launch. It is a continuous learning loop.
How we measure whether IVR is actually improving
A prompt change performing better in a handful of examples does not mean the IVR capability has truly improved. To make each iteration comparable, we combine offline regression testing with monitoring of real calls.
Building test sets from real failures
Our test sets come from patterns that repeatedly appeared in real calls, but they do not ask the agent to memorize a fixed phone tree. Each case provides only the task goal, what the agent has heard so far, and the necessary user information. We then observe whether the agent can choose the correct action without knowing the remaining menu structure in advance.
The test sets include menus split across multiple segments, options that have not finished playing, transitions to deeper menu levels after a key, silence during transfers, rejected inputs or repeated menus, and recorded audio that suddenly gives way to a live person.
At important points in the call, we use checkpoints to determine whether the agent:
- invoked DTMF when keypad input was clearly required instead of speaking the number;
- waited for complete menu information and selected the option that matched the user’s goal;
- recognized a new menu, transfer, or waiting state after sending a key instead of repeating the previous action; and
- stopped rather than guessing when an account number, PIN, or other required information was unavailable.
These checks evaluate behavior rather than requiring the agent to produce a particular sentence. The same cases can be run against different versions, allowing us to test whether a prompt or runtime change resolves an old failure, and whether that fix breaks a different IVR scenario. Cases that depend on real audio, timing, and DTMF transmission are also tested through the actual telephone network. Offline regression and production monitoring answer different questions, and we use both.
Key takeaway: A reliable voice agent must understand silence, too
Voice AI is often judged by how natural it sounds. A natural voice, appropriate pacing, and smooth interruption handling all matter.
But IVR navigation taught us that a reliable phone agent must also know when to send a key, when to recognize that a key did not work, when to choose a different path, and when it can finally be confident that a person is on the other end of the line.
CALL-E’s goal is not merely to let AI place a call or make the conversation sound more human. We want it to navigate the imperfect phone systems that exist in the real world, keep acting toward the user’s goal, and actually complete the task.
One task. One call.
Behind every call is a deeper understanding of the real world.
For product support, contact the CALL-E support team: support@heycall-e.com
Want to follow the latest CALL-E updates or share a real-world calling scenario with us?
Join the CALL-E Discord community: https://discord.gg/SDcGdhgRzj
