Jev VAD: Turn Detection That Understands What You Say

Splitting turn-taking in an Agora Conversational AI agent between a fast acoustic gate and TypeSafe’s Jev, starting from a quickstart recipe.

Voice agents are very good at noticing silence. They’re much worse at knowing what the silence means.

Say “So what I want to know is…” and pause while you look for the next word. In a silence-driven turn detector, that pause can be enough to close the turn and send an unfinished thought to the agent. In a basic VAD-driven setup the language model may never see the problem: the turn was already closed because the audio pipeline interpreted the pause as end of speech.

Silence doesn’t tell you whether someone has finished their turn. A speaker may be searching for a word. An “uh-huh” during the agent’s response may only be a backchannel, and speech picked up by the microphone might not be directed at the agent at all. Audio activity can tell you that something happened; it gives you very little information about the speaker’s intent.

What I built

Instead of asking one detector to solve the entire problem, I split turn-taking into two jobs. Audio still detects that someone has started speaking. Once text begins arriving from speech recognition, Jev decides whether the speaker has actually yielded the floor.

The project is called Jev VAD, although Jev itself isn’t doing voice activity detection. The VAD-like part is a lightweight acoustic gate in the browser; Jev handles the semantic decisions that conventional VAD can’t make. It runs on Agora’s Conversational AI Engine, and it’s open source.

Jev is TypeSafe’s System One model. Instead of generating text, it takes application state and focused questions and returns typed probabilistic decisions. That makes it useful here because I don’t need another model to write anything. I need a few numbers the application can branch on.

Architecture at a glance

mic
 ↓
acoustic gate (browser)
 ↓
manualSOS
 ↓
Agora ASR → streaming transcript → /judgeTurn → Jev
                                                 ↓
                         turn_complete · holding_floor ·
                         addressed_to_agent · taking_floor
                                                 ↓
                         browser: manualEOS, interrupt(), or keep listening

The short version of the design is open fast, decide late.

Speech onset and end-of-turn detection have very different latency requirements. For onset, I care more about speed than intelligence, because waiting too long risks clipping the first word. The browser watches microphone energy against a moving noise floor and opens the turn after roughly 110 ms of sustained activity. It also drops obvious impulsive bursts, like a single click or thump, before they open anything.

Some non-speech sounds can still get through the gate and open a turn. That is intentionally cheap: if speech recognition produces no text, the controller never closes the turn or sends anything to the agent.

End of turn is where Jev comes in. As the streaming transcript evolves, the browser sends it to Jev, which returns four scores in one call:

• turn_complete: has the speaker actually handed over the floor? "So what I want to know is…" hasn't.
• holding_floor: are they explicitly asking for more time, as in "hold on, let me type something"?
• addressed_to_agent: is the latest speech intended for the assistant? "Hey, can you hold my coffee?" usually isn't.
• taking_floor: while the agent is talking, is this a real interruption or just a backchannel like "right"?

In my test sessions the Jev call usually came back in roughly 300 ms. That is too slow to sit in front of speech onset, but perfectly usable once speech recognition is already streaming words, which is why the two jobs are split the way they are.

The client still measures silence, but it no longer treats a fixed silence duration as the decision. Instead, silence gradually relaxes the score Jev needs to reach before the client closes the turn.

Start from the recipe with the Agora CLI

Jev VAD is available in the Agora recipes catalog. Once the CLI is authenticated, agora init creates a local copy and binds it to one of your Agora projects.

1. Install the CLI and log in.

curl -fsSL https://dl.agora.io/cli/install.sh | sh
agora login

2. Find the recipe.

agora recipes list --type ai
agora recipes show jev-vad

3. Create your copy.

agora init my-jev-vad --recipe jev-vad

It can create or reuse an Agora project, clone the recipe, and write the project credentials into the local environment file (server/.env). Then it prints the next steps.

4. Add your TypeSafe key and start it.

cd my-jev-vad
echo 'TYPESAFE_API_KEY=<your-key>' >> server/.env
bun run setup
bun run dev

Open http://localhost:3000, click Start Conversation, and start talking. Jev's scores move in the side panel as you speak.

Already have a ConvoAI app?

The recipe also has a smaller drop-in path for an existing Conversational AI app. It keeps the two decisions that matter most for turn-taking, completion and interruption, and leaves out the demo’s side-talk handling, patient mode, telemetry and tuning UI.

There’s also a copyable coding-agent prompt at the top of the recipe guide, so the same instructions can be applied to an existing project instead of starting from the demo. The rest of this post describes the full demo.

What changed from the Agora quickstart

I started from the Agora Python quickstart rather than building the agent stack from scratch. It already provides the Next.js client, the FastAPI backend, the transcription flow and the agent lifecycle. Jev VAD changes the parts responsible for deciding when a user turn opens, closes, or interrupts the agent.

1. Switch Agora’s turn detection to manual (agent config)

In the backend’s agent configuration, start of speech and end of speech are set to manual, and automatic interruption is disabled:

turn_detection={
    "mode": "default",
    "config": {
        "start_of_speech": {"mode": "manual"},
        "end_of_speech":   {"mode": "manual"},
    },
},
interruption={"enable": False, "disabled_config": {"strategy": "append"}},

Agora still handles the realtime session, transcription, agent execution and speech output. What it no longer owns in this demo is turn control: the browser explicitly sends manualSOS, manualEOS and interrupt() through the Agora client toolkit.

With interruption disabled, strategy: "append" keeps speech captured while the agent is talking instead of throwing it away before the client has decided what it means.

2. Add one judgment route (API routes)

/judgeTurn receives the current transcript along with enough conversational state to interpret it: recent turns, the agent's last message and current state, mic silence, and some information about how the partial transcript has changed. Jev evaluates all four questions against that state in one call. The browser reaches it through the same /api/* path as the rest of the app, so the TypeSafe key stays on the server.

Normal transcript changes are debounced by about 150 ms before calling /judgeTurn. During a potential interruption the controller evaluates partials immediately, which is why a real "wait, wait" can stop the agent quickly.

The Jev questions define what the model judges; thresholds and timing determine how aggressively the client acts on those judgments.

3. The browser state machine (conversation UI)

The browser owns the state machine. Jev returns probabilities; it never calls Agora directly. Stripped down, the branches look like this:

complete + addressed + quiet   → manualEOS
side-talk or backchannel       → keep listening
taking_floor                   → interrupt()

The completion threshold starts at 0.72 and relaxes toward 0.55 as mic silence approaches three seconds. A separate quiet-confirmation timer prevents a high semantic score from ending the turn while the user is still audibly speaking, and five seconds remains the hard fallback. When Jev says the user is holding the floor, the threshold doesn’t relax and the fallback stretches to twelve seconds.

In practice a user turn produces several judgments rather than one, usually three to six. The partial transcript changes as the user speaks, and the client may ask again during silence before closing the turn.

I also added a panel that shows the four scores live, and a Tune drawer for changing thresholds and timers mid-call without restarting anything. Most of my tuning happened there, watching the numbers move while I talked.

What happened in actual calls

These are scores from my test sessions, not fixed outputs you should expect from Jev.

The test I cared about most was a long pause in the middle of an unfinished sentence. I said “So what I want to know is…”, stayed silent for three seconds, and finished with “…did you jump in?” During the pause turn_complete sat around 0.05, so the turn stayed open even as the threshold relaxed. The agent answered about a second after the question.

I also wanted to see what happened when I explicitly asked for time. “Hold on, let me type something” scored 0.8 to 0.9 on holding_floor, so the agent stayed quiet while I typed and then answered the question I asked afterwards.

For backchannels, I asked the agent to walk me through how a call works and reacted the way I normally would while it talked. “Right.” scored 0.03 on taking_floor and "Okay." scored 0.04; both were held and the agent kept going. "Wait, wait. Let me stop you there." scored 0.97, and the agent stopped within about a second.

Side-talk was the last case. I turned away and said “Hey, can you hold my coffee, please?”, which scored 0.09 on addressed_to_agent, so nothing was sent to the agent. "Sorry, that wasn't for you." scored 0.79, and the agent replied that it had noticed.

Wrapping up

The useful part of this experiment wasn’t replacing VAD with another model. I still use audio to open the turn. What changed was treating end of turn as a judgment about the conversation rather than a timer attached to silence.

If you want to experiment with the same approach, the full demo and the smaller drop-in recipe are both available below.

• The recipe: recipes.agora.io/recipes/jev-vad, or agora init my-jev-vad --recipe jev-vad
• The demo source: convoai-jev-vad
• The Agora CLI: AgoraIO/cli
• The base recipe: agent-quickstart-python
• Agora Conversational AI: docs.agora.io
• TypeSafe Jev: typesafe.ai