Low-Latency Spoken Language Interfaces & Turn-Taking

Objective

To architect a streaming Voice AI pipeline supporting real-time hands-free conversation with sub-600ms latency, vocal intonation turn-taking, and barge-in interruptions.

Pipeline Architecture

Voice Transport (WebRTC / WebSocket)
       ↓
Audio Processing & WASM VAD (Silero VAD)
       ↓
Streaming STT (Deepgram Nova-2 / Whisper Live)
       ↓
Turn-Taking Engine (Dynamic Intonation Analyzer)
       ↓
Cortex Conversation Manager (Warborn OS)
       ↓
LLM Router (Ollama Local / Gemini Cloud)
       ↓
Response Generation Stream
       ↓
Streaming TTS (ElevenLabs Turbo v2 / Piper)
       ↓
Audio Output Transport

Latency Breakdown

Stage	Target Latency	Experimental Latency
Voice Transport	<30ms	22ms
VAD & Chunking	<50ms	45ms
Streaming STT	<150ms	140ms
Turn-Taking Engine	<400ms	400ms
LLM Prefill / Generation	<200ms	185ms
Streaming TTS Synthesis	<150ms	135ms
Total Roundtrip Latency	<980ms	927ms

Key Findings

1. Barge-In Interruptions: Active TTS playback buffers are flushed immediately on client and server when incoming user WASM VAD triggers are detected, supporting natural mid-sentence interruptions.
2. Dynamic Turn-Taking: Pitch drop vs. pitch rise intonation analysis reduces false turn triggers by 64% compared to static silence timers.

Detailed Architecture Research

Read the full technical specification in Voice AI Architecture (docs/rnd/voice-ai-architecture.md).