Executive Overview
Architecting full-duplex real-time audio streaming, WebSocket PCM buffer management, and Three.js 3D avatar lip-sync synchronization.
Key Architectural Takeaways
- Full-duplex WebSocket audio streaming reduces conversational turn latency under 300ms.
- Web Audio Worklet nodes handle zero-pop PCM 24kHz playback buffer queuing.
- Real-time audio frequency analysis drives Three.js morph target lip-sync visemes.
- Empathy-first conversational AI requires low-latency streaming and persistent user memory.
Conventional voice assistants rely on multi-stage pipelines: Speech-to-Text -> Text LLM -> Text-to-Speech synthesis. This turn-based architecture introduces 2-4 seconds of latency, creating a jarring, mechanical experience. By implementing full-duplex WebSocket audio streaming powered by Gemini Live API and synchronized with Web Audio API viseme processing, we achieved sub-300ms natural conversational audio flow with a 3D visual avatar.
The Latency Wall in Voice Conversational Systems
Design Goal: Fluid conversational timing where users can interrupt mid-sentence and receive immediate acoustic response without robotic delays.
Human conversation relies on subtle timing cues—overlaps, brief pauses, and back-channel acknowledgments happen in under 300 milliseconds. Traditional sequential voice stacks (whisper STT -> GPT text completion -> ElevenLabs TTS) break down because latency compounds at every layer. Full-duplex streaming over WebSocket transport eliminates discrete audio boundaries, allowing user input and AI audio output to flow continuously.
PCM Buffer Queuing with Web Audio API
Receiving streaming base64 PCM 24kHz audio chunks over WebSockets requires precise client-side timing to prevent audio crackles or starvation gaps. We built a custom AudioStreamHandler using the Web Audio API that queues incoming PCM float32 buffers and schedules contiguous playback timestamps.
Real-time Avatar Lip-Sync & Memory Garden Integration
As audio streams into the speakers, an AnalyserNode computes fast Fourier transforms (FFT) to extract instantaneous amplitude and frequency bands. These values map to 3D facial morph targets (visemes) on a Three.js Ghibli-inspired avatar in real time. Combined with a persistent Firebase memory engine, the assistant remembers past conversations, creating a deeply human conversational companion.
Topics & Domain Keywords
Voice AIGemini Live APIWeb Audio APIWebSocketsThree.jsReact