A real-time voice AI system built from raw protocols rather than a framework: a hand-rolled orchestration layer over LiveKit's WebRTC transport, with streaming STT/LLM/TTS, live barge-in, cross-session memory, and RAG-grounded answers — plus a separate batch studio for emotionally-directed, multi-character story narration.
curl https://livekitdemo.app-me.online/healthTwo independent pipelines share one client and server. Live Conversation is latency-bound and stays on a streaming path end to end; Script Reader is a stateless batch call with no latency budget, which is what lets it use a materially more expressive TTS model the streaming path can't.
Not a framework demo — the hard parts of a voice pipeline, built from source protocols and live-verified against real vendor accounts.
Deepgram streams interim/final transcripts; Claude streams a response with real multi-turn history and a system prompt tuned to coach punctuation-based pacing into the generated text — the actual source of natural pauses, not a post-process.
A new transcript arriving mid-response aborts the in-flight LLM/TTS and clears queued audio immediately. Interruptible for real, not simulated.
At session end, Claude reduces the transcript into a merged, deduplicated fact list (corrections replace, not append), stored in Redis keyed by LiveKit identity with a 180-day TTL, and injected into the next session's prompt.
Each turn embeds the human's utterance (Voyage AI) and runs a vector search over a hand-authored knowledge base (Postgres + pgvector), folding matches into the prompt. Retrieval failures are caught so a KB hiccup never breaks a turn.
Redis-backed session bookkeeping so a resume call survives a token-server restart; falls back to an in-memory store for zero-setup local dev.
A separate batch studio: paste a story, let Claude mark it up with tone/voice/emphasis tags itself, then synthesize through ElevenLabs' v3 model, which reads bracketed performance tags as genuine direction.
One structured JSON line per turn marking speech-start, STT-final, LLM-first-token, LLM-complete, TTS-first-byte, and TTS-first-frame.
A headless harness joins real sessions via @livekit/rtc-node and publishes real synthesized speech: p50 first-turn round-trip 4169ms across 24 concurrent turns, plus a cross-restart memory-survival check.
The kind of thing that doesn't show up in a framework tutorial — found with real evidence (ffmpeg/ffprobe, source diffs, trace logs), not guessed.
| Symptom | Root cause | Fix |
|---|---|---|
| TTS produced no audio, no error anywhere | ElevenLabs' free-plan voice restriction surfaces as a normal message on an already-open socket, not a WebSocket error | Switched to a premade account voice; documented the gotcha |
| Audio played back as static | TTS chunks pushed to the publisher fire-and-forget while mutating a shared buffer across awaited native calls — chunks landing microseconds apart interleaved | Extracted framing into a serialized, independently-tested module |
| Audio then played at ~10x speed | AudioFrame's native serialization ignored the array's byteOffset, so every frame after the first silently resent the first frame's bytes | .slice() + a regression test encoding the exact bug |
| Memory extraction occasionally invented facts | The prompt let Claude continue the transcript instead of summarizing it | Wrapped the transcript in clearly-delimited tags |
| Duplicate "end" event on barge-in | A turn's normal-completion handler could still fire after a barge-in had already superseded it | Stream-identity guard, caught while writing tests |
| Multi-segment scripts merged incorrectly | A markup-tag regex used \S+, greedily matching straight through the tag's own closing ] | Narrowed the pattern; caught by tests before shipping |