Real-Time Voice AI with Gemini Live API: PCM Streaming, React Hooks, and Three Critical Bugs

Gemini's native audio streaming API enables real-time voice conversations: speak, and Gemini responds with synthesized speech in under a second, with full transcription of both sides. The API itself is elegant. But getting it to work correctly in a React application requires solving three non-obvious bugs that the documentation doesn't mention.

Architecture

Browser Microphone
  ↓ getUserMedia() → AudioWorklet (16kHz, mono PCM)
  ↓ Int16Array chunks → base64
WebSocket → Gemini Live API
  model: gemini-2.5-flash-native-audio-preview-12-2025
  ↓
Gemini processes audio + responds in real-time
  ↓
WebSocket ← Audio chunks (PCM, 24kHz)
           ← Transcription events (input + output)
  ↓
WebAudio API playback (streaming, no buffer wait)
  + Real-time transcript display (both sides)

Session Configuration — Get This Exactly Right

The session config has two common mistakes that fail silently. Transcription must be at the top level of config, not inside generationConfig. And the model name must never appear inside config:

typescript

1import { GoogleGenAI } from '@google/genai'
2
3const ai = new GoogleGenAI({ apiKey: process.env.VITE_GEMINI_API_KEY })
4
5// ✅ CORRECT configuration
6const session = await ai.live.connect({
7  model: 'gemini-2.5-flash-native-audio-preview-12-2025', // model here, NOT in config
8  config: {
9    outputAudioTranscription: {},   // TOP LEVEL ← critical
10    inputAudioTranscription: {},    // TOP LEVEL ← critical
11    generationConfig: {
12      responseModalities: ['AUDIO'],
13      // outputAudioTranscription does NOT go here
14    },
15    systemInstruction: {
16      parts: [{ text: 'You are a helpful voice assistant. Be concise.' }],
17    },
18  },
19})
20
21// ❌ WRONG — transcription silently ignored
22// config: { generationConfig: { outputAudioTranscription: {} } }
23
24// ❌ WRONG — WebSocket error 1007
25// config: { model: 'gemini-...', generationConfig: { ... } }

Bug 1: PCM Alignment with Int16Array

The WebAudio AudioWorklet outputs Uint8Array views into a larger ArrayBuffer. When you pass such a view to new Int16Array(buffer), you get the underlying buffer — ignoring the view's byteOffset and byteLength. This misaligns the PCM data, causing corrupted audio or silence.

typescript

1// ❌ WRONG — ignores byteOffset if uint8Array is a view
2const int16 = new Int16Array(uint8Array.buffer)
3
4// ✅ CORRECT — respects the view's position within the buffer
5const int16 = new Int16Array(
6  uint8Array.buffer,
7  uint8Array.byteOffset,
8  uint8Array.byteLength / 2
9)
10
11// Then encode to base64 for Gemini:
12const base64 = btoa(String.fromCharCode(...new Uint8Array(int16.buffer)))
13
14// Send to Gemini Live:
15session.sendRealtimeInput({
16  audio: {
17    data: base64,
18    mimeType: 'audio/pcm;rate=16000',
19  },
20})

Bug 2: Stale useRef in Audio Callbacks

Audio callbacks (from AudioWorklet processors and WebSocket message handlers) capture the closure's initial values. If you store the Gemini session in useState, the audio callback will always see null — the initial state value — not the connected session.

typescript

1// ❌ WRONG — audio callback captures initial null
2const [session, setSession] = useState<LiveSession | null>(null)
3
4audioWorklet.port.onmessage = (event) => {
5  // session is always null here, even after connect() completes
6  session?.sendRealtimeInput({ audio: { data: encode(event.data), mimeType: 'audio/pcm;rate=16000' } })
7}
8
9// ✅ CORRECT — useRef is mutable, audio callback always reads current value
10const sessionRef = useRef<LiveSession | null>(null)
11
12const connect = async () => {
13  const newSession = await ai.live.connect({ model: MODEL, config: CONFIG })
14  sessionRef.current = newSession  // Mutate ref — no re-render needed
15}
16
17audioWorklet.port.onmessage = (event) => {
18  // sessionRef.current is always the latest value
19  sessionRef.current?.sendRealtimeInput({
20    audio: { data: encode(event.data), mimeType: 'audio/pcm;rate=16000' }
21  })
22}

Bug 3: React 19 Ref Callbacks Return Values

React 19 introduced cleanup functions for ref callbacks. An arrow function that assigns to an array element returns the assigned value. React 19 treats this as a cleanup function and calls it on unmount — causing unexpected behavior.

typescript

1// ❌ WRONG in React 19 — (el) => arr[i] = el returns el,
2// which React 19 treats as a cleanup function
3const audioRefs = useRef<HTMLAudioElement[]>([])
4{items.map((_, i) => (
5  <audio ref={(el) => audioRefs.current[i] = el} />
6))}
7
8// ✅ CORRECT — explicit block, no implicit return
9{items.map((_, i) => (
10  <audio ref={(el) => { audioRefs.current[i] = el }} />
11))}

Handling Transcription Events

typescript

1session.onmessage = (msg) => {
2  // Audio playback chunks
3  if (msg.serverContent?.modelTurn?.parts) {
4    for (const part of msg.serverContent.modelTurn.parts) {
5      if (part.inlineData?.mimeType?.startsWith('audio/')) {
6        playAudioChunk(part.inlineData.data)
7      }
8    }
9  }
10
11  // Assistant transcript (what Gemini said)
12  if (msg.serverContent?.outputTranscription?.text) {
13    setAssistantTranscript(prev => prev + msg.serverContent.outputTranscription.text)
14  }
15
16  // User transcript (what was heard from the microphone)
17  if (msg.serverContent?.inputTranscription?.text) {
18    setUserTranscript(prev => prev + msg.serverContent.inputTranscription.text)
19  }
20
21  // Turn complete — stop displaying "speaking" indicator
22  if (msg.serverContent?.turnComplete) {
23    setIsSpeaking(false)
24  }
25}

Two Variants

Streamlit prototype (Python): streamlit-mic-recorder → Gemini → gTTS playback. Fast to deploy, great for demos. Limited to request/response (not true streaming).
React + TypeScript production (this article): true streaming PCM → WebSocket → streaming audio playback. Dual transcription (user + assistant). Used as the voice layer for the Bringo shopping assistant.

Voice response latency (end-to-end): <1s
Real-time transcription streams (user + assistant): 2
Non-obvious bugs that block production builds: 3
Microphone PCM sample rate (Gemini requirement): 16kHz