Audio and text synchronization issues with OpenAI realtime API

Other Questions OpenAI
1

I’m working with OpenAI’s realtime API (gpt-4o-realtime-preview-2024-12-17) in a React app for live voice chat and transcription. The main problem is that the text and audio don’t sync up right. The text shows up too early or the audio comes out late.

My setup:

  • WebSocket connection for streaming audio to OpenAI
  • RealtimeClient for handling live audio responses
  • WavRecorder and WavStreamPlayer for audio capture and playback (16bitPCM format)
  • Text updates happen as responses come in from the API

Here’s how I connect to the API:

const startConnection = useCallback(async () => {
  const apiClient = clientRef.current;
  const audioRecorder = recorderRef.current;
  const audioPlayer = playerRef.current;

  await audioRecorder.start();
  await audioPlayer.initialize();

  try {
    const result = await apiClient.connect();
    if (result) {
      setIsLoading(false);
      apiClient.sendMessage([{ type: "input_text", text: "Hi there!" }]);
      
      if (apiClient.getDetectionMode() === "server_vad") {
        await audioRecorder.capture((audioData) => apiClient.addAudioInput(audioData.mono));
      }
    }
  } catch (err) {
    console.error("Connection failed:", err);
  }
}, []);

And here’s the response handler:

apiClient.on("conversation.updated", async ({ message, changes }) => {
  if (message.role === "assistant" && changes?.audio) {
    audioPlayer.play16BitPCM(changes.audio, message.id);
    displayTextRef.current = message.formatted.transcript;
  } else if (changes?.text) {
    displayTextRef.current = message.formatted.transcript;
  }

  if (message.status === "completed" && message.formatted.audio?.length) {
    const audioFile = await WavRecorder.process(message.formatted.audio, 24000, 24000);
    setAudioSource(audioFile.url);
  }
});

Main Issue:
Can’t get the text to scroll in sync with the audio playback.

My scrolling approach uses 150 words per minute:

const animateTextScroll = () => {
  if (!textContainerRef.current) return;

  const element = textContainerRef.current;
  const now = Date.now();
  const timeElapsed = now - scrollTimerRef.current;
  const totalDuration = calculateScrollTime(displayText);

  if (timeElapsed >= totalDuration) {
    element.scrollTop = element.scrollHeight - element.clientHeight;
    return;
  }

  const scrollProgress = timeElapsed / totalDuration;
  const maxScroll = element.scrollHeight - element.clientHeight;

  const smoothEasing = (t) =>
    t < 0.5 ? 2 * t * t : 1 - Math.pow(-2 * t + 2, 2) / 2;

  element.scrollTop = maxScroll * smoothEasing(scrollProgress);
  requestRef.current = requestAnimationFrame(animateTextScroll);
};

What I tried:

  1. Converting 16-bit PCM to audio source but the conversion delay messes up timing
  2. Using word count timing (150 WPM) which works for short responses but fails on longer ones

My questions:

  1. What’s the best way to sync text scrolling with real-time audio?
  2. Are there good libraries for handling this kind of synchronization?

Thanks for any help!

2

This timing mess happens because you’re dealing with multiple moving pieces - OpenAI streams, WebSocket latency, audio buffers, React renders. They never sync up when handled separately.

The real fix isn’t better timing math or buffering tricks. You need to treat the whole thing as one coordinated process instead of separate streams fighting each other.

Trying to sync manually means you’re always chasing edge cases. Network hiccups, varying response speeds, browser quirks - too many variables for custom code.

Better approach: treat it as workflow automation. Connect OpenAI realtime API, audio processing, text rendering, and UI updates as synchronized steps in one pipeline.

Latenode handles this exact multi-stream coordination. Set up OpenAI, audio processing, and text sync as connected workflow nodes. It manages timing, buffering, and sync automatically - no custom timing logic needed.

No more manual buffer calculations or predicting audio duration. The platform coordinates everything so streams stay synced naturally.

3

Timing issues with realtime APIs are the worst. The problem’s your audio buffer fighting with instant text updates - that’s what creates the offset.

Don’t try predicting scroll timing. Just pause text rendering until audio actually starts. Store the incoming text chunks but don’t show them yet - wait for audio.play() to fire, then release the text in sync. Handles all buffering delays without messy calculations.

4

Ah, the classic sync nightmare. Been down this rabbit hole with realtime voice interfaces.

Your root issue? You’re manually trying to time OpenAI’s streaming responses with UI updates. Gets messy fast - variable network latency, audio buffer delays, unpredictable API chunks.

What actually works: ditch the manual timing calculations. Stop trying to predict audio duration with WPM estimates. You need event-driven sync that reacts to actual audio playback.

WavStreamPlayer should emit position events as audio plays. Hook into those to drive text scrolling:

audioPlayer.on('timeupdate', (currentTime, duration) => {
  const scrollProgress = currentTime / duration;
  updateTextScroll(scrollProgress);
});

Honestly though, building this sync logic from scratch is painful. You’re rebuilding what should be handled by a proper automation platform.

I’d move this entire flow to Latenode. Has native OpenAI integration and handles WebSocket connections, audio streaming, and UI sync as one coordinated workflow. Set up the realtime API connection, audio processing, and text sync as connected nodes - it handles all timing coordination automatically.

No more manual buffer management or timing calculations. Just clean, automated sync.

5

Your sync problem happens because you’re treating the streaming response like one big chunk when it’s actually broken into pieces. Audio chunks and text chunks arrive at different speeds with different processing delays.

The issue with your conversation.updated handler is timing - you play audio right away while updating text at the same time, but audio buffering creates latency that text doesn’t have. That’s what’s causing your offset.

Ditch the WPM scroll timing and use a buffering strategy instead. Collect both audio and text chunks as they stream in, but don’t render anything until you’ve buffered enough content to keep playback smooth. A 300ms buffer window works great with OpenAI’s API.

For scrolling, scrap the duration-based calculations completely. Track your audio player’s actual playback progress and map that to text character positions. When streaming text comes in, store character offsets with timestamps from when the matching audio chunk arrived.

Here’s the key: make your text display react to real audio playback events, not predicted timing. This automatically handles wonky network conditions and API response patterns without you having to manually tweak anything.

6

Had the same sync nightmare building voice chat last year. You’re treating audio and text as separate streams when they need to sync from the start.

Ditch the WPM calculations. Track actual audio playback position instead. Most audio players have currentTime properties or playback events - use those to drive text scrolling directly instead of guessing.

With OpenAI’s realtime API, audio chunks come with timing metadata. Store cumulative audio duration as chunks arrive, then map text positions to those timestamps. When audio plays, use its position to figure out which text should be highlighted.

Buffer both audio and text updates before showing them. Gives you a small window to coordinate release and handle processing delays. 100-200ms works great - users won’t notice but it smooths out timing issues.

Make audio playback your master clock. Everything else follows it, don’t try predicting timing separately.