OpenAI Realtime API WebRTC Connection Issues in Production Environment

Other Questions OpenAI
1

I have a real-time chat app that uses WebRTC to connect with OpenAI’s Realtime API through a signaling server on Cloudflare Workers. Everything works great when I test it locally, but once I deploy it to production the WebRTC connection just fails and goes to a failed state.

Here’s what happens:

  1. Browser creates SDP offer and sends to /connect-rtc endpoint
  2. Worker forwards the offer to OpenAI API
  3. Browser tries to set remote description and exchange ICE candidates

Client side code:

connection = new RTCPeerConnection();

responseChannel = connection.createDataChannel('messages');

connection.createOffer().then((sdpOffer) => {
    connection.setLocalDescription(sdpOffer);
    fetch('/connect-rtc', {
        method: 'POST',
        body: sdpOffer.sdp,
        headers: {
            'Content-Type': 'application/sdp',
        },
    })
    .then((response) => response.text())
    .then((sdpAnswer) => {
        connection.setRemoteDescription({
            sdp: sdpAnswer,
            type: 'answer',
        });
    })
    .catch((err) => console.error("Signaling failed:", err));
});

Server endpoint:

router.post('/connect-rtc', async (context) => {
    const requestBody = await context.req.text();
    const apiUrl = new URL('https://api.openai.com/v1/realtime');
    apiUrl.searchParams.set('model', 'gpt-4o-realtime-preview-2024-12-17');
    apiUrl.searchParams.set('instructions', SYSTEM_PROMPT);
    apiUrl.searchParams.set('voice', 'nova');

    const apiResponse = await fetch(apiUrl.toString(), {
        method: 'POST',
        body: requestBody,
        headers: {
            Authorization: `Bearer ${context.env.OPENAI_API_KEY}`,
            'Content-Type': 'application/sdp',
        },
    });

    if (!apiResponse.ok) {
        const errorMsg = await apiResponse.text();
        console.error('API Error:', errorMsg);
        return context.text(errorMsg, 500);
    }
    
    const answerSdp = await apiResponse.text();
    console.log('Received SDP Answer:', answerSdp);

    return context.body(answerSdp, {
        headers: {
            'Content-Type': 'application/sdp',
        },
    });
});

Any ideas what could be causing this production vs local difference?

2

you’re missing ICE candidate handling completely. WebRTC won’t work without exchanging those candidates between peers - it’ll just timeout. also check if your production domain has STUN/TURN servers set up properly. Cloudflare Workers might be blocking the default ones.

3

I’ve encountered similar issues with WebRTC in production environments. The most common culprit is the setup of your HTTPS connections. OpenAI’s Realtime API requires secure channels, and WebRTC is very sensitive to certificate issues when deployed.

Ensure that your SSL certificates are properly configured and that everything is being accessed via HTTPS. Additionally, review your Cloudflare Workers configuration; they can sometimes interfere with the Content-Type headers needed for SDP negotiation.

Check your production environment for any firewall rules that might be preventing ICE candidates from being exchanged, and consider adding debugging logs around connection state changes and ICE gathering. This can help you pinpoint where the connection is failing.

4

Had the same issue with my WebRTC setup. Problem was I used HTTP locally but HTTPS in production - browsers handle getUserMedia permissions and ICE candidates differently between the two. You’re also not handling ICE gathering state correctly. The connection’s probably failing before all candidates get collected. Add connection state listeners to see where it breaks. Check if Cloudflare Workers has timeout settings that might kill the SDP exchange. OpenAI’s Realtime API can be slow responding with answer SDPs.

5

Sounds like a CORS issue with your Cloudflare Worker. Had the exact same problem with my WebRTC app - worked fine locally but crashed in production because the Worker wasn’t handling preflight requests right. Your browser’s probably blocking the SDP exchange since the CORS headers are missing. Add proper CORS headers to your Worker response, especially for OPTIONS requests. Also watch out for Worker execution timeouts - OpenAI’s Realtime API can be slow to respond with the SDP answer, so your Worker might timeout before finishing the handshake. Add timeout handling and proper error responses in your OpenAI fetch call.