How to use Currai with OpenAI Realtime voice and TTS

A voice assistant can sound polished and still fail the person using it.

The WebRTC connection may be healthy while the transcription misses a product name. The model may answer the wrong question in a convincing voice. Audio may be generated but blocked by the browser. The caller may repeat the same request because the agent never acknowledged it.

Connection status and latency matter, but they do not tell you what happened in the conversation.

A useful production record needs to connect:

• what the user said;
• what the assistant answered;
• which OpenAI model and voice produced the response;
• the call boundary, duration, and failure state;
• the model execution nested under the correct conversation turn;
• stable session and event identities that survive retries.

This guide shows how to build that record with Currai using native HTTP capture. It covers the two OpenAI audio paths developers most often mix together:

1. OpenAI Realtime: a live, two-way speech-to-speech conversation over WebRTC.
2. OpenAI TTS: one request that converts existing text into audio through the Speech endpoint.

The working voice-agent architecture in this guide uses Realtime. TTS appears later as a focused adaptation.

By the end, one Realtime call will appear in Currai as:

session: one complete browser voice call

event: openai.realtime_call

agent: openai.conversation.turn
└── model: openai.realtime.model

The top-level agent event contains readable user and assistant evidence. That is what turns a technical trace into a Currai User Story.

Run the reference implementation

The complete application is available in the public Currai examples repository. The OpenAI voice example includes the assistant builder, direct Realtime WebRTC transport, protected SDP exchange, live transcript reconciliation, best-effort Currai capture, tests, and the corrected CRLF-preserving SDP handler.

git clone git@github.com:curraiapp/examples.git
cd examples/openai
cp .env.example .env.local
pnpm install
pnpm dev

Add your own OpenAI and Currai credentials to .env.local. Never commit that file. The public repository contains only placeholder environment templates and excludes credentials, dependencies, caches, and build output.

Realtime voice is not the same as TTS

OpenAI Realtime maintains an interactive session. The browser streams microphone audio to a model, receives model audio, and exchanges lifecycle and transcript events over a WebRTC data channel.

The Speech API is simpler: your server sends text and receives an audio file or stream. It does not, by itself, know who the user is, where the text came from, or whether the spoken response solved a request.

The official OpenAI WebRTC guide recommends WebRTC for browser and mobile Realtime clients. OpenAI's gpt-4o-mini-tts model page describes the separate text-in, audio-out model used through v1/audio/speech.

What Currai adds

Currai connects user experience to execution evidence. For a voice application, that lets product and engineering teams answer questions such as:

• Which requests are callers repeating?
• Which intents are ending without a useful answer?
• Did the model answer correctly before audio playback failed?
• Which prompt, model, or voice was active during a regression?
• Are failures concentrated around one browser, release, or route?
• Did a model or prompt change improve real conversations?

Currai does not replace OpenAI, your media transport, or application logs. OpenAI runs the voice interaction. Currai receives a normalized, privacy-reviewed record of the result.

The integration does not require a Currai SDK.

The architecture

The browser owns live media. The server owns credentials and capture.

Browser
  ├── microphone + RTCPeerConnection
  ├── Realtime transcript reconciliation
  ├── POST SDP offer to /api/calls
  └── POST completed evidence to /api/currai/voice
                                  │
Trusted server                    ├── OpenAI /v1/realtime/calls
  ├── validate session + SDP      ├── Currai capture-session
  ├── redact capture payload      └── Currai capture-event
  └── preserve parent-child event nesting

Three boundaries are important:

1. OpenAI connection boundary. The server sends the browser's SDP offer and validated session configuration to OpenAI using OPENAI_API_KEY.
2. Browser conversation boundary. The component that owns the peer connection also owns transcript reconciliation, call state, and the final capture payload.
3. Currai ingestion boundary. A server route validates and redacts the completed evidence before authenticating to Currai.

Instrument the live peer connection and data channel, not only a route that stores assistant settings. Configuration is control-plane work. The user experience happens later in the live voice session.

What you need

Prepare:

• an OpenAI project with Realtime access;
• a working browser application using HTTPS or localhost;
• a server runtime that can make outbound HTTP requests;
• a Currai workspace and ingestion key pair;
• a privacy and consent policy appropriate for voice transcripts;
• a stable product user ID or anonymous UUID strategy.

Create the Currai ingestion key during onboarding or in Workspace Settings → API Keys. Store the secret when it is shown. Never place it in browser code, logs, screenshots, or a public environment variable.

Step 1: prove the OpenAI call works without Currai

Start your application and complete one short conversation.

For a pnpm-based application:

pnpm install
pnpm dev

Open the local URL, grant microphone access, start the call, speak a clear request, and wait for the model to answer.

Confirm:

• microphone permission succeeds;
• the peer connection reaches connected;
• the oai-events data channel opens;
• remote audio plays;
• completed user transcription events arrive;
• assistant transcript deltas become a final assistant transcript.

If that baseline fails, fix it first. Currai can show that a voice request failed; it cannot make an invalid SDP offer, denied microphone, or blocked audio element work.

Step 2: configure the server environment

Add these values to the server environment:

OPENAI_API_KEY=<your-openai-api-key>
CURRAI_PUBLIC_KEY=<your-currai-public-key>
CURRAI_SECRET_KEY=<your-currai-secret-key>
CURRAI_BASE_URL=https://www.currai.app

None of these credentials should use NEXT_PUBLIC_, VITE_, or another browser-public prefix.

Restart the application after changing environment values.

Step 3: install and use the Currai Skill

The fastest path is to let your coding agent find and instrument the real conversation boundary. Install the published Currai Skill at the root of the application repository:

npx skills add https://github.com/curraiapp/skills --skill currai

Then give the agent this prompt:

Use $currai to connect this OpenAI Realtime browser voice application to Currai
using OPENAI_API_KEY, CURRAI_PUBLIC_KEY, CURRAI_SECRET_KEY, and CURRAI_BASE_URL
already configured in the server environment. Find the component that owns the
live RTCPeerConnection and OpenAI Realtime data channel, not only assistant
configuration or SDP creation. Retain completed user transcriptions and finalized
assistant audio transcripts once, with no duplicate deltas. Create one UUID session
per complete call and emit openai.conversation.turn agent events with nested
openai.realtime.model events. Keep capture dependency-free, best-effort, and
server-side. Restart or deploy the application, complete one real call, and verify
the readable User Story and nested trace in Currai. Never print or copy credentials
into source code or chat.

The Currai Skill should:

1. detect the application and locate the live AI entrypoint;
2. distinguish WebRTC call handling from assistant configuration;
3. generate a native HTTP helper rather than installing a Currai SDK;
4. create one session per complete call and await it before sending events;
5. preserve readable agent evidence and parent-child model nesting;
6. redact credentials and keep failures independent from the voice call;
7. verify one real conversation in both Events and User Stories.

A successful connectivity test is useful, but it is not proof that the live OpenAI path is integrated.

Step 4: create the Realtime session through your server

With the unified WebRTC interface, the browser creates an SDP offer and sends it to your server. Your server combines it with a validated Realtime session configuration and calls OpenAI.

export async function POST(request: Request) {
  const body = await request.json();
  const sdp = typeof body.sdp === "string" ? body.sdp : "";

  if (sdp.trim().length < 20 || !sdp.trim().startsWith("v=0")) {
    return Response.json(
      { error: "A valid WebRTC SDP offer is required." },
      { status: 400 },
    );
  }

  const session = buildValidatedRealtimeSession(body.assistant);
  const form = new FormData();
  form.set("sdp", sdp);
  form.set("session", JSON.stringify(session));

  const upstream = await fetch("https://api.openai.com/v1/realtime/calls", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
    },
    body: form,
    cache: "no-store",
  });

  const answerSdp = await upstream.text();
  if (!upstream.ok) {
    return Response.json(
      { error: normalizeOpenAIError(upstream.status, answerSdp) },
      { status: upstream.status >= 500 ? 502 : upstream.status },
    );
  }

  return Response.json({ sdp: answerSdp });
}

Preserve the original SDP when forwarding it. You may use trimmed text for validation, but do not call .trim() on the value placed in the multipart form. Removing the final CRLF can produce:

Failed to parse offer: failed to unmarshal SDP: EOF

Validate the assistant name, instructions, first message, model, and voice again on the server. The browser must not be able to select arbitrary provider settings.

Step 5: open the browser peer connection

The browser needs a peer connection, remote audio element, microphone track, and data channel:

const peer = new RTCPeerConnection();
const channel = peer.createDataChannel("oai-events");
const remoteAudio = document.createElement("audio");
remoteAudio.autoplay = true;

peer.ontrack = (event) => {
  remoteAudio.srcObject = event.streams[0];
  void remoteAudio.play().catch(showAudioPlaybackButton);
};

const microphone = await navigator.mediaDevices.getUserMedia({ audio: true });
for (const track of microphone.getTracks()) {
  peer.addTrack(track, microphone);
}

const offer = await peer.createOffer();
await peer.setLocalDescription(offer);

const response = await fetch("/api/calls", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ sdp: offer.sdp, assistant }),
});

const body = await response.json();
if (!response.ok) throw new Error(body.error ?? "OpenAI call failed.");

await peer.setRemoteDescription({ type: "answer", sdp: body.sdp });

The standard OpenAI key remains on the server. The browser receives only the SDP answer needed to complete the peer connection.

Step 6: create stable call identity before connecting

Create the capture context when the user starts the call, before microphone or network operations can fail:

type TranscriptTurn = {
  itemId: string;
  role: "user" | "assistant";
  text: string;
  final: boolean;
};

type VoiceCapture = {
  sessionId: string;
  userId: string;
  boundaryEventId: string;
  assistantId: string;
  voiceId: string;
  callId?: string;
  startedAt: number;
  endedAt?: number;
  success: boolean;
  error?: string;
};

const capture: VoiceCapture = {
  sessionId: crypto.randomUUID(),
  userId: getCurrentOrAnonymousUserId(),
  boundaryEventId: crypto.randomUUID(),
  assistantId,
  voiceId,
  startedAt: Date.now(),
  success: true,
};

Keep those identifiers stable. Build the final payload once and reuse it if a temporary Currai failure requires a retry.

If OpenAI returns a call identifier in the Location response header, store the safe final path segment as callId. That links the Currai evidence to the provider call without exposing the authorization header.

Step 7: reconcile OpenAI transcript events

Realtime transcript events are not an append-only list. Assistant text can arrive as several deltas before a final transcript replaces the active row.

Useful event types include:

conversation.item.input_audio_transcription.completed
response.output_audio_transcript.delta
response.output_audio_transcript.done
response.done

Normalize them into a retained transcript keyed by item ID:

function upsertTurn(
  turns: TranscriptTurn[],
  next: TranscriptTurn,
): TranscriptTurn[] {
  const index = turns.findIndex((turn) => turn.itemId === next.itemId);
  if (index === -1) return [...turns, next];

  const copy = [...turns];
  copy[index] = next;
  return copy;
}

channel.addEventListener("message", (message) => {
  const event = JSON.parse(message.data);

  if (event.type === "conversation.item.input_audio_transcription.completed") {
    transcript = upsertTurn(transcript, {
      itemId: event.item_id,
      role: "user",
      text: event.transcript.trim(),
      final: true,
    });
  }

  if (event.type === "response.output_audio_transcript.delta") {
    transcript = appendAssistantDelta(transcript, event.item_id, event.delta);
  }

  if (event.type === "response.output_audio_transcript.done") {
    transcript = upsertTurn(transcript, {
      itemId: event.item_id,
      role: "assistant",
      text: event.transcript.trim(),
      final: true,
    });
  }
});

Display the active delta in the live transcript, but persist the final text once. Suppress duplicate final events. Keep completed turns visible after the call ends so the user and tester can inspect the whole conversation.

Step 8: capture success and failure boundaries

Do not capture only successful calls. A denied microphone, provider error, broken peer connection, and blocked playback are part of the user experience.

peer.onconnectionstatechange = () => {
  if (peer.connectionState === "failed") {
    capture.success = false;
    capture.error = "The WebRTC peer connection failed.";
    void finishAndCapture();
  }
};

channel.addEventListener("message", (message) => {
  const event = JSON.parse(message.data);
  if (event.type !== "error") return;

  capture.success = false;
  capture.error = safeOpenAIError(event.error);
  void finishAndCapture();
});

Store a safe error description, not the complete provider response. Never forward authorization headers, cookies, tokens, or raw SDP as metadata.

Step 9: post the completed evidence to your server

Freeze the completed payload when the call ends:

async function finishAndCapture() {
  capture.endedAt ??= Date.now();

  const payload = {
    ...capture,
    transcript: transcript
      .filter((turn) => turn.final && turn.text.trim())
      .map(({ role, text }) => ({ role, text })),
  };

  for (const delay of [0, 750, 2_000]) {
    if (delay) await new Promise((resolve) => setTimeout(resolve, delay));

    try {
      const response = await fetch("/api/currai/voice", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify(payload),
        keepalive: true,
      });

      if (response.ok) return;
      if (response.status < 500 && response.status !== 429) return;
    } catch {
      // Capture is best-effort. The voice call outcome must not change.
    }
  }
}

The server route should reject:

• invalid session, user, assistant, or boundary UUIDs;
• unknown transcript roles;
• empty or oversized text;
• more transcript entries than the application allows;
• unexpected provider identifiers;
• any nested sensitive key such as authorization, apiKey, or accessToken.

Step 10: send native HTTP capture from the server

Build Basic authentication only in trusted server code:

function curraiAuthorization() {
  const publicKey = process.env.CURRAI_PUBLIC_KEY;
  const secretKey = process.env.CURRAI_SECRET_KEY;
  if (!publicKey || !secretKey) return null;

  return "Basic " + Buffer.from(`${publicKey}:${secretKey}`).toString("base64");
}

async function sendToCurrai(path: string, body: unknown) {
  const authorization = curraiAuthorization();
  const baseUrl = process.env.CURRAI_BASE_URL;
  if (!authorization || !baseUrl) return false;

  try {
    const response = await fetch(`${baseUrl.replace(/\/$/, "")}${path}`, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        Authorization: authorization,
      },
      body: JSON.stringify(redact(body)),
      signal: AbortSignal.timeout(2_000),
    });
    return response.ok;
  } catch {
    return false;
  }
}

Redact recursively before serialization. A shallow top-level filter is not enough because provider responses and metadata can contain nested credentials.

Step 11: create the Currai session first

One complete call maps to one Currai session:

const sessionAccepted = await sendToCurrai("/api/v1/capture-session", {
  session_id: capture.sessionId,
  user_data: {
    user_id: capture.userId,
    properties: { source: "openai-realtime-web" },
  },
  metadata: {
    route: "browser-voice-call",
    environment: process.env.NODE_ENV ?? "development",
  },
  timestamp: capture.startedAt,
  client_config: "currai-native-http",
});

Await session creation before the first event. If it fails, return a retryable capture response and leave the OpenAI call result untouched.

Step 12: preserve conversational evidence and trace nesting

First emit the call boundary:

await captureEvent({
  event_id: capture.boundaryEventId,
  session_id: capture.sessionId,
  parent_id: null,
  kind: "event",
  primitive_name: "openai.realtime_call",
  args: JSON.stringify({
    assistantId: capture.assistantId,
    voiceId: capture.voiceId,
  }),
  result: JSON.stringify({
    transcriptEntries: transcript.length,
  }),
  success: capture.success,
  latency: capture.endedAt - capture.startedAt,
  metadata: {
    provider: "openai",
    model: "gpt-realtime-2.1",
    callId: capture.callId,
  },
});

Then group consecutive user and assistant messages into readable turns. The agent event belongs at the top level:

await captureEvent({
  event_id: agentEventId,
  session_id: capture.sessionId,
  parent_id: null,
  kind: "agent",
  primitive_name: "openai.conversation.turn",
  args: JSON.stringify({ input: userText }),
  result: JSON.stringify({ output: assistantText }),
  success: capture.success,
  metadata: {
    provider: "openai",
    model: "gpt-realtime-2.1",
    voiceId: capture.voiceId,
  },
});

Nest OpenAI execution beneath that turn:

await captureEvent({
  event_id: modelEventId,
  session_id: capture.sessionId,
  parent_id: agentEventId,
  kind: "model",
  primitive_name: "openai.realtime.model",
  args: JSON.stringify({ input: userText }),
  result: JSON.stringify({ output: assistantText }),
  success: capture.success,
  metadata: {
    provider: "openai",
    model: "gpt-realtime-2.1",
  },
});

Currai recognizes user evidence in fields such as args.input, args.prompt, and args.message. It recognizes assistant evidence in result.output, result.text, result.response, and result.answer.

A custom transcript field can be useful trace metadata, but it does not replace recognized top-level agent evidence. Without that evidence, the trace may exist while the User Story remains empty.

Step 13: instrument one-way OpenAI TTS correctly

TTS begins after your application already knows what text it wants spoken. Keep the API key on the server:

const startedAt = Date.now();
const response = await fetch("https://api.openai.com/v1/audio/speech", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "gpt-4o-mini-tts",
    voice: "coral",
    input: textToSpeak,
    response_format: "mp3",
  }),
});

Do not upload the audio buffer to Currai. Record the text, safe output properties, model, voice, latency, and success:

await captureEvent({
  event_id: crypto.randomUUID(),
  session_id: sessionId,
  parent_id: agentEventId,
  kind: "model",
  primitive_name: "openai.tts",
  args: JSON.stringify({ input: textToSpeak }),
  result: JSON.stringify({
    audioGenerated: response.ok,
    format: "mp3",
  }),
  success: response.ok,
  latency: Date.now() - startedAt,
  metadata: {
    provider: "openai",
    model: "gpt-4o-mini-tts",
    voice: "coral",
  },
});

If a separate language model wrote textToSpeak, capture that decision as its own model event. TTS is the rendering step. It should not take credit for reasoning that happened somewhere else.

If the product is only a read-aloud feature and no user-to-assistant interaction exists, capture the TTS model event without inventing an agent conversation turn.

Step 14: verify one real production-shaped interaction

Restart or deploy the application and make one real call. Use an input with a clear intent and a response that is easy to recognize:

I need to move my appointment from Tuesday afternoon to Friday morning.

Verify all of the following:

1. OpenAI returns audible speech and the browser retains the full transcript.
2. Observe → Events contains one session for the complete call.
3. The session contains one openai.realtime_call boundary.
4. Every openai.realtime.model event is nested beneath the correct openai.conversation.turn.
5. Analyse → User Stories shows at least one readable user or assistant message.
6. Provider, model, voice, environment, route, and call ID appear only when known.
7. A failed call is captured with success: false and a safe error.
8. Invalid or unavailable Currai capture does not interrupt OpenAI audio.

For TTS, verify the trace contains the exact text that was spoken and safe generation metadata, while excluding audio bytes and credentials.

Common failures

Failed to parse offer: failed to unmarshal SDP: EOF

Your server is probably modifying the SDP offer. Preserve the exact string from offer.sdp when placing it in the multipart form. In particular, do not trim away its final CRLF.

The WebRTC call works but no User Story appears

Check the capture route rather than the provider connection. Confirm the Currai credentials are available to the server process, the session request succeeds before events, and the call-end handler posts the finalized transcript.

Then inspect event fields. The top-level agent event needs args.input and/or result.output. A nested model event or a custom transcript object is not enough to render the conversation.

Assistant transcript text is duplicated

Do not append every delta. Key transcript rows by OpenAI item ID, update the active assistant row as deltas arrive, replace it with the final transcript, and suppress repeated completed events.

The model answered, but no sound played

Treat browser autoplay separately from model success. Keep the assistant transcript as evidence, record an audio-playback failure at the call boundary, and present a user gesture that retries audio.play().

Currai temporarily rejects capture

Use short retries for 429 and 5xx responses. Reuse the exact session and event IDs. Do not retry unchanged 400 payloads, and never keep the browser call open while waiting for observability.

TTS traces look like model reasoning

Nest openai.tts below the agent turn or generation that supplied the text. Record only what TTS actually did: convert known text into a particular audio format and voice.

Privacy and security checklist

Before shipping:

• obtain appropriate consent for microphone use and transcript retention;
• keep OpenAI and Currai credentials server-side;
• redact authorization headers, cookies, passwords, secrets, and tokens recursively;
• do not send SDP, raw audio, or generated audio bytes to Currai;
• cap transcript entries and text length;
• use a privacy-reviewed stable user identifier;
• separate live call success from capture success;
• include only provider and model metadata known at runtime;
• verify retention and deletion behavior against your product policy.

The result

A good voice integration gives you more than a green connection indicator. It shows the complete path from request to response:

User speaks
  ↓
OpenAI Realtime transcribes and responds
  ↓
Browser retains finalized evidence
  ↓
Server validates and redacts
  ↓
Currai session + readable conversation turn
  └── nested OpenAI model execution

That structure lets you evaluate real voice-agent behavior, find repeated and abandoned intents, connect user outcomes to model changes, and investigate provider failures without exposing credentials or coupling observability to the call.

For the implementation checklist and copy-ready snippets, read Integrate Currai with OpenAI voice. You can also compare the Vapi and Retell AI workflows.