Streaming Voice Web Agents Architecture: WebRTC and LiveKit

Каскадный HTTP-пайплайн (Waterfall: 2500 - 4000 мс):
[Mic] -> (HTTP) -> [STT Whisper] -> (HTTP) -> [LLM GPT-4] -> (HTTP) -> [TTS] -> (HTTP) -> [Speaker]
       +600ms            +700ms              +600ms             +500ms          +200ms

Потоковый Speech-to-Speech через WebRTC (< 300 мс):
[Mic] ===(WebRTC RTP / Opus 20ms)====> [LiveKit SFU / Gateway] ===> [Multimodal Live LLM]
[Speaker] <==(WebRTC RTP / Opus 20ms)== [LiveKit SFU / Gateway] <=== (Audio Tokens Direct)

АРХИТЕКТУРА ГОЛОСОВОГО АГЕНТА (WEBRTC + LIVEKIT)
                                
  +-----------------------------------------------------------------------------------------+
  | БРАУЗЕР КЛИЕНТА (NEXT.JS 15 CLIENT COMPONENT)                                          |
  |                                                                                         |
  |   +-------------------+       +-----------------------+       +---------------------+   |
  |   | getUserMedia()    | ----> | Web Audio API / VAD   | ----> | RTCPeerConnection   |   |
  |   | (Echo Cancel: on) |       | (PCM 24kHz / Opus)    |       |                     |   |
  |   +-------------------+       +-----------------------+       +----------+----------+   |
  |                                                                          |              |
  |   +-------------------+       +-----------------------+                  |              |
  |   | AudioContext Play | <---- | Remote MediaStream    | <----------------+              |
  |   | (Jitter Buffer)   |       | (Opus 20ms Frames)    |                  |              |
  |   +-------------------+       +-----------------------+                  |              |
  |                                                                          |              |
  |   +---------------------------------------------------+                  |              |
  |   | RTCDataChannel ("oai-events" / Control Channel)   | <----------------+              |
  |   | JSON: session.update, response.cancel, tool_calls |                                 |
  |   +---------------------------------------------------+                                 |
  +--------------------------------------------|--------------------------------------------+
                                               | (SRTP / UDP + SCTP DataChannel)
                                               v
  +-----------------------------------------------------------------------------------------+
  | LIVEKIT SFU / WEBRTC GATEWAY (EDGE NODE)                                                |
  |                                                                                         |
  |   - ICE / STUN / TURN Терминация                                                        |
  |   - Демультиплексирование RTP/RTCP пакетов                                              |
  |   - Стриминг бинарных аудиофреймов Opus (20ms payload) по защищенному каналу            |
  +--------------------------------------------|--------------------------------------------+
                                               | (gRPC / Bidirectional TLS Stream)
                                               v
  +-----------------------------------------------------------------------------------------+
  | MULTIMODAL REALTIME MODEL (OPENAI REALTIME API / GEMINI LIVE)                           |
  |                                                                                         |
  |   - Нативный Speech-to-Speech инференс аудио-токенов                                     |
  |   - Встроенный серверный VAD (Voice Activity Detection)                                 |
  |   - Стриминг аудио-ответа фреймами по 20 мс                                             |
  |   - Параллельная эмиссия JSON Function Calls через DataChannel                          |
  +-----------------------------------------------------------------------------------------+

+---------------------------------------------------------------+----------+
| Этап конвейера обработки                                      | Задержка |
+---------------------------------------------------------------+----------+
| 1. Захват аудио, нативное эхоподавление (AEC) и фрейминг      | 20 мс    |
| 2. Кодирование Opus (фрейм 20 мс + Lookahead 2.5 мс)          | 22.5 мс  |
| 3. Сетевой транзит: Браузер -> Edge SFU (UDP/RTP)             | 25-45 мс |
| 4. Транзит: Edge SFU -> Инференс-кластер модели               | 10-25 мс |
| 5. Инференс первого аудио-токена модели (TTFT Audio)          | 80-130 мс|
| 6. Обратный транзит: Модель -> Edge SFU -> Браузер            | 35-65 мс |
| 7. Декодирование Opus, адаптивный Jitter Buffer браузера      | 20-30 мс |
+---------------------------------------------------------------+----------+
| ИТОГОВАЯ GLASS-TO-GLASS LATENCY:                              | ~215-330 |
+---------------------------------------------------------------+----------+

import { NextResponse } from "next/server";

interface OpenAISessionResponse {
  id: string;
  client_secret: {
    value: string;
    expires_at: number;
  };
}

export async function POST() {
  const apiKey = process.env.OPENAI_API_KEY;
  if (!apiKey) {
    return NextResponse.json(
      { error: "OPENAI_API_KEY не сконфигурирован на сервере" },
      { status: 500 }
    );
  }

  try {
    // Запрашиваем эфемерный ключ с ограниченным временем жизни
    const response = await fetch("https://api.openai.com/v1/realtime/sessions", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        model: "gpt-4o-realtime-preview-2024-12-17",
        voice: "verse",
        input_audio_format: "pcm16",
        output_audio_format: "pcm16",
        turn_detection: {
          type: "server_vad",
          threshold: 0.5,
          prefix_padding_ms: 300,
          silence_duration_ms: 500,
        },
      }),
    });

    if (!response.ok) {
      const errorText = await response.text();
      throw new Error(`OpenAI API error [${response.status}]: ${errorText}`);
    }

    const data: OpenAISessionResponse = await response.json();
    return NextResponse.json({ clientSecret: data.client_secret.value });
  } catch (err: unknown) {
    const message = err instanceof Error ? err.message : "Неизвестная ошибка создания сессии";
    console.error("[Realtime Session Error]:", message);
    return NextResponse.json({ error: message }, { status: 500 });
  }
}

"use client";

import { useState, useRef, useCallback, useEffect } from "react";

export interface RealtimeMessage {
  type: string;
  [key: string]: unknown;
}

export function useRealtimeVoiceWebRTC() {
  const [isConnected, setIsConnected] = useState<boolean>(false);
  const [isSpeaking, setIsSpeaking] = useState<boolean>(false);
  const [error, setError] = useState<string | null>(null);

  const pcRef = useRef<RTCPeerConnection | null>(null);
  const dcRef = useRef<RTCDataChannel | null>(null);
  const localStreamRef = useRef<MediaStream | null>(null);
  const remoteAudioRef = useRef<HTMLAudioElement | null>(null);

  const disconnect = useCallback(() => {
    if (dcRef.current) {
      dcRef.current.close();
      dcRef.current = null;
    }
    if (pcRef.current) {
      pcRef.current.close();
      pcRef.current = null;
    }
    if (localStreamRef.current) {
      localStreamRef.current.getTracks().forEach((track) => track.stop());
      localStreamRef.current = null;
    }
    if (remoteAudioRef.current) {
      remoteAudioRef.current.srcObject = null;
    }
    setIsConnected(false);
    setIsSpeaking(false);
  }, []);

  const connect = useCallback(async () => {
    try {
      setError(null);

      // 1. Получаем эфемерный токен через наш API
      const sessionRes = await fetch("/api/realtime/session", { method: "POST" });
      if (!sessionRes.ok) {
        throw new Error("Не удалось получить эфемерный токен сессии");
      }
      const { clientSecret } = await sessionRes.json();

      // 2. Инициализируем RTCPeerConnection с STUN сервером
      const pc = new RTCPeerConnection({
        iceServers: [{ urls: "stun:stun.l.google.com:19302" }],
      });
      pcRef.current = pc;

      // 3. Подключаем аудиоэлемент для удаленного трека
      const remoteAudio = document.createElement("audio");
      remoteAudio.autoplay = true;
      remoteAudioRef.current = remoteAudio;

      pc.ontrack = (event: RTCTrackEvent) => {
        if (event.streams[0]) {
          remoteAudio.srcObject = event.streams[0];
        }
      };

      // 4. Захватываем микрофон с аппаратным подавлением эха и шума
      const mediaStream = await navigator.mediaDevices.getUserMedia({
        audio: {
          channelCount: 1,
          sampleRate: 24000,
          echoCancellation: true,
          noiseSuppression: true,
          autoGainControl: true,
        },
      });
      localStreamRef.current = mediaStream;

      // Добавляем локальный аудио-трек в PeerConnection
      mediaStream.getTracks().forEach((track) => pc.addTrack(track, mediaStream));

      // 5. Открываем DataChannel для управляющих JSON-событий
      const dataChannel = pc.createDataChannel("oai-events");
      dcRef.current = dataChannel;

      dataChannel.onopen = () => {
        setIsConnected(true);
      };

      dataChannel.onmessage = (event: MessageEvent) => {
        try {
          const msg: RealtimeMessage = JSON.parse(event.data);
          if (msg.type === "input_audio_buffer.speech_started") {
            setIsSpeaking(true);
          } else if (msg.type === "input_audio_buffer.speech_stopped") {
            setIsSpeaking(false);
          }
        } catch (err: unknown) {
          console.warn("[DataChannel] Ошибка парсинга события:", err);
        }
      };

      // 6. Формируем SDP Offer и отправляем в OpenAI Realtime WebRTC Gateway
      const offer = await pc.createOffer();
      await pc.setLocalDescription(offer);

      const baseUrl = "https://api.openai.com/v1/realtime";
      const model = "gpt-4o-realtime-preview-2024-12-17";
      const sdpResponse = await fetch(`${baseUrl}?model=${model}`, {
        method: "POST",
        body: offer.sdp,
        headers: {
          Authorization: `Bearer ${clientSecret}`,
          "Content-Type": "application/sdp",
        },
      });

      if (!sdpResponse.ok) {
        throw new Error(`Ошибка SDP Handshake: ${sdpResponse.statusText}`);
      }

      const answerSdp = await sdpResponse.text();
      const answer: RTCSessionDescriptionInit = {
        type: "answer",
        sdp: answerSdp,
      };

      await pc.setRemoteDescription(answer);
    } catch (err: unknown) {
      const msg = err instanceof Error ? err.message : "Ошибка инициализации WebRTC";
      setError(msg);
      disconnect();
    }
  }, [disconnect]);

  useEffect(() => {
    return () => {
      disconnect();
    };
  }, [disconnect]);

  const sendEvent = useCallback((event: RealtimeMessage) => {
    if (dcRef.current && dcRef.current.readyState === "open") {
      dcRef.current.send(JSON.stringify(event));
    }
  }, []);

  return {
    isConnected,
    isSpeaking,
    error,
    connect,
    disconnect,
    sendEvent,
  };
}

[Browser Client] 
      |  WebRTC (RTP/Opus 20ms)
      v
[LiveKit SFU Room] 
      |  Internal Audio Subscription (zero-copy)
      v
[LiveKit Agent Worker (Node.js / Python)]
      |  Bidirectional Realtime Stream
      v
[OpenAI / Gemini Multimodal API]