techlist.io
KREN
cloudflare

Add voice to your agent (opens in new tab)

database-designreactsqlitespeech-to-textagents-sdkwebsockettext-to-speech

Cloudflare’s experimental @cloudflare/voice package adds real-time voice to existing Agents SDK applications without requiring a separate voice framework. Voice interactions use the same Durable Object, WebSocket connection, tools, and SQLite-backed history as text interactions. The package provides ready-made STT and TTS integrations while keeping provider interfaces open for alternative speech, telephony, and transport systems.

Voice Support for Existing Agents

  • withVoice(Agent) enables full conversational voice agents.
  • withVoiceInput(Agent) supports speech-to-text-only features such as dictation and voice search.
  • React applications can use useVoiceAgent and useVoiceInput.
  • Framework-independent clients can use VoiceClient.
  • Built-in Workers AI providers include:
    • Deepgram Flux for continuous speech-to-text
    • Deepgram Nova 3 for speech-to-text
    • Deepgram Aura for text-to-speech
  • Developers can get started without external API keys.

Minimal Server and Client Setup

  • A voice agent extends a class created with withVoice(Agent).
  • The server configures a transcriber and TTS provider, then implements onTurn().
  • onTurn() receives the user’s transcript and returns the agent’s response.
  • React clients can display:
    • Connection status
    • Interim and finalized transcripts
    • Conversation messages
    • Start, end, and mute controls
  • Non-React applications can connect through @cloudflare/voice/client.

How the Voice Pipeline Works

  • The browser captures 16 kHz mono PCM microphone audio.
  • Audio streams over the agent’s existing WebSocket connection.
  • A continuous STT session remains active for the duration of the call.
  • The speech-to-text model detects completed utterances and produces stable transcripts.
  • Each transcript is passed to onTurn() for application or LLM logic.
  • The response is synthesized into audio and streamed back to the client.
  • Streamed responses can be sentence-chunked so audio begins playing before the full response is complete.
  • User and agent messages are persisted in the Durable Object’s SQLite database, surviving reconnections and deployments.

Extensible Provider Architecture

  • The package is designed not to lock developers into one fixed voice stack.
  • Small provider interfaces allow speech, telephony, and transport providers to build integrations.
  • Developers can mix and match components based on their application’s requirements.
  • Voice therefore becomes another interaction mode for the same stateful agent rather than a separate application architecture.

Cloudflare’s approach is best suited to developers who already use the Agents SDK and want to add conversational voice while preserving existing state, tools, persistence, and connection patterns. Since the package is experimental, teams should evaluate provider support and API stability before relying on it in production.