Skip to content

Add ElevenLabs realtime agents support#16284

Draft
31Carlton7 wants to merge 4 commits into
mainfrom
carlton/elevenlabs-realtime
Draft

Add ElevenLabs realtime agents support#16284
31Carlton7 wants to merge 4 commits into
mainfrom
carlton/elevenlabs-realtime

Conversation

@31Carlton7

@31Carlton7 31Carlton7 commented Jun 20, 2026

Copy link
Copy Markdown
Contributor

What changed

  • Added elevenLabs.experimental_realtime() for ElevenAgents WebSocket sessions.
  • Added signed URL client-secret creation for agent_id-based conversations.
  • Added ElevenAgents event mapping for text, audio, transcripts, tool calls, ping/pong, interruptions, and response completion.
  • Added ElevenLabs realtime session options for conversation overrides, dynamic variables, and custom LLM body fields.
  • Extended the existing shared Next.js realtime example with ElevenLabs, using the agent-configured prompt, voice, and turn detection.
  • Added provider docs and a changeset.

Why

ElevenLabs has realtime agent conversations, but its WebSocket protocol differs from the existing OpenAI/Google/xAI realtime protocol. This adapter normalizes ElevenAgents into the AI SDK realtime interface while preserving ElevenLabs-specific session overrides.

Validation

  • pnpm --filter @ai-sdk/elevenlabs type-check
  • pnpm --filter @ai-sdk/elevenlabs test:node (38 tests)
  • pnpm --filter @ai-sdk/elevenlabs test:edge (38 tests)
  • pnpm lint-staged
  • Live example connection through /api/realtime/setup, including signed URL creation, WebSocket session, text response, audio events, interruption, and ping/pong
  • Browser verification of the shared provider selector, agent defaults, connection lifecycle, transcript, and event log
  • git diff --check

Follow-up

AI Gateway wiring is intentionally separate. Gateway can reuse the new exported realtime model/mapper after this package is available, rather than duplicating the ElevenAgents protocol translation.

@gr2m

gr2m commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

compare #15747. Tools work differently with Elevenlabs, if I recall correctly they need to be registered ahead of time, I wanted to investigate further if there is another way. Did the e2e Elevenalbs example work for you out of the box for you?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants