Skip to content

Secure AI Gateway realtime client secrets - #16284

Open
31Carlton7 wants to merge 1 commit into
mainfrom
carlton/elevenlabs-realtime
Open

Secure AI Gateway realtime client secrets#16284
31Carlton7 wants to merge 1 commit into
mainfrom
carlton/elevenlabs-realtime

Conversation

@31Carlton7

@31Carlton7 31Carlton7 commented Jun 20, 2026

Copy link
Copy Markdown
Contributor

What changed

  • Exposed allowedOrigins on gateway.experimental_realtime.getToken() so browser redemption can be bound to trusted app origins.
  • Pinned quotaEntityId, tags, and user from the server-side session configuration into the Gateway client secret.
  • Rejected request-scoped BYOK credentials before minting; provider credentials must be configured in AI Gateway rather than sent through a browser session.
  • Exported Gateway-specific realtime token option and factory types.
  • Updated Gateway realtime documentation to describe the current single-use vcst_ flow and its security boundaries.
  • Removed direct ElevenAgents support and the demo changes from this PR.

Why

Gateway should remain the credential and traffic boundary. Browser clients receive a short-lived, single-use token bound to an exact model and optionally an origin, while trusted attribution and quota options are fixed by the server. This avoids exposing broad provider credentials or allowing one client to select another customer resource.

The working ElevenAgents adapter has been preserved on carlton/elevenlabs-agents-realtime for a future PR. Actual ElevenLabs agent support can build on Gateway private-resource mapping and team-scoped BYOK without putting raw agent IDs or provider credentials in the public client contract.

Validation

  • pnpm --filter @ai-sdk/gateway test:node (467 passed)
  • pnpm --filter @ai-sdk/gateway test:edge (462 passed, 5 skipped)
  • pnpm --filter @ai-sdk/gateway type-check
  • pnpm --filter @ai-sdk/gateway build
  • pnpm exec ultracite check ... (0 warnings, 0 errors)
  • pnpm validate:docs (493 MDX files)
  • git diff --check

@gr2m

gr2m commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

compare #15747. Tools work differently with Elevenlabs, if I recall correctly they need to be registered ahead of time, I wanted to investigate further if there is another way. Did the e2e Elevenalbs example work for you out of the box for you?

@31Carlton7

31Carlton7 commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

compare #15747. Tools work differently with Elevenlabs, if I recall correctly they need to be registered ahead of time, I wanted to investigate further if there is another way. Did the e2e Elevenalbs example work for you out of the box for you?

Yes. They worked just fine for me. You can try out the example I left there.

EDIT: Taking a closer look at the PR you linked, my implementation doesn't work for tool calls, so will submit a couple new commits to get this up and running

Comment thread packages/elevenlabs/src/realtime/elevenlabs-realtime-event-mapper.ts Outdated
@31Carlton7
31Carlton7 force-pushed the carlton/elevenlabs-realtime branch from 83e49c9 to c07083e Compare June 23, 2026 18:34
@31Carlton7 31Carlton7 changed the title Add ElevenLabs realtime agents support Secure AI Gateway realtime client secrets Jun 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants