Skip to content

feat(media-use): use CLI free HeyGen usage #2027

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
miguel-heygen wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(media-use): use CLI free HeyGen usage #2027

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions packages/cli/src/auth/client.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { AuthClient, apiBaseUrl, buildAuthHeaders } from "./client.js";
import {
AuthClient,
HEYGEN_CLI_SOURCE,
HEYGEN_CLI_SOURCE_HEADER,
apiBaseUrl,
buildAuthHeaders,
} from "./client.js";
import { isAuthError } from "./errors.js";
import type { ResolvedCredential } from "./resolver.js";

Expand Down Expand Up @@ -51,11 +57,17 @@ describe("auth/client", () => {
source: "file_json",
refreshable: false,
};
expect(buildAuthHeaders(cred)).toEqual({ authorization: "Bearer at_123" });
expect(buildAuthHeaders(cred)).toEqual({
authorization: "Bearer at_123",
[HEYGEN_CLI_SOURCE_HEADER]: HEYGEN_CLI_SOURCE,
});
});

it("buildAuthHeaders uses x-api-key for api_key", () => {
expect(buildAuthHeaders(apiKeyCred())).toEqual({ "x-api-key": "hg_x" });
expect(buildAuthHeaders(apiKeyCred())).toEqual({
"x-api-key": "hg_x",
[HEYGEN_CLI_SOURCE_HEADER]: HEYGEN_CLI_SOURCE,
});
});

it("getCurrentUser parses a wrapped {data: {...}} payload", async () => {
Expand Down
9 changes: 7 additions & 2 deletions packages/cli/src/auth/client.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ import { scrubCredentials } from "./scrub.js";
import type { OAuthTokens } from "./store.js";

const DEFAULT_BASE_URL = "https://api.heygen.com";
export const HEYGEN_CLI_SOURCE_HEADER = "X-HeyGen-Source";
export const HEYGEN_CLI_SOURCE = "cli";

export function apiBaseUrl(): string {
const override = process.env["HEYGEN_API_URL"];
Expand Down Expand Up @@ -177,9 +179,12 @@ export class AuthClient {

export function buildAuthHeaders(credential: ResolvedCredential): Record<string, string> {
if (credential.type === "oauth") {
return { authorization: `Bearer ${credential.access_token}` };
return {
authorization: `Bearer ${credential.access_token}`,
[HEYGEN_CLI_SOURCE_HEADER]: HEYGEN_CLI_SOURCE,
};
}
return { "x-api-key": credential.key };
return { "x-api-key": credential.key, [HEYGEN_CLI_SOURCE_HEADER]: HEYGEN_CLI_SOURCE };
}

async function safeText(res: Response): Promise<string> {
Expand Down
4 changes: 2 additions & 2 deletions skills-manifest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,8 @@
"files": 10
},
"media-use": {
"hash": "49352d2f3b42c0ba",
"files": 101
"hash": "ce37666f527790cf",
"files": 102
},
"motion-graphics": {
"hash": "f5a432862116b39a",
Expand Down
44 changes: 23 additions & 21 deletions skills/media-use/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,13 +38,13 @@ Returns one line: `resolved <id> → <path> (<type>, <metadata>)`

### Types

| Type | What it finds | Provider |
| ------- | ------------------- | ---------------------------------------- |
| `bgm` | Background music | HeyGen audio catalog (10k+ tracks) |
| `sfx` | Sound effects | Bundled 19-file library + HeyGen catalog |
| `image` | Photos, backgrounds | HeyGen asset search (75k+ vectors) |
| `icon` | Icons, logos | HeyGen asset search (type=icon) |
| `voice` | TTS voiceover | Local Kokoro (free); HeyGen TTS upsell |
| Type | What it finds | Provider |
| ------- | ------------------- | -------------------------------------------------------- |
| `bgm` | Background music | HeyGen audio catalog (10k+ tracks) |
| `sfx` | Sound effects | Bundled 19-file library + HeyGen catalog |
| `image` | Photos, backgrounds | HeyGen asset search (75k+ vectors) |
| `icon` | Icons, logos | HeyGen asset search (type=icon) |
| `voice` | TTS voiceover | HeyGen TTS (OAuth free allowance); Kokoro local fallback |

### Examples

Expand Down Expand Up @@ -115,15 +115,17 @@ ladder, `describeModelLadder`); the agent can see the ladder and override.
| ------------- | --------------------------------------------------------------------------------------- |
| bgm/sfx | heygen catalog (free) |
| image | heygen search, then local mflux (best FLUX for your RAM), then codex `image_gen` upsell |
| voice | local **Kokoro** (free, on-device), then **heygen tts** paid upsell |
| voice | **heygen tts** via OAuth/web-plan allowance, then local **Kokoro** fallback |
| icon | heygen asset search |
| video (local) | local LTX (`videogen` ladder); `heygen video create` avatar upsell |

Local Kokoro (voice), mflux (image), and LTX (video) run on-device (free,
private, offline once cached). Paid/cloud upsells sit behind them: HeyGen TTS
for voice, the `codex` CLI (ChatGPT sub) for a better image, the `heygen` CLI
for avatar video. Cost rule (X4): the agent confirms before an agent-initiated
paid call; a user-requested one just runs.
private, offline once cached). Credentialed HeyGen TTS should be tried first for
voice so OAuth CLI users consume the free web-plan allowance (10 min/month);
API-key usage and overage follow the user's HeyGen billing path. Paid/cloud
upsells remain the `codex` CLI (ChatGPT sub) for a better image and the
`heygen` CLI for avatar video. Cost rule (X4): the agent confirms before an
agent-initiated paid call; a user-requested one just runs.

To force a specific generator (e.g. a user says "make this image with codex"),
pass `--provider codex`: it pins resolution to that provider and skips the
Expand Down Expand Up @@ -218,15 +220,15 @@ if a local tool is absent, resolve degrades gracefully to the free/cloud path,
so nothing here is strictly required except `ffmpeg`/`ffprobe`. Install a local
tool to unlock its free, private, on-device path. media-use holds no keys.

| Tool | Serves | Install |
| ------------------ | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------- |
| `ffmpeg`/`ffprobe` | adopt probing, cut, duck bake, loudnorm | system package (`brew install ffmpeg`) |
| `heygen` | catalog (bgm/sfx/image/icon), TTS + avatar upsell | `curl -fsSL https://static.heygen.ai/cli/install.sh \| bash` then `heygen auth login --key <key>` (needs >= v0.1.6) |
| `mflux-generate` | local image gen (FLUX), best-for-RAM | `uv venv ~/.venvs/mflux && VIRTUAL_ENV=~/.venvs/mflux uv pip install mflux==0.9.6` |
| `codex` | image gen upsell (ChatGPT sub) | Codex CLI, logged in via ChatGPT (owns its own auth) |
| `parakeet-mlx` | local transcription (default ASR, best) | `uv venv ~/.venvs/parakeet && VIRTUAL_ENV=~/.venvs/parakeet uv pip install parakeet-mlx` |
| `ltx-2-mlx` | local video gen | `git clone https://github.com/dgrauet/ltx-2-mlx && cd ltx-2-mlx && uv sync --all-extras` |
| `npx hyperframes` | Kokoro TTS (voice), whisper.cpp (transcribe fallback), remove-background | bundled with the hyperframes CLI |
| Tool | Serves | Install |
| ------------------ | ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------- |
| `ffmpeg`/`ffprobe` | adopt probing, cut, duck bake, loudnorm | system package (`brew install ffmpeg`) |
| `heygen` | catalog (bgm/sfx/image/icon), TTS + avatar upsell | `curl -fsSL https://static.heygen.ai/cli/install.sh \| bash` then `heygen auth login --oauth` or API-key login (needs >= v0.1.6) |
| `mflux-generate` | local image gen (FLUX), best-for-RAM | `uv venv ~/.venvs/mflux && VIRTUAL_ENV=~/.venvs/mflux uv pip install mflux==0.9.6` |
| `codex` | image gen upsell (ChatGPT sub) | Codex CLI, logged in via ChatGPT (owns its own auth) |
| `parakeet-mlx` | local transcription (default ASR, best) | `uv venv ~/.venvs/parakeet && VIRTUAL_ENV=~/.venvs/parakeet uv pip install parakeet-mlx` |
| `ltx-2-mlx` | local video gen | `git clone https://github.com/dgrauet/ltx-2-mlx && cd ltx-2-mlx && uv sync --all-extras` |
| `npx hyperframes` | Kokoro TTS (voice), whisper.cpp (transcribe fallback), remove-background | bundled with the hyperframes CLI |

The RAM-graded local-model shortlist + exact per-tier install/invoke lives in
`scripts/lib/local-models.mjs` (the agent can read `describeModelLadder(cap, specs)`
Expand Down
4 changes: 2 additions & 2 deletions skills/media-use/audio/references/requirements.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,13 @@ Run `npx hyperframes auth status` to see what's configured and which engines a w
| **Kokoro** (TTS, no key) | always — final voice fallback | `pip install kokoro-onnx soundfile` |
| **MusicGen** (BGM, no key) | always — final music fallback | `pip install transformers torch soundfile numpy` |

`hyperframes auth login` (browser OAuth) is the recommended setup: one sign-in, every project, no per-repo `.env`. An OAuth login is sent as `Authorization: Bearer`; an API key as `X-Api-Key`. With no HeyGen credential, voice/BGM run fully locally (Kokoro / MusicGen) — `hyperframes auth status` and `hyperframes doctor` both report whether those local deps are installed.
`hyperframes auth login` (browser OAuth) is the recommended setup: one sign-in, every project, no per-repo `.env`. An OAuth login is sent as `Authorization: Bearer`; an API key as `X-Api-Key`; both are tagged with `X-HeyGen-Source: cli`. OAuth CLI users can consume the web-plan free allowance for HeyGen TTS (10 min/month); API keys follow the normal API billing path. With no HeyGen credential, voice/BGM run fully locally (Kokoro / MusicGen) — `hyperframes auth status` and `hyperframes doctor` both report whether those local deps are installed.

## Model caches & system dependencies

Each command downloads its own model on first run and caches it under `~/.cache/hyperframes/`:

- **TTS (HeyGen)** — no local deps; needs a HeyGen credential + `ffmpeg` on PATH (to transcode the mp3 response to `.wav`). Credential resolves like the CLI: `$HEYGEN_API_KEY` → `$HYPERFRAMES_API_KEY` → `~/.heygen/credentials` (shared with heygen-cli; run `npx hyperframes auth login`). An OAuth login is sent as `Authorization: Bearer`; an API key as `X-Api-Key`.
- **TTS (HeyGen)** — no local deps; needs a HeyGen credential + `ffmpeg` on PATH (to transcode the mp3 response to `.wav`). Credential resolves like the CLI: `$HEYGEN_API_KEY` → `$HYPERFRAMES_API_KEY` → `~/.heygen/credentials` (shared with heygen-cli; run `npx hyperframes auth login`). An OAuth login is sent as `Authorization: Bearer`; an API key as `X-Api-Key`; both include `X-HeyGen-Source: cli` so the backend can apply CLI OAuth free usage.
- **TTS (ElevenLabs)** — same as HeyGen: API key + `ffmpeg`.
- **TTS (Kokoro)** — Kokoro-82M (~311 MB) + voices (~27 MB) in `tts/`. Requires Python 3.8+ with `kokoro-onnx` and `soundfile` (`pip install kokoro-onnx soundfile`). Non-English text also needs `espeak-ng` system-wide.
- **BGM (Lyria)** — needs `$GEMINI_API_KEY` or `$GOOGLE_API_KEY` + `pip install google-genai`. No local model cache.
Expand Down
7 changes: 5 additions & 2 deletions skills/media-use/audio/references/tts.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,11 @@ The script resolves a HeyGen credential the same way the CLI does — first sour
wins: `$HEYGEN_API_KEY` → `$HYPERFRAMES_API_KEY` → a project `.env` (auto-loaded,
walks up ≤5 dirs) → `~/.heygen/credentials` (shared with heygen-cli;
`$HEYGEN_CONFIG_DIR` overrides the dir). An OAuth login is sent as
`Authorization: Bearer`; an API key as `X-Api-Key`. If the only credential is an
expired OAuth token it stops with a hint to run `npx hyperframes auth refresh`.
`Authorization: Bearer`; an API key as `X-Api-Key`; both include
`X-HeyGen-Source: cli`. OAuth CLI users can consume the web-plan free allowance
(10 min/month) before paid usage; API keys follow normal API billing. If the
only credential is an expired OAuth token it stops with a hint to run
`npx hyperframes auth refresh`.

```bash
# Only needed if you haven't run `npx hyperframes auth login`:
Expand Down
2 changes: 1 addition & 1 deletion skills/media-use/audio/scripts/heygen-tts.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
//
// Flags: -o/--output (.wav → ffmpeg transcode; .mp3 → raw bytes), --words,
// --voice (starfish id), --speed, --lang, --list.
// Requires: $HEYGEN_API_KEY (or ~/.heygen) and ffmpeg for .wav output.
// Requires: $HEYGEN_API_KEY / OAuth ~/.heygen credentials and ffmpeg for .wav output.

import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { dirname, resolve } from "node:path";
Expand Down
3 changes: 2 additions & 1 deletion skills/media-use/audio/scripts/lib/heygen.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import { homedir } from "node:os";
import { dirname, join, resolve } from "node:path";

export const HEYGEN_BASE = "https://api.heygen.com/v3";
export const HEYGEN_CLI_SOURCE_HEADERS = { "X-HeyGen-Source": "cli" };

// Walk up ≤5 dirs from startDir; load the first .env (shell env always wins).
export function loadEnvFromDir(startDir) {
Expand Down Expand Up @@ -71,7 +72,7 @@ export function heygenCredential() {
// → auth headers object, or throw with a fix hint.
export function heygenAuthHeaders() {
const cred = heygenCredential();
if (cred?.headers) return cred.headers;
if (cred?.headers) return { ...cred.headers, ...HEYGEN_CLI_SOURCE_HEADERS };
if (cred?.expired)
throw new Error(
"HeyGen OAuth token expired — run `npx hyperframes auth refresh` (or `npx hyperframes auth login`)",
Expand Down
59 changes: 59 additions & 0 deletions skills/media-use/audio/scripts/lib/heygen.test.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { heygenAuthHeaders } from "./heygen.mjs";

function withCleanHeygenEnv(fn) {
const previousApiKey = process.env.HEYGEN_API_KEY;
const previousHyperframesApiKey = process.env.HYPERFRAMES_API_KEY;
const previousConfigDir = process.env.HEYGEN_CONFIG_DIR;
try {
delete process.env.HEYGEN_API_KEY;
delete process.env.HYPERFRAMES_API_KEY;
delete process.env.HEYGEN_CONFIG_DIR;
return fn();
} finally {
if (previousApiKey === undefined) delete process.env.HEYGEN_API_KEY;
else process.env.HEYGEN_API_KEY = previousApiKey;
if (previousHyperframesApiKey === undefined) delete process.env.HYPERFRAMES_API_KEY;
else process.env.HYPERFRAMES_API_KEY = previousHyperframesApiKey;
if (previousConfigDir === undefined) delete process.env.HEYGEN_CONFIG_DIR;
else process.env.HEYGEN_CONFIG_DIR = previousConfigDir;
}
}

test("heygenAuthHeaders tags API-key requests as CLI traffic", () => {
withCleanHeygenEnv(() => {
process.env.HEYGEN_API_KEY = "hg_test";
assert.deepEqual(heygenAuthHeaders(), {
"X-Api-Key": "hg_test",
"X-HeyGen-Source": "cli",
});
});
});

test("heygenAuthHeaders tags OAuth requests as CLI traffic", () => {
withCleanHeygenEnv(() => {
const dir = mkdtempSync(join(tmpdir(), "heygen-cred-"));
try {
process.env.HEYGEN_CONFIG_DIR = dir;
writeFileSync(
join(dir, "credentials"),
JSON.stringify({
oauth: {
access_token: "at_test",
expires_at: "2099-01-01T00:00:00Z",
},
}),
);
assert.deepEqual(heygenAuthHeaders(), {
Authorization: "Bearer at_test",
"X-HeyGen-Source": "cli",
});
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
});
4 changes: 3 additions & 1 deletion skills/media-use/references/operations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,9 @@ avatar upsell (decision X3).
Verified on 24GB: 512x320 x 33f with audio.
- **HeyGen avatar upsell (better, script-driven): the `heygen` CLI**, NOT the
raw API. For a talking-head / avatar video, `heygen video create` (avatar
engine IV by default) beats a generative clip when you want a real presenter:
engine IV by default) beats a generative clip when you want a real presenter.
Browser OAuth uses the web-plan/free avatar-video allowance where eligible;
API keys follow the normal API billing path:

```bash
# discover an avatar + a starfish voice, then create + wait
Expand Down
Loading
Loading