feat(experimentalist): add the tau3-bench banking benchmark suite (#935)

* feat(experimentalist): make the benchmark harness suite-driven

The runner hardcoded Terminal Bench's workspace and framework skill and assumed a
package held exactly one domain, so a second suite could not be expressed as a
manifest.

A manifest now carries its own workspace and framework_skills, and may set
task_id_prefix to scope a multi-domain package to one domain; coverage is then
enforced against the matching subset. Skill directories resolve before the
baseline evaluation, because otherwise a typo'd skill name surfaces hours of
image builds later.

A config may also name a user_simulator, which exports the OpenAI-style
credentials that tau-style task environments and their NL-assertion judge read.
Those are assigned rather than defaulted: a developer's ambient OPENAI_API_KEY
paired with this gateway's base URL would 401 on every call.

Signed-off-by: Aditya Pandey <aditypandey@nvidia.com>

* feat(experimentalist): add the tau3-bench banking suite and configs

Scopes sierra-research/tau3-bench@1 to its 97 banking_knowledge tasks. The
quality partition reuses the 41/28/28 split from optimization-datasets
feat/tau2-other-domains, whose tau2-banking-knowledge-NNN names map onto the
canonical Hub IDs; only task IDs are recorded here, never task content.

The 6/3/3 fast partition draws solely from the 87 tasks whose reward_basis is
pure database state, so smoke runs score without an LLM judge. Both partitions
were validated against live Harbor Hub.

Signed-off-by: Aditya Pandey <aditypandey@nvidia.com>

* feat(experimentalist): add the tau3 NOOA agent under test

A NOOA CodeAct agent that reaches the tau2 domain tools over MCP from the task's
tau3-runtime sidecar. AGENT-SPEC.md stays domain-generic because the policy
arrives at runtime in the task instruction, and it is what the Experimentalist
mutates.

Three things the task images required:

Installing from the committed lock uses the pip that python:3.12-slim already
ships, since tau3 images carry neither uv nor the curl its installer wants.
Terminal Bench's uv-upload machinery is unnecessary here because every tau3 task
shares that one base image.

Model credentials are handed over at exec time. A task's [environment].env only
interpolates its docker-compose file, and upstream wires those variables into the
sidecar; the main service declares no environment, so the agent had none.

mcp_timeout.py replaces nooa 0.0.6's MCP transport, which is built without an
httpx timeout and so inherits httpx's 5 second default. The sidecar runs the user
simulator inside start_conversation and send_message_to_user, which take longer;
the server completed the work and advanced its state, but the reply landed on an
abandoned stream and nooa surfaced no timeout, so the agent hung indefinitely.

Verified end to end on the fast test split: all three tasks complete with no
harness error, one solved, for a 1/3 baseline the optimizer can improve on.

Signed-off-by: Aditya Pandey <aditypandey@nvidia.com>

* docs(experimentalist): document the tau3 banking benchmark

Turns the single-suite README into one that covers both suites: how a manifest
carries its own workspace, framework skills, and optional domain prefix, the
banking provenance and partition derivation, and the two-container topology with
its memory and build-timeout consequences.

Also records that upstream's tau3 Dockerfiles clone tau2-bench without a ref, so
a cold layer cache can pick up newer commits than an earlier run saw.

Signed-off-by: Aditya Pandey <aditypandey@nvidia.com>

* refactor(experimentalist): write partition entries relative to the task-id prefix

Every ID in a domain-scoped package repeats the domain prefix, so listing 109 of
them buried the part that actually differs and made the banking manifest 143
lines of near-identical strings.

Partition entries are now names inside the suite's task_id_prefix, joined back on
at load so the rest of the runner keeps working in canonical IDs. The banking
manifest drops to 50 lines and its splits are legible at a glance. Suites without
a prefix, meaning Terminal Bench, are unaffected.

Signed-off-by: Aditya Pandey <aditypandey@nvidia.com>

* refactor(experimentalist): name benchmark configs after their suite

A config carries the models and optimizer depth for one suite, but the original
pair was named only for its partition. Once tau3-smoke.yaml and tau3-quality.yaml
sat beside them, smoke.yaml and quality.yaml read as the generic pair rather than
the Terminal Bench one.

Signed-off-by: Aditya Pandey <aditypandey@nvidia.com>

* docs(experimentalist): trim the tau3 section to what a reader needs

Drops the container topology, the memory and build-timeout guidance, and the note
about upstream's unpinned tau2-bench clone. What remains mirrors the Terminal
Bench section: package, hash, Hub record, partition derivation, and the one
config field the suite introduces.

Signed-off-by: Aditya Pandey <aditypandey@nvidia.com>

* build(experimentalist): repin nooa to the upstream MCP timeout fix

nooa built its streamable-http transport with an httpx client that had no
timeout, so httpx's 5 second default governed the MCP read stream and any tool
call taking longer than that hung forever. The tau3 agent hit this on every
turn, because the tau3-runtime sidecar runs a user-simulator LLM inside
start_conversation and send_message_to_user.

That is fixed upstream in NVIDIA-NeMo/labs-OO-Agents#46, so this drops the local
mcp_timeout.py shim and goes back to MCPManager.create_from_server, which now
takes tool_call_timeout directly. The agent still asks for 300 seconds: nooa's
60 second default would do today but leaves no room for a slower model.

The pin is a commit rather than a tag because no release carries the fix yet.
The lock's extra wheel rows are generated: no package version moves except nooa.

Verified against a live sidecar that start_conversation now returns in 7.3s
instead of hanging, and that a full Harbor trial completes with no harness error.

Signed-off-by: Aditya Pandey <aditypandey@nvidia.com>

---------

Signed-off-by: Aditya Pandey <aditypandey@nvidia.com>

Author: callingmedic911

plugins/nemo-experimentalist/AGENTS.md

@@ -72,9 +72,11 @@ instead of restoring Curator imports, configuration, or aliases. The obsolete
   Configure Platform services, trace storage, and Insights analysis according
   to the Platform documentation; this repository does not own a service,
   scheduler, or testbed setup.
-- `nooa` is pinned to a tagged public GitHub release in `pyproject.toml`.
-  Update the tag and lock file together. Keep the Platform-supplied Insights
-  plugin separate.
+- `nooa` is pinned to an immutable public GitHub revision in `pyproject.toml`,
+  currently a commit rather than a tag because the MCP transport-timeout fix
+  landed after `v0.0.6`. Update the revision, the matching pin in
+  `examples/tau3-nooa-agent/pyproject.toml`, and both lock files together. Keep
+  the Platform-supplied Insights plugin separate.
 - This branch uses merged Platform PR 718 contracts only. After the Platform
   handoff lands, rebase and repin before adopting any new Platform testbed or
   installer interfaces.

plugins/nemo-experimentalist/README.md

@@ -28,8 +28,8 @@ export NEMO=/path/to/nemo-platform/.venv/bin/nemo
 ```
 
 The source dependencies are pinned to tagged or immutable revisions in
-`pyproject.toml`. NVIDIA-labs OO Agents (NOOA) is pinned to its public
-GitHub `v0.0.6` release.
+`pyproject.toml`. NVIDIA-labs OO Agents (NOOA) is pinned to a public GitHub
+commit, currently one past `v0.0.6` that carries an MCP transport-timeout fix.
 
 ## Insight-to-experiment flow
 

plugins/nemo-experimentalist/benchmarks/README.md

@@ -1,13 +1,20 @@
 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 
-# Canonical Experimentalist benchmark
+# Canonical Experimentalist benchmarks
 
-This benchmark measures M2 Experimentalist optimization on Harbor's unmodified
-Terminal-Bench 2.1 package. It does not cover M1 Insight → Eval Author behavior
-or Tau2's interactive domains.
+These benchmarks measure M2 Experimentalist optimization on unmodified Harbor Hub
+packages. They do not cover M1 Insight → Eval Author behavior.
 
-## Provenance
+Two suites ship today. Both store only task IDs and download task definitions into
+a local cache; no task content is vendored here.
+
+| Suite | Package | Tasks | Agent under test |
+| --- | --- | --- | --- |
+| `suites/terminal-bench-2.1.yaml` (default) | `terminal-bench/terminal-bench-2-1@6` | 89 | `examples/terminal-bench-agent` |
+| `suites/tau3-banking.yaml` | `sierra-research/tau3-bench@1`, banking scoped | 97 | `examples/tau3-nooa-agent` |
+
+## Terminal-Bench provenance
 
 The suite uses `terminal-bench/terminal-bench-2-1@6` from Harbor Hub:
 
@@ -17,21 +24,41 @@ The suite uses `terminal-bench/terminal-bench-2-1@6` from Harbor Hub:
 - Harbor Hub record:
   <https://hub.harborframework.com/datasets/terminal-bench/terminal-bench-2-1/6>
 
-The repository stores only task IDs. Task definitions are downloaded into a
-local cache and are never vendored here.
-
 The 38/25/26 quality partition and 35/12/12 fast partition came from Gaia's
 `optimization-datasets` commit
 `1b7688ad257dacd1a3267dddb88db6cefdc31376`. The manifest corrects
 `install-windows-3-11` to the canonical ID `install-windows-3.11`. At startup,
 the runner asks Harbor Hub for revision 6, verifies its content hash, and
 asserts that the quality partition covers all 89 canonical IDs exactly once.
 
+## tau3 banking provenance
+
+The suite scopes `sierra-research/tau3-bench@1` to its `banking_knowledge` domain:
+
+- 97 of the package's 375 tasks, selected by the
+  `tau3-bench__tau3-banking_knowledge-` task-ID prefix
+- dataset content hash
+  `sha256:a57304f682894ac061090769af771a3617664f3ff6e5417d4eadf8e30433e4d9`
+- Harbor Hub record:
+  <https://hub.harborframework.com/datasets/sierra-research/tau3-bench/1>
+
+The 41/28/28 quality partition came from `optimization-datasets`
+`feat/tau2-other-domains` commit `025ecd2ef2b518ad81f6b22d3f3937af8906fb01`,
+whose `tau2-banking-knowledge-NNN` names map onto the canonical
+`tau3-bench__tau3-banking_knowledge-task-NNN` IDs. The 6/3/3 fast partition draws
+only from the 87 tasks whose `reward_basis` is pure database state, so smoke runs
+score deterministically without an LLM judge.
+
+Each task also runs a `tau3-runtime` sidecar hosting the tau2 environment and user
+simulator. Tau-style suites set `models.user_simulator`, which makes the runner
+export `OPENAI_API_KEY`, `OPENAI_BASE_URL`, `TAU2_USER_MODEL`, and
+`TAU2_NL_ASSERTIONS_MODEL` for the sidecar and the verifier.
+
 ## Held-out evaluation
 
 The runner:
 
-1. evaluates the unchanged LangChain baseline on the test split;
+1. evaluates the unchanged baseline agent on the test split;
 2. gives only train and validation IDs to `run_experimentalist`;
 3. resolves the validation-selected winner from the Experimentalist run;
 4. evaluates that winner on the same test IDs and number of attempts.
@@ -52,7 +79,7 @@ export INFERENCE_API_KEY=...
 ```
 
 `EXPERIMENTALIST_API_KEY` may be provided instead. The runner maps either credential
-to both the LangChain AUT and optimizer, using
+to both the AUT and optimizer, using
 `https://inference-api.nvidia.com/v1` unless an API base is explicitly set.
 
 Validate provenance and task IDs without Docker or model calls:
@@ -65,20 +92,28 @@ Run the bounded fast benchmark:
 
 ```bash
 uv run python benchmarks/run.py \
-  --config benchmarks/configs/smoke.yaml
+  --config benchmarks/configs/terminal-bench-smoke.yaml
 ```
 
 Run the reproducible quality benchmark:
 
 ```bash
 uv run python benchmarks/run.py \
-  --config benchmarks/configs/quality.yaml
+  --config benchmarks/configs/terminal-bench-quality.yaml
+```
+
+A non-default suite needs its own suite, config, and agent:
+
+```bash
+uv run python benchmarks/run.py \
+  --suite benchmarks/suites/tau3-banking.yaml \
+  --config benchmarks/configs/tau3-smoke.yaml \
+  --agent examples/tau3-nooa-agent
 ```
 
-Both setup and agent execution require network access. The AUT installs a
-checksum-pinned static `uv`, uv-managed Python 3.12, and dependencies from its
-committed `uv.lock` directly in each canonical task container. It does not need
-the image's system Python, package manager, a sidecar, or a Docker socket.
+Both setup and agent execution require network access. Each AUT installs its own
+dependencies from its committed `uv.lock` inside the task container and does not
+need a Docker socket.
 
 Pass the same `--output` directory to resume interrupted Harbor jobs. Use a new
 output directory for an intentionally fresh run.

plugins/nemo-experimentalist/benchmarks/configs/tau3-quality.yaml

@@ -0,0 +1,34 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+suite_partition: quality
+test_attempts: 3
+models:
+  aut: openai/openai/openai/gpt-5-mini
+  experimentalist_smart: openai/openai/openai/gpt-5-mini
+  experimentalist_mid: openai/openai/openai/gpt-5-mini
+  experimentalist_fast: openai/openai/openai/gpt-5-mini
+  user_simulator: openai/openai/openai/gpt-5-mini
+optimizer:
+  max_rounds: 3
+  min_rounds_before_stopping: 2
+  max_survivors: 2
+  max_candidates: 3
+  max_trajectory_tasks: 8
+  max_train_batch_tasks: 16
+  train_batch_seed: 20260727
+  disable_trajectory_scoring: false
+  disable_convergence_check: false
+  evaluator:
+    n_attempts: 2
+    # Keep at 3 or lower: each task requests 8192 MB and parallel image builds
+    # have triggered Docker Hub rate limiting and DNS failures.
+    n_concurrent_trials: 3
+    quiet: true
+    agent_setup_timeout_multiplier: 2.0
+    # tau3 builds two images per task, and the first build clones and installs
+    # tau2-bench (observed 2m36s cold, 26s warm).
+    environment_build_timeout_multiplier: 3.0
+  eval_author:
+    max_traces: 10
+    max_validation_repair_attempts: 5

plugins/nemo-experimentalist/benchmarks/configs/tau3-smoke.yaml

@@ -0,0 +1,32 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+suite_partition: fast
+test_attempts: 1
+models:
+  aut: openai/openai/openai/gpt-5-mini
+  experimentalist_smart: openai/openai/openai/gpt-5-mini
+  experimentalist_mid: openai/openai/openai/gpt-5-mini
+  experimentalist_fast: openai/openai/openai/gpt-5-mini
+  user_simulator: openai/openai/openai/gpt-5-mini
+optimizer:
+  max_rounds: 1
+  min_rounds_before_stopping: 1
+  max_survivors: 1
+  max_candidates: 1
+  max_trajectory_tasks: 2
+  max_train_batch_tasks: 4
+  train_batch_seed: 20260727
+  disable_trajectory_scoring: true
+  disable_convergence_check: true
+  evaluator:
+    n_attempts: 1
+    n_concurrent_trials: 1
+    quiet: true
+    agent_setup_timeout_multiplier: 2.0
+    # tau3 builds two images per task, and the first build clones and installs
+    # tau2-bench (observed 2m36s cold, 26s warm).
+    environment_build_timeout_multiplier: 3.0
+  eval_author:
+    max_traces: 3
+    max_validation_repair_attempts: 2

…entalist/benchmarks/configs/quality.yaml …arks/configs/terminal-bench-quality.yamlplugins/nemo-experimentalist/benchmarks/configs/quality.yaml renamed to plugins/nemo-experimentalist/benchmarks/configs/terminal-bench-quality.yaml plugins/nemo-experimentalist/benchmarks/configs/quality.yaml renamed to plugins/nemo-experimentalist/benchmarks/configs/terminal-bench-quality.yaml

@@ -1,7 +1,7 @@
 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 
-"""Run the canonical Terminal-Bench Experimentalist benchmark."""
+"""Run a canonical Experimentalist benchmark suite."""
 
 import argparse
 import asyncio
@@ -32,7 +32,7 @@
 PLUGIN_ROOT = Path(__file__).resolve().parents[1]
 BENCHMARK_ROOT = Path(__file__).resolve().parent
 DEFAULT_SUITE = BENCHMARK_ROOT / "suites" / "terminal-bench-2.1.yaml"
-DEFAULT_CONFIG = BENCHMARK_ROOT / "configs" / "smoke.yaml"
+DEFAULT_CONFIG = BENCHMARK_ROOT / "configs" / "terminal-bench-smoke.yaml"
 DEFAULT_AGENT = PLUGIN_ROOT / "examples" / "terminal-bench-agent"
 DEFAULT_DATASET_CACHE = PLUGIN_ROOT / "tmp" / "benchmark-datasets"
 DEFAULT_RUNTIME_CACHE = PLUGIN_ROOT / "tmp" / "runtime-cache"
@@ -45,6 +45,7 @@ class CanonicalDatasetSpec(BaseModel):
     registry_url: str
     source_url: str
     expected_task_count: int = Field(gt=0)
+    task_id_prefix: str | None = None
 
     @property
     def requested_reference(self) -> str:
@@ -88,13 +89,44 @@ class SuiteSpec(BaseModel):
     schema_version: Literal[1]
     dataset: CanonicalDatasetSpec
     partitions: PartitionsSpec
+    workspace: str
+    framework_skills: list[str] = Field(min_length=1)
+
+    @model_validator(mode="after")
+    def expand_partition_task_ids(self) -> Self:
+        """Read partition entries as names inside the suite's ``task_id_prefix``.
+
+        Every ID in a domain-scoped package repeats that prefix, which buries the part
+        that differs. Manifests write the distinguishing name and the prefix is joined
+        back on here, so the rest of the runner still works in canonical IDs. A manifest
+        that writes full IDs anyway fails the quality-coverage check with both forms in
+        the message.
+        """
+        prefix = self.dataset.task_id_prefix
+        if prefix is None:
+            return self
+        for split in (self.partitions.quality, self.partitions.fast):
+            for role in ("train", "validation", "test"):
+                setattr(split, role, [f"{prefix}{name}" for name in getattr(split, role)])
+        return self
+
+    def framework_skills_dirs(self, plugin_root: Path) -> list[Path]:
+        """Resolve framework-skill names to directories, failing on unknown names."""
+        dirs: list[Path] = []
+        for name in self.framework_skills:
+            path = plugin_root / "framework-skills" / name
+            if not path.is_dir():
+                raise ValueError(f"Unknown framework skill {name!r}: {path} is not a directory")
+            dirs.append(path)
+        return dirs
 
 
 class ModelSpec(BaseModel):
     aut: str
     experimentalist_smart: str
     experimentalist_mid: str
     experimentalist_fast: str
+    user_simulator: str | None = None
 
 
 class BenchmarkConfig(BaseModel):
@@ -126,13 +158,23 @@ def validate_canonical_suite(
     *,
     canonical_task_ids: set[str],
     resolved_ref: str,
-) -> None:
-    """Verify immutable revision, quality coverage, and all partition IDs."""
+) -> set[str]:
+    """Verify immutable revision, quality coverage, and all partition IDs.
+
+    Returns the canonical task IDs scoped to this suite's domain. When the suite
+    sets ``task_id_prefix``, the package may hold other domains; coverage is then
+    enforced against the matching subset rather than the whole package.
+    """
     if resolved_ref != suite.dataset.resolved_ref:
         raise ValueError(
             f"Dataset {suite.dataset.requested_reference} resolved to {resolved_ref}, "
             f"expected {suite.dataset.resolved_ref}"
         )
+    prefix = suite.dataset.task_id_prefix
+    if prefix is not None:
+        canonical_task_ids = {task_id for task_id in canonical_task_ids if task_id.startswith(prefix)}
+        if not canonical_task_ids:
+            raise ValueError(f"No canonical tasks match task_id_prefix {prefix!r}")
     if len(canonical_task_ids) != suite.dataset.expected_task_count:
         raise ValueError(
             f"Canonical dataset has {len(canonical_task_ids)} tasks, expected {suite.dataset.expected_task_count}"
@@ -147,6 +189,7 @@ def validate_canonical_suite(
     unknown_fast = sorted(set(suite.partitions.fast.all_ids()) - canonical_task_ids)
     if unknown_fast:
         raise ValueError(f"Fast partition contains unknown canonical task IDs: {unknown_fast}")
+    return canonical_task_ids
 
 
 def _configure_models(models: ModelSpec) -> None:
@@ -164,6 +207,16 @@ def _configure_models(models: ModelSpec) -> None:
     os.environ["EXPERIMENTALIST_MID_MODEL_NAME"] = models.experimentalist_mid
     os.environ["EXPERIMENTALIST_FAST_MODEL_NAME"] = models.experimentalist_fast
     os.environ.setdefault("NEMO_EXPERIMENTALIST_RUNTIME_CACHE", str(DEFAULT_RUNTIME_CACHE))
+    if models.user_simulator is not None:
+        # tau-style tasks run a user simulator and NL-assertion judge inside the task
+        # environment, which read OpenAI-style variables and need the /v1 suffix.
+        # Assigned rather than setdefault: an ambient OPENAI_API_KEY from the developer's
+        # shell would otherwise be paired with this gateway base URL and always 401.
+        trimmed_base = api_base.rstrip("/")
+        os.environ["OPENAI_API_KEY"] = api_key
+        os.environ["OPENAI_BASE_URL"] = trimmed_base if trimmed_base.endswith("/v1") else f"{trimmed_base}/v1"
+        os.environ["TAU2_USER_MODEL"] = models.user_simulator
+        os.environ["TAU2_NL_ASSERTIONS_MODEL"] = models.user_simulator
 
 
 def _agent_digest(agent_dir: Path) -> str:
@@ -342,7 +395,12 @@ async def run_benchmark(args: argparse.Namespace) -> Path:
     package_client = PackageDatasetClient()
     metadata = await package_client.get_dataset_metadata(suite.dataset.requested_reference)
     canonical_task_ids = {task.name for task in metadata.task_ids}
-    validate_canonical_suite(suite, canonical_task_ids=canonical_task_ids, resolved_ref=metadata.version)
+    canonical_task_ids = validate_canonical_suite(
+        suite, canonical_task_ids=canonical_task_ids, resolved_ref=metadata.version
+    )
+    # Resolved up front: the optimizer only needs these after the baseline evaluation,
+    # which is hours of image builds to discover a typo'd skill name.
+    framework_skills_dirs = suite.framework_skills_dirs(PLUGIN_ROOT)
 
     if args.validate_only:
         print(
@@ -397,10 +455,10 @@ async def run_benchmark(args: argparse.Namespace) -> Path:
             metadata={"id": f"{benchmark_config.suite_partition}-validation", "task_ids": split.validation},
         ),
         experiment_dir=experimentalist_dir,
-        workspace="canonical-terminal-bench-2-1",
+        workspace=suite.workspace,
         client=None,
         config=benchmark_config.optimizer,
-        framework_skills_dirs=[PLUGIN_ROOT / "framework-skills" / "langchain-framework"],
+        framework_skills_dirs=framework_skills_dirs,
     )
     run_document = json.loads((experimentalist_dir / "eval-and-optimize" / "run.json").read_text(encoding="utf-8"))
     winner_label = run_document.get("winner_agent")