fix(skills): Update Agent Skills Based on Observed Failures Modes in Trials (#1945)

### What does this PR do?

Type of change: documentation

Updates the agent skills based on findings from the June 5 Day-0 trials:

- Strengthens PTQ checkpoint handoff, serving-readiness, and
representative calibration guidance.
- Adds external baseline sanity checks while allowing comparisons when
no credible external score exists.
- Pins SciCode to task-level `parallelism: 8` and `num_repeats: 1`.
- Adds positional BF16 layer exclusions as a quantization-recipe search
axis.
- Keeps day-0 release decisions autonomous and maps external baseline
mismatches to an explicit failure class.

Refs:
1. https://lab-e57330.gitlab-master-pages.nvidia.com/day0_jun5_analysis/
2.
https://docs.google.com/spreadsheets/d/1H-YvO5CDwKoG2IvvNWVdDI7rViW1Kc0ayVEJAP7RH_A/edit?gid=30281591#gid=30281591&range=A2

### Usage

N/A — agent skill guidance only.

### Testing

- Ran structural validation for all five changed skill packages.
- Ran targeted pre-commit hooks on all nine changed files; all passed.
- Ran `git diff --check`.

I'll test the skill changes by re-running day 0 trials after merging.

### Before your PR is "*Ready for review*"

- Is this change backward compatible?: ✅
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: N/A
- Did you write any new necessary tests?: N/A — documentation/skill
guidance only
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
N/A
- Did you get Claude approval on this PR?: ❌ — not run

### Additional Information

The branch contains one signed commit and excludes WAAP state,
test-fixture changes, and example changes.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Strengthened evaluation workflows with an “External Baseline Sanity
Check” requirement and clearer success/failure handling when external
verification fails or is inconclusive.
* Updated day-0 and compare-results guidance to follow the new
gate-driven sequence and to report external reference scores and sanity
status.
* Refined SciCode instructions to use fixed repeat settings and
consistent scoring output.
* Expanded PTQ checkpoint validation with downstream serving-readiness
canary requirements.
* Enhanced quantization recipe search with a layer-position axis,
positional exclusion rules, and stricter promotion criteria.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Chad Voegele <cvoegele@nvidia.com>

Author: chadvoegele

.agents/skills/compare-results/SKILL.md

@@ -32,11 +32,19 @@ change is being measured, typically a further quantized version of the baseline.
 5. For each task, use the canonical score field from the matching
    `.agents/skills/evaluation/recipes/tasks/<task>.md` Score Extraction
    section.
-6. Compute exact deltas outside the chat context when there are multiple tasks
+6. Read and perform `.agents/skills/evaluation/references/run-validation.md`
+   **External Baseline Sanity Check**. Record each source URL, protocol
+   difference, and task status before applying the candidate-delta gate. A
+   failed baseline blocks a success verdict; correct and rerun it first. If no
+   credible comparable reference exists, label the baseline externally
+   unverified rather than claiming the check passed, then continue using the
+   validated measured baseline.
+7. Compute exact deltas outside the chat context when there are multiple tasks
    or repeated runs.
-7. Report comparability and quantized-feasibility verdicts before interpreting
-   the delta as model quality. If the user did not provide an acceptance
-   threshold, report feasibility as inconclusive instead of inventing one.
+8. Report comparability, external baseline sanity, and quantized-feasibility
+   verdicts before interpreting the delta as model quality. If the user did not
+   provide an acceptance threshold, report feasibility as inconclusive instead
+   of inventing one.
 
 ## Comparability Checklist
 
@@ -64,17 +72,31 @@ the validated runs are comparable:
    precision the baseline is**, and apply the gate relative to that baseline
    rather than to an assumed BF16.
 
+For SciCode, keep `num_repeats: 1` to limit sandbox workload. If variance is a
+concern, run multiple independent matched baseline/candidate pairs instead of
+increasing repeats within one run.
+
 If any item differs, either rerun with matched settings or label the result as
 not an apples-to-apples quantization comparison.
 
+These checks compare the baseline and candidate to each other. The external
+baseline check in `evaluation/references/run-validation.md` separately tests
+whether the baseline's absolute score is credible; both guards must be reported.
+
 ## Report Format
 
 Include:
 
 - Baseline and candidate identifiers.
 - Per-task metric path, baseline score, candidate score, delta, and stderr if
   available.
+- Per-task external reference score, source URL, known protocol differences,
+  percentage-point difference, and sanity status (`verified`, `failed`, or
+  `externally unverified`).
 - Comparability status for prompt/template, generation settings, sample counts,
   reasoning handling, judge/simulator setup, and score field.
 - Comparability verdict: comparable, not comparable, or inconclusive.
 - Quantization feasibility verdict: acceptable, not acceptable, or inconclusive.
+  Never report `acceptable` when external baseline sanity failed. An externally
+  unverified baseline does not block `acceptable`; apply the candidate-delta
+  gate and report the missing external corroboration.

.agents/skills/day0-release/SKILL.md

@@ -52,9 +52,9 @@ progress:
 - [ ] Step 0: Resolve inputs; confirm threshold and eval set
 - [ ] Step 1: Setup gate — creds present, cluster reachable
 - [ ] Step 2: PTQ (ptq skill) → gate_ptq.py
-- [ ] Step 3: Baseline eval (evaluation skill, deploys source) → gate_run.py   [skip if cached, see below]
+- [ ] Step 3: Baseline eval (evaluation skill, deploys source) → gate_run.py
 - [ ] Step 4: Quantized eval (evaluation skill, deploys candidate) → gate_run.py
-- [ ] Step 5: Compare (compare-results skill) → gate_compare.py → decision
+- [ ] Step 5: Compare (compare-results skill) → external sanity → gate_compare.py → decision
 - [ ] Step 6: Closeout — report + publish recommendation
 ```
 
@@ -83,10 +83,8 @@ unvalidated checkpoint.
 ### Step 3 — Baseline eval
 
 The baseline is the **source** (pre-quantization) model on the same task set and
-sampling params. **Look it up first** — if a matching baseline run already
-exists in MLflow (same model, task set, sampling params), reuse it and skip this
-stage. Otherwise run it via the **evaluation** skill (which deploys the source
-model itself). Gate with `gate_run.py`.
+sampling params. Always run a fresh baseline via the **evaluation** skill,
+which deploys the source model itself. Gate with `gate_run.py`.
 
 ### Step 4 — Quantized eval
 
@@ -110,7 +108,14 @@ dropped samples) — do **not** compare scores from it.
 
 ### Step 5 — Compare
 
-Invoke the **compare-results** skill to produce per-task deltas, then gate:
+Invoke the **compare-results** skill. It must perform the shared external
+baseline sanity check before the candidate-delta gate. A failed check is
+`ANOMALOUS` with failure class `EXTERNAL_BASELINE_MISMATCH`: investigate and
+rerun the baseline. If no credible comparable external score exists, record the
+baseline as externally unverified and continue using the validated measured
+baseline.
+
+After recording the external status, produce per-task deltas and run:
 
 ```bash
 python .agents/skills/day0-release/scripts/gate_compare.py \
@@ -125,22 +130,25 @@ normalizes the drop accordingly, so `--threshold 0.01` means "≤1 pt on a 0-100
 task / ≤0.01 on a 0-1 task" uniformly. Pass `--scales '{"task": max}'` to
 override inference if a task's scores happen to fall in an ambiguous range.
 
-Decision from `gate_compare.py`:
+`gate_compare.py` checks only the candidate delta; it cannot override a failed
+external baseline check. Combined decision:
 
-- **ACCEPT** — every task within threshold → go to Step 6.
+- **ACCEPT** — no external check failed and every task is within the candidate
+  threshold → go to Step 6. A missing comparable external score is not a
+  failure; report it as externally unverified.
 - **REGRESSION** — one or more tasks exceed threshold. **v1 stops here and
   reports** which tasks regressed by how much. (Picking the next recipe and
   re-running is deferred — see Scope.)
-- **ANOMALOUS** — scores present but implausible (e.g. baseline lower than
-  candidate by a large margin, or a task score outside its valid range) →
-  surface to the user.
+- **ANOMALOUS** — external baseline sanity failed, or scores are otherwise
+  implausible (e.g. baseline lower than candidate by a large margin, or a task
+  score is outside its valid range) → correct the baseline or surface it.
 
 ### Step 6 — Closeout
 
 Report the decision with: source vs output size + ratio, per-task baseline /
-candidate / delta / within-threshold, MLflow run IDs, and a publish
-recommendation (publish / do-not-publish / needs-human). Archive artifacts to
-the workspace.
+candidate / delta / within-threshold, external source and sanity status, MLflow
+run IDs, and a publish recommendation (publish / do-not-publish).
+Archive artifacts to the workspace.
 
 ## Triage (gate failure → decision)
 
@@ -154,8 +162,9 @@ Map a gate's `failure_class` to the next action:
 | `DEPLOYMENT_HEALTH_FAILED` | Drop to the **deployment** skill: reproduce serving standalone (`/health` + one generation), debug flags / image / TP / env, then carry the working command into NEL's `deployment.command` and retry the eval. If it can't serve, `POINT_INFEASIBLE`. |
 | `EVAL_JUDGE_FAILED` | Usually transient (auth / rate limit) — wait and retry. |
 | `SAMPLE_ACCOUNTING_FAILED` | Investigate dropped/failed samples before trusting scores. |
-| `USER_CONFIG_ERROR` | Stop and ask the user. |
-| `UNKNOWN` | Stop and surface to the user (`NEEDS_HUMAN`). |
+| `EXTERNAL_BASELINE_MISMATCH` | Investigate baseline configuration, correct it, rerun the baseline, and repeat external sanity before comparison. |
+| `USER_CONFIG_ERROR` | Correct it from the request, workspace, or model/config metadata and retry; if irrecoverable, return `ANOMALOUS` with evidence. |
+| `UNKNOWN` | Investigate with the owning domain skill; if unresolved, return `ANOMALOUS` with the evidence and next automated retry or patch action. |
 
 `SYSTEMIC` (cluster down, dataset unavailable) aborts the whole run.
 `POINT_INFEASIBLE` means this (model, recipe) can't work as configured.
@@ -166,7 +175,7 @@ Return a decision, not a raw artifact:
 
 - `ACCEPT` + report + publish recommendation
 - `REGRESSION` + which tasks failed the threshold and by how much
-- `ANOMALOUS` / `INFEASIBLE` / `NEEDS_HUMAN` + reason
+- `ANOMALOUS` / `INFEASIBLE` + reason and next automated action
 - Always: workspace path + MLflow run IDs for traceability
 
 ## Scope (v1)

.agents/skills/evaluation/SKILL.md

@@ -409,7 +409,7 @@ Remove `limit_samples` overrides; keep canary-validated parallelism. If the cana
 
 ### Step 9 — Verify completed run
 
-Before pulling/reporting scores, validate the run. Read `references/run-validation.md` for NEL timeout/resume behavior, completed-run validation, diagnostics, score harvesting, and the handoff to `compare-results` for baseline-vs-candidate deltas.
+Before pulling/reporting scores, validate the run. Read `references/run-validation.md` for NEL timeout/resume behavior, completed-run validation, diagnostics, and score harvesting. For a baseline that will be compared with a candidate, also perform its **External Baseline Sanity Check** before a success verdict, then hand the validated runs to `compare-results` for baseline-vs-candidate deltas.
 
 ---
 

.agents/skills/evaluation/recipes/tasks/aa/scicode.md

@@ -17,6 +17,8 @@ harvesting requirements beyond the task YAML fragment.
   multi-step prompts can exceed 32K). The example template's default of
   `--max-model-len 131072` satisfies this and is preferred — do not lower
   it unless you have a memory reason to.
+- **Parallelism:** set task-level `parallelism: 8` exactly. Use the same value
+  for baseline and candidate.
 
 ## YAML Fragment
 
@@ -28,13 +30,12 @@ Use this inside the top-level `evaluation.tasks` list:
   nemo_evaluator_config:
     config:
       params:
+        parallelism: 8
         extra:
           args: ++prompt_config=eval/scicode/default ++with_background=true
-          num_repeats: 8
+          num_repeats: 1
 ```
 
 ## Score Extraction from mlflow
 
-Result (0-100): `scicode_pass_at_1_avg-of-N_subtask_accuracy`
-
-N is the repeat count.  If the repeat count is unknown, use the highest available `avg-of-N`.
+Result (0-100): `scicode_pass_at_1_avg-of-1_subtask_accuracy`

.agents/skills/evaluation/references/run-validation.md

@@ -35,6 +35,55 @@ accounting, reasoning/answer parsing status, and any errors or warnings found.
 If any validation item fails, either rerun/fix it or label the result as
 incomplete or invalid.
 
+## External Baseline Sanity Check
+
+For a baseline-vs-candidate comparison, perform this check after run validation
+and before applying the candidate-delta gate or issuing a success verdict. This
+is additional to, not a replacement for, the apples-to-apples and baseline
+precision checks in `compare-results`.
+
+For each baseline task:
+
+1. Search for a published score for the exact model in its Hugging Face model
+   card and on Artificial Analysis (<https://artificialanalysis.ai/>); use
+   either credible source. A score for a sibling size, release, or precision is
+   not an exact-model reference.
+2. Match model variant, benchmark and version, metric, reasoning/thinking mode,
+   prompt and chat template, sampling and token budget, sample count, and
+   evaluation protocol as closely as possible. Record the external score,
+   source URL, and every known protocol difference. Do not treat a mismatched
+   result as directly comparable; find a closer source or mark the task
+   externally unverified.
+3. Put both scores on a 0-100 scale, then calculate, for higher-is-better
+   metrics:
+
+   ```text
+   difference (pp) = abs(measured baseline - external)
+   ```
+
+   Treat a credible comparable result as verified only when the absolute
+   difference is approximately 5 percentage points or less. A difference
+   greater than approximately 5 points fails the check even if the candidate is
+   within its normal delta gate (for example, `<1pp`). For an external score of
+   60, the approximate range is 55-65; a baseline of 54 is 6 points away and
+   fails.
+
+Report each task as `verified`, `failed`, or `externally unverified`. A large
+upward difference also does not establish a clean match; investigate protocol
+differences before marking it verified. If no credible comparable score exists,
+state `externally unverified` and do not invent a reference or claim the sanity
+check passed. This status does not block comparison or publication: use the
+validated measured baseline, apply the candidate-delta gate, and report that no
+external corroboration was available. Only a `failed` external check blocks the
+comparison.
+
+If any task fails, do not report the quantized evaluation as successful and do
+not apply the candidate-delta gate to that baseline. Investigate disabled
+reasoning/thinking, reasoning parser or adapter handling, prompt/chat-template
+differences, sampling or token-budget differences, benchmark version or metric,
+incomplete samples, and serving failures. Rerun a corrected baseline, validate
+it, and repeat this check before comparing the candidate.
+
 For score harvesting, use the `Score Extraction` section from the matching task
 reference in `recipes/tasks/<task>.md`. Do not rely on ad hoc `results.yml`
 greps when a task reference defines the canonical score and stderr fields.

.agents/skills/ptq/SKILL.md

@@ -1,12 +1,21 @@
 ---
 name: ptq
-description: This skill should be used when the user asks to "quantize a model", "run PTQ", "post-training quantization", "NVFP4 quantization", "FP8 quantization", "INT8 quantization", "INT4 AWQ", "quantize LLM", "quantize MoE", "quantize VLM", or needs to produce a quantized HuggingFace or TensorRT-LLM checkpoint from a pretrained model using ModelOpt.
+description: >-
+  Use when the user asks to "quantize a model", "run PTQ", "post-training
+  quantization", "NVFP4 quantization", "FP8 quantization", "INT8
+  quantization", "INT4 AWQ", "quantize LLM", "quantize MoE", "quantize VLM",
+  or needs to produce a quantized HuggingFace checkpoint from a pretrained
+  model using ModelOpt. Do NOT use for multi-candidate recipe
+  exploration or optimization (use quant-recipe-search).
 ---
 
 # ModelOpt Post-Training Quantization
 
 Produce a quantized checkpoint from a pretrained model. **Read `examples/hf_ptq/README.md` first** — it has the support matrix, CLI flags, and accuracy guidance.
 
+Use `quant-recipe-search` for multi-candidate recipe exploration or
+optimization. Use this skill for each selected recipe's PTQ run.
+
 ## Step 1 — Environment
 
 Read `skills/common/environment-setup.md` and `skills/common/workspace-management.md`. After completing them you should know:
@@ -75,6 +84,24 @@ If the source checkpoint is already quantized and the requested recipe/config re
 For **listed models** (4A/4B): run full calibration directly (`--calib_size 512`).
 For **unlisted models** (4C): run a smoke test first (`--calib_size 4`), wait for success, then full calibration.
 
+### Recommended calibration datasets
+
+- **Text-only LLM PTQ:** Prefer the representative `nemotron-post-training-v3` blend. `modelopt/torch/utils/dataset_utils.py` expands it to seven registered Nemotron SFT domains. Configure Hugging Face credentials where required.
+
+  ```bash
+  python examples/hf_ptq/hf_ptq.py ... \
+      --dataset nemotron-post-training-v3
+  ```
+
+- **VLM PTQ:** Include image-text calibration with `--calib_with_images`. This path uses `nemotron_vlm_dataset_v2` with the current default subsets `sparsetables`, `plotqa_cot`, and `wiki_en`; `examples/hf_ptq/hf_ptq.py` and `modelopt/torch/utils/vlm_dataset_utils.py` are the source of truth.
+
+  ```bash
+  python examples/hf_ptq/hf_ptq.py ... \
+      --calib_with_images
+  ```
+
+`--dataset` selects text-only calibration and cannot substitute for multimodal examples. Use `--dataset cnn_dailymail` only as a fallback when representative data is unavailable, such as without gated-data access or with only a local public cache.
+
 ### Which path?
 
 ```text
@@ -138,17 +165,16 @@ Report the path and size to the user.
 
 ### Post-quantization validation
 
-This is a required gate before any deployment or evaluation submission. Do not submit an eval, start a serving job, or hand off the checkpoint as ready until the gate has passed.
+This is a required gate before any deployment or evaluation submission. Do not submit an eval, start a production serving job, or hand off the checkpoint as ready until the gate, including its serving canary, has passed.
 
-Read `references/checkpoint-validation.md` and perform all three validation groups on the exact checkpoint path that will be deployed/evaluated:
+Read `references/checkpoint-validation.md` and perform all four validation groups on the exact checkpoint path that will be deployed/evaluated:
 
 1. Check output size and estimated bits per weight against the baseline/source checkpoint.
 2. Check quantized-weight coverage against the requested qformat/recipe/config.
 3. Check metadata consistency against the baseline/source model.
+4. Complete the required downstream handoff and serving-readiness validation.
 
-Report the gate result before moving on. The report must include source size, output size, output/source size ratio, layer precision counts (for example NVFP4, FP8, INT4, BF16/unquantized excluded, unexpected unquantized, declaration mismatches), and metadata diffs. If the output/source ratio is >= 1.0 for a compression recipe, if any intended layer group is missing quantization, or if metadata changed unexpectedly, stop and fix the checkpoint or ask the user before proceeding.
-
-**Next steps**: If the user wants to deploy or evaluate the quantized checkpoint, use the **deployment** or **evaluation** skill. The checkpoint workspace carries over. If the model required patches during PTQ (e.g., transformers upgrade), the same fixes will likely be needed at deployment and evaluation time.
+Report the gate result before moving on. Follow the canonical report format and all blocking conditions in `references/checkpoint-validation.md`; do not hand off a checkpoint unless every required check passes.
 
 ## Key API Rules
 
@@ -163,7 +189,7 @@ Report the gate result before moving on. The report must include source size, ou
 
 - **Model-specific dependencies**: Models with `trust_remote_code` may import packages not in the container (e.g., `mamba-ssm` for hybrid Mamba models). See Step 2.5. Use `EXTRA_PIP_DEPS` env var with the launcher, or install manually before running `hf_ptq.py`
 - **Transformers version**: New models may need a newer version of transformers than what's installed. Check `config.json` for `transformers_version`. In containers, beware of `PIP_CONSTRAINT` blocking upgrades — see `references/slurm-setup-ptq.md` for workarounds
-- **Gated datasets**: Some calibration datasets require HF authentication. Ensure `HF_TOKEN` is set in the job environment, or use `--dataset cnn_dailymail` as a non-gated alternative
+- **Gated datasets**: Some calibration datasets require HF authentication. Set `HF_TOKEN` in the job environment. Use `--dataset cnn_dailymail` only as the constrained-environment fallback described in Step 4, not as the preferred calibration set
 - **NFS root_squash + Docker**: See `skills/common/slurm-setup.md` section 5
 
 ## References