Skip to content

Commit 8f9a4db

Browse files
authored
Merge branch 'main' into jennifchen/te_per_expert
2 parents 16c2f0b + 42458de commit 8f9a4db

122 files changed

Lines changed: 7855 additions & 1195 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/_example_tests_runner.yml

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,17 +27,26 @@ on:
2727
required: false
2828
type: string
2929
default: "linux-amd64-gpu-rtxpro6000-latest-1"
30+
allow_failure:
31+
description: "If true, test failures are reported as a warning and do not fail the job (used to keep a known-broken example non-blocking)"
32+
required: false
33+
type: boolean
34+
default: false
3035

3136
jobs:
3237
run-test:
3338
runs-on: ${{ inputs.runner }}
3439
timeout-minutes: ${{ inputs.timeout_minutes }}
40+
permissions:
41+
contents: read
3542
container:
3643
image: ${{ inputs.docker_image }}
3744
options: --shm-size=2gb # TRT-LLM tests on 2-GPU runner needs more shared memory
3845
env:
3946
PIP_CONSTRAINT: "" # Disable pip constraint for upgrading packages
4047
HF_TOKEN: ${{ secrets.HF_TOKEN }}
48+
# Build CUDA kernels only for the runner's RTX PRO 6000 (sm_120), not the image's ~6 archs.
49+
TORCH_CUDA_ARCH_LIST: "12.0"
4150
steps:
4251
- uses: actions/checkout@v6
4352
- uses: nv-gha-runners/setup-proxy-cache@main
@@ -64,6 +73,8 @@ jobs:
6473
6574
find examples/${{ inputs.example }} -name "requirements.txt" | while read req_file; do python -m pip install -r "$req_file" || exit 1; done
6675
- name: Run tests
76+
id: run_tests
77+
continue-on-error: ${{ inputs.allow_failure }}
6778
env:
6879
# Absolute paths so subprocesses running from different working directories
6980
# all find the config and write .coverage.* files to the same location.
@@ -72,6 +83,10 @@ jobs:
7283
run: |
7384
echo "Running tests for: ${{ inputs.example }}"
7485
python -m pytest tests/examples/${{ inputs.example }} --cov
86+
- name: Flag allowed failure
87+
if: ${{ inputs.allow_failure && steps.run_tests.outcome == 'failure' }}
88+
run: |
89+
echo "::warning title=Allowed example failure::'${{ inputs.example }}' failed but is in the allow-failure list (vars.ALLOW_FAILURE_EXAMPLE_TESTS); not blocking. Remove it from the variable once fixed."
7590
- name: Upload coverage to Codecov
7691
uses: codecov/codecov-action@v5
7792
with:

.github/workflows/bump_uv_lock.yml

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,10 @@ jobs:
3737
- name: Check for changes
3838
id: changes
3939
run: |
40-
if git diff --quiet; then
40+
# Scope to uv.lock: the pyproject.toml torch-override step above rewrites
41+
# the whole file (toml.dump drops comments), so an unscoped diff is always
42+
# dirty and the no-update path below would fail on an empty commit.
43+
if git diff --quiet -- uv.lock; then
4144
echo "changed=false" >> "$GITHUB_OUTPUT"
4245
else
4346
echo "changed=true" >> "$GITHUB_OUTPUT"

.github/workflows/code_quality.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ on:
1111

1212
concurrency:
1313
# Cancel previous runs if new commit is pushed to the same PR
14-
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
14+
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
1515
cancel-in-progress: true
1616

1717
jobs:

.github/workflows/example_tests.yml

Lines changed: 27 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,9 @@ concurrency:
1414
group: ${{ github.workflow }}-${{ startsWith(github.ref, 'refs/heads/pull-request/') && github.ref || github.sha }}
1515
cancel-in-progress: true
1616

17+
# Each job's `allow_failure` reads the repo variable ALLOW_FAILURE_EXAMPLE_TESTS:
18+
# a comma-separated list of example names whose test failures should be non-blocking, e.g. "torch_trt,llm_qat"
19+
1720
jobs:
1821
pr-gate:
1922
uses: ./.github/workflows/_pr_gate.yml
@@ -22,6 +25,8 @@ jobs:
2225
secrets: inherit
2326
with:
2427
files: |
28+
.github/actions/cache-extensions/**
29+
.github/workflows/_example_tests_runner.yml
2530
.github/workflows/example_tests.yml
2631
examples/**
2732
modelopt/**
@@ -40,13 +45,16 @@ jobs:
4045
- example: speculative_decoding
4146
docker_image: "26.01"
4247
uses: ./.github/workflows/_example_tests_runner.yml
48+
permissions:
49+
contents: read
4350
secrets: inherit
4451
with:
45-
docker_image: "nvcr.io/nvidia/pytorch:${{ matrix.docker_image || '26.05' }}-py3"
52+
docker_image: "nvcr.io/nvidia/pytorch:${{ matrix.docker_image || '26.06' }}-py3"
4653
example: ${{ matrix.example }}
4754
timeout_minutes: 30
4855
pip_install_extras: "[hf,dev-test]"
4956
runner: ${{ startsWith(github.ref, 'refs/heads/pull-request/') && 'linux-amd64-gpu-rtxpro6000-latest-1' || 'linux-amd64-gpu-rtxpro6000-latest-2' }}
57+
allow_failure: ${{ contains(format(',{0},', vars.ALLOW_FAILURE_EXAMPLE_TESTS), format(',{0},', matrix.example)) }}
5058

5159
##### TensorRT-LLM Example Tests (pr/non-pr split: non-pr runs extra eval examples) #####
5260
trtllm-pr:
@@ -57,12 +65,15 @@ jobs:
5765
matrix:
5866
example: [hf_ptq]
5967
uses: ./.github/workflows/_example_tests_runner.yml
68+
permissions:
69+
contents: read
6070
secrets: inherit
6171
with:
62-
docker_image: "nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc19"
72+
docker_image: "nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc20"
6373
example: ${{ matrix.example }}
6474
pip_install_extras: "[hf,dev-test]"
6575
runner: linux-amd64-gpu-rtxpro6000-latest-1
76+
allow_failure: ${{ contains(format(',{0},', vars.ALLOW_FAILURE_EXAMPLE_TESTS), format(',{0},', matrix.example)) }}
6677

6778
trtllm-non-pr:
6879
if: ${{ !startsWith(github.ref, 'refs/heads/pull-request/') }}
@@ -71,25 +82,31 @@ jobs:
7182
matrix:
7283
example: [llm_eval, hf_ptq]
7384
uses: ./.github/workflows/_example_tests_runner.yml
85+
permissions:
86+
contents: read
7487
secrets: inherit
7588
with:
76-
docker_image: "nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc19"
89+
docker_image: "nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc20"
7790
example: ${{ matrix.example }}
7891
pip_install_extras: "[hf,dev-test]"
7992
runner: linux-amd64-gpu-rtxpro6000-latest-2
93+
allow_failure: ${{ contains(format(',{0},', vars.ALLOW_FAILURE_EXAMPLE_TESTS), format(',{0},', matrix.example)) }}
8094

8195
##### Megatron Example Tests #####
8296
megatron:
8397
needs: [pr-gate]
8498
if: needs.pr-gate.outputs.any_changed == 'true'
8599
uses: ./.github/workflows/_example_tests_runner.yml
100+
permissions:
101+
contents: read
86102
secrets: inherit
87103
with:
88104
docker_image: "nvcr.io/nvidia/nemo:26.06"
89105
example: megatron_bridge
90-
timeout_minutes: 45
106+
timeout_minutes: 60
91107
pip_install_extras: "[hf,puzzletron,dev-test]"
92108
runner: ${{ startsWith(github.ref, 'refs/heads/pull-request/') && 'linux-amd64-gpu-rtxpro6000-latest-1' || 'linux-amd64-gpu-rtxpro6000-latest-2' }}
109+
allow_failure: ${{ contains(format(',{0},', vars.ALLOW_FAILURE_EXAMPLE_TESTS), ',megatron_bridge,') }}
93110

94111
##### ONNX/TensorRT Example Tests #####
95112
onnx:
@@ -100,13 +117,19 @@ jobs:
100117
matrix:
101118
example: [diffusers, torch_onnx, torch_trt]
102119
uses: ./.github/workflows/_example_tests_runner.yml
120+
permissions:
121+
contents: read
103122
secrets: inherit
104123
with:
124+
# Pinned to 26.05 (TensorRT 10): torch-tensorrt is capped at <2.13 (== 2.12.1),
125+
# which needs libnvinfer.so.10; newer tensorrt containers drop it. Bump only once
126+
# a torch-tensorrt build for the newer TensorRT is available.
105127
docker_image: "nvcr.io/nvidia/tensorrt:26.05-py3"
106128
example: ${{ matrix.example }}
107129
timeout_minutes: 45
108130
pip_install_extras: "[onnx,hf,dev-test]"
109131
runner: ${{ startsWith(github.ref, 'refs/heads/pull-request/') && 'linux-amd64-gpu-rtxpro6000-latest-1' || 'linux-amd64-gpu-rtxpro6000-latest-2' }}
132+
allow_failure: ${{ contains(format(',{0},', vars.ALLOW_FAILURE_EXAMPLE_TESTS), format(',{0},', matrix.example)) }}
110133

111134
##### Required Check for PR #####
112135
example-pr-required-check:

.github/workflows/gpu_tests.yml

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ jobs:
2222
secrets: inherit
2323
with:
2424
files: |
25+
.github/actions/cache-extensions/**
2526
.github/workflows/gpu_tests.yml
2627
modelopt/**
2728
noxfile.py
@@ -40,13 +41,15 @@ jobs:
4041
include:
4142
- example: gpu
4243
timeout: 60
44+
# Pinned to 26.05: benchmark.py uses trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH,
45+
# which newer TensorRT (26.06) removed. Bump once the source is updated for TensorRT 10.
4346
container_image: nvcr.io/nvidia/pytorch:26.05-py3
4447
- example: gpu_megatron
45-
timeout: 75
48+
timeout: 60
4649
container_image: nvcr.io/nvidia/nemo:26.06
4750
- example: gpu_trtllm
4851
timeout: 15
49-
container_image: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc19
52+
container_image: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc20
5053
- example: gpu_vllm
5154
timeout: 15
5255
container_image: docker.io/vllm/vllm-openai:v0.20.0
@@ -58,6 +61,8 @@ jobs:
5861
GIT_DEPTH: 1000 # For correct version
5962
PIP_CONSTRAINT: "" # Disable pip constraint for upgrading packages
6063
HF_TOKEN: ${{ secrets.HF_TOKEN }}
64+
# Build CUDA kernels only for the runner's RTX PRO 6000 (sm_120), not the image's ~6 archs.
65+
TORCH_CUDA_ARCH_LIST: "12.0"
6166
steps:
6267
- name: Install git
6368
# The vllm container ships without git; needed for a real checkout (correct

.github/workflows/regression_tests.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,8 @@ jobs:
4444
GIT_DEPTH: 1000 # For correct version
4545
PIP_CONSTRAINT: "" # Disable pip constraint for upgrading packages
4646
HF_TOKEN: ${{ secrets.HF_TOKEN }}
47+
# Build CUDA kernels only for the runner's RTX PRO 6000 (sm_120), not the image's ~6 archs.
48+
TORCH_CUDA_ARCH_LIST: "12.0"
4749
steps:
4850
- uses: actions/checkout@v6
4951
- uses: nv-gha-runners/setup-proxy-cache@main

.github/workflows/unit_tests.yml

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,7 @@ jobs:
6969
env:
7070
COVERAGE_PROCESS_START: ${{ github.workspace }}/pyproject.toml
7171
COVERAGE_FILE: ${{ github.workspace }}/.coverage
72-
run: pip install nox uv && nox -s "unit-3.12(torch_212, tf_latest)"
72+
run: pip install nox uv && nox -s "unit-3.12(torch_213, tf_latest)"
7373
- name: Upload coverage reports to Codecov
7474
uses: codecov/codecov-action@v5
7575
with:
@@ -86,13 +86,15 @@ jobs:
8686
needs: [linux, check-file-changes]
8787
runs-on: windows-latest
8888
timeout-minutes: 15
89+
permissions:
90+
contents: read
8991
steps:
9092
- uses: actions/checkout@v6
9193
- uses: actions/setup-python@v6
9294
with:
9395
python-version: "3.12"
9496
- name: Run unit tests (without coverage)
95-
run: pip install nox uv && nox -s "unit-3.12(torch_212, tf_latest)"
97+
run: pip install nox uv && nox -s "unit-3.12(torch_213, tf_latest)"
9698
multi-version:
9799
if: needs.check-file-changes.outputs.any_changed == 'true'
98100
needs: [linux, check-file-changes]
@@ -102,15 +104,19 @@ jobs:
102104
fail-fast: false
103105
matrix:
104106
include:
105-
- {nox_session: "unit-3.10(torch_212, tf_latest)", python_version: "3.10"}
106-
- {nox_session: "unit-3.11(torch_212, tf_latest)", python_version: "3.11"}
107-
- {nox_session: "unit-3.13(torch_212, tf_latest)", python_version: "3.13"}
108-
- {nox_session: "unit-3.14(torch_212, tf_latest)", python_version: "3.14"}
107+
# Default torch (2.13) across the other supported Python versions
108+
- {nox_session: "unit-3.10(torch_213, tf_latest)", python_version: "3.10"}
109+
- {nox_session: "unit-3.11(torch_213, tf_latest)", python_version: "3.11"}
110+
- {nox_session: "unit-3.13(torch_213, tf_latest)", python_version: "3.13"}
111+
- {nox_session: "unit-3.14(torch_213, tf_latest)", python_version: "3.14"}
112+
# Older torch versions on the default Python (3.12) for back-compat.
109113
- {nox_session: "unit-3.12(torch_28, tf_latest)", python_version: "3.12"}
110114
- {nox_session: "unit-3.12(torch_29, tf_latest)", python_version: "3.12"}
111115
- {nox_session: "unit-3.12(torch_210, tf_latest)", python_version: "3.12"}
112116
- {nox_session: "unit-3.12(torch_211, tf_latest)", python_version: "3.12"}
113-
- {nox_session: "unit-3.12(torch_212, tf_min)", python_version: "3.12"}
117+
- {nox_session: "unit-3.12(torch_212, tf_latest)", python_version: "3.12"}
118+
# Minimum supported transformers on the default torch.
119+
- {nox_session: "unit-3.12(torch_213, tf_min)", python_version: "3.12"}
114120
steps:
115121
- uses: actions/checkout@v6
116122
- uses: ./.github/actions/ubuntu-setup

.pre-commit-config.yaml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,8 @@ repos:
116116
modelopt/torch/speculative/plugins/modeling_domino.py|
117117
modelopt/torch/speculative/plugins/hf_dflash.py|
118118
modelopt/torch/speculative/plugins/modeling_dflash.py|
119+
modelopt/torch/speculative/plugins/hf_dspark.py|
120+
modelopt/torch/speculative/plugins/modeling_dspark.py|
119121
modelopt/torch/speculative/plugins/hf_medusa.py|
120122
modelopt/torch/utils/plugins/megatron_mmlu.py|
121123
examples/deepseek/deepseek_v3/quantize_to_nvfp4.py|

CHANGELOG.rst

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@ Changelog
1111

1212
**Deprecations**
1313

14+
- ``examples/hf_ptq`` AutoQuantize is now driven by an **AutoQuantize recipe** (``--recipe``). The ``--auto_quantize_bits``, ``--auto_quantize_method``, ``--auto_quantize_score_size``, ``--auto_quantize_cost_model``, and ``--auto_quantize_active_moe_expert_ratio`` flags are **deprecated** but still work: they are converted into an ``AutoQuantizeConfig`` on the fly (emitting a ``DeprecationWarning``) and will be removed in a future release. Prefer a recipe under ``modelopt_recipes/general/auto_quantize/``. See ``examples/hf_ptq/README.md``.
15+
1416
- Renamed ``examples/llm_ptq`` to ``examples/hf_ptq`` to reflect that it covers Hugging Face LLM **and** VLM PTQ. A relative symlink ``examples/llm_ptq`` -> ``hf_ptq`` keeps existing paths and commands working; it will be removed in a future release. Please update references to the new ``examples/hf_ptq`` path.
1517
- Consolidated ``examples/vlm_ptq`` into ``examples/hf_ptq``. Vision-language model PTQ now shares the ``hf_ptq.py`` entry point and ``scripts/huggingface_example.sh``; pass ``--vlm`` to run the TensorRT-LLM multimodal quickstart smoke test. The ``examples/vlm_ptq/scripts/huggingface_example.sh`` entry point is deprecated: it now prints a warning and forwards to the ``hf_ptq`` script with ``--vlm``, and will be removed in a future release. See `examples/hf_ptq/README.md <https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/hf_ptq#vlm-quantization>`__.
1618
- Dropped VILA / NVILA vision-language model support in ``examples/hf_ptq``. VILA's modeling code requires ``transformers<=4.50.0``, which conflicts with ModelOpt's minimum supported ``transformers`` version. The VILA-specific bootstrap (repo clone, ``requirements-vila.txt``) and loading paths in ``example_utils.py`` have been removed.
@@ -40,11 +42,14 @@ Changelog
4042
- ``hf_ptq.py`` also unwraps ``ModelOutput`` dataclasses from ``.generate()`` so the preview decode works on diffusion models. Non-tied models see no behavioral change.
4143
- Add **Domino** speculative-decoding training: the parallel DFlash draft backbone plus a lightweight GRU causal correction head, selected via ``dflash_architecture_config.projector_type=domino``. Trained with a base/final dual loss whose ``dflash_lambda_base_start``/``dflash_lambda_base_decay_ratio`` curriculum decays the base-loss weight 1→0. Exports in the z-lab drafter format; recipe at ``modelopt_recipes/general/speculative_decoding/domino.yaml``. Training only — the inference path is not wired up yet.
4244
- Add Torch-TensorRT FP8 deployment example for HuggingFace ViT (``examples/torch_trt/``): ``torch_tensorrt_ptq.py`` covers ``mtq.quantize`` → ``torch_tensorrt.compile(ir="dynamo")``, and ``torch_tensorrt_accuracy.py`` reports the compiled model's ImageNet-1k top-1/top-5 accuracy via the ``onnx_ptq`` ``evaluate`` harness (the unquantized baseline is Torch-TensorRT-compiled too, for an apples-to-apples comparison). Ships a ViT-tuned FP8 PTQ recipe under ``modelopt_recipes/huggingface/vit/ptq/`` (``fp8.yaml``) composed from the shared ``modelopt_recipes/configs/`` units: it quantizes the encoder Linears, patch-embed ``nn.Conv2d``, ``classifier``, and per-block LayerNorm inputs plus the attention Q/K/V BMMs and softmax. Verified on ``google/vit-base-patch16-224`` (ImageNet-1k 50k validation): FP8 stays within 0.13 pp Top-1 of the FP16 baseline.
45+
- Add **AutoQuantize recipe** support: ``mtq.auto_quantize`` can be driven declaratively from a YAML recipe (``RecipeType.AUTO_QUANTIZE`` / ``AutoQuantizeConfig``) specifying candidate formats, the ``effective_bits`` target, cost model (incl. ``active_moe`` and ``excluded_module_name_patterns``), scoring method, and disabled layers. Adds an ``effective_bits`` cost-model override on ``QuantizeConfig`` / ``QuantizerAttributeConfig`` (block-scale-accurate NVFP4 = 4.5 via ``configs/numerics/nvfp4``). Shipped recipes live under ``modelopt_recipes/general/auto_quantize/`` and model-specific ones under ``modelopt_recipes/huggingface/<model>/auto_quantize/``.
46+
- Add ``rotate.mode`` to torch quantizer configs. The default ``"rotate"`` keeps the existing rotate-before-quantize behavior; ``"rotate_back"`` enables fake-quant rotate → quantize → rotate-back for TensorQuantizer.
4347

4448
**Bug Fixes**
4549

4650
- Fix ``ShapeInferenceError`` during ONNX INT8 + FP16 quantization (``--high_precision_dtype fp16``) of weakly-typed models (e.g. TensorFlow exports) that carry stale rank-0 ``graph.output`` shapes or ops such as ``TopK`` that ONNX's static shape inference cannot resolve. ``clear_stale_value_info`` now reconciles stale output shapes via symbolic shape inference (keeping every output's shape field populated), and AutoCast runs ONNX shape inference in strict mode and falls back to schema-based standalone type inference when it fails, so unresolved ops no longer leave tensors untyped.
4751
- Fused MoE expert auto-detection (``register_fused_experts_on_the_fly``) no longer requires an ``act_fn`` attribute. Some fused-expert modules (e.g. ``MiniMaxM3VLExperts``) apply a custom gated activation between the two ``F.linear`` calls instead of exposing ``act_fn``; they were silently skipped, leaving routed experts unquantized (an experts-only recipe matched nothing) and failing HF export with ``NotImplementedError``. ``_QuantFusedExperts`` is activation-agnostic (it only intercepts the two ``F.linear`` calls), so the requirement was unnecessary. This enables NVFP4/FP8 quantization and export for MiniMax-M2 / MiniMax-M3.
52+
- Fix unified HF export emitting transformers' *in-memory* (post-``conversion_mapping``) tensor names instead of the original model-hub names, breaking the unified-checkpoint contract (observed on MiniMax-M3: exported ``model.language_model.*`` / ``mlp.experts.*.gate_proj`` instead of hub ``language_model.model.*`` / ``block_sparse_moe.experts.*.w{1,2,3}``). transformers' own save-side ``revert_weight_conversion`` is disabled by ModelOpt because it raises ``RuntimeError`` on 0-d scalar scale tensors, so a new quant-aware reverse conversion (``modelopt/torch/export/quant_aware_conversion.py``) derives rename/split rules from the model's conversion mapping via transformers' ``reverse_transform()`` and carries each weight's companion scale tensors (``weight_scale``, ``weight_scale_2``, ``input_scale``, ``weight_scale_inv``, ``bias``) through the renames and un-fusions, so quantized exports round-trip to the hub names. Any mapping op that cannot be reversed quant-aware yet (e.g. still-stacked fused experts) falls back to the previous in-memory names instead of aborting the export.
4853

4954
0.45 (2026-07-02)
5055
^^^^^^^^^^^^^^^^^

LICENSE

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -247,6 +247,7 @@ the following copyright holders, licensed under the MIT License:
247247
Copyright (c) 2023 Deep Cognition and Language Research (DeCLaRe) Lab
248248
Copyright (c) 2023 DeepSeek
249249
Copyright (c) 2025 sgl-project
250+
Copyright (c) 2026 The DeepSpec Authors
250251

251252
Permission is hereby granted, free of charge, to any person obtaining a copy
252253
of this software and associated documentation files (the "Software"), to deal

0 commit comments

Comments
 (0)