[Bug]: DeepGemm accuracy auto-disable (_DEEPGEMM_BLACKWELL_EXCLUDED_MODEL_TYPES) does not apply to the FP8 MoE path

[ryansummerskill]

Your current environment

• Hardware: NVIDIA GB10, SM121 / compute capability 12.1 (also relevant to SM120 generally)
• vLLM 0.26.0 (standard PyPI wheel, aarch64), torch 2.11.0+cu130, --enforce-eager, TP=1
• Model: Qwen3.6-35B-A3B-FP8 — model_type: **qwen3_5_moe_text**, 256-expert FP8 MoE

🐛 Describe the bug

vLLM correctly identifies that DeepGemm is inaccurate for this model type and logs so at startup:

WARNING [vllm.py:1008] Auto-disabled DeepGemm for model_type=qwen3_5_moe_text on Blackwell.
  DeepGemm E8M0 scale format causes accuracy degradation for this architecture.
  Falling back to CUTLASS. To disable DeepGemm globally, set VLLM_USE_DEEP_GEMM=0.

…and then selects DeepGemm anyway for the MoE layers:

INFO [fp8.py:405] Using DEEPGEMM Fp8 MoE backend out of potential backends:
  ['AITER', 'FLASHINFER_TRTLLM', 'FLASHINFER_CUTLASS', 'DEEPGEMM', 'TRITON', 'MARLIN', ...]

The auto-disable sets quant_config.use_deep_gemm = False (vllm/config/vllm.py), and
qwen3_5_moe_text is present in _DEEPGEMM_BLACKWELL_EXCLUDED_MODEL_TYPES
(vllm/utils/deep_gemm.py) — so the denylist is doing its job. But use_deep_gemm appears to have
no references anywhere under vllm/model_executor/layers/fused_moe/; the FP8 MoE backend
selector honors only the VLLM_USE_DEEP_GEMM / VLLM_MOE_USE_DEEP_GEMM env vars. Net effect: the
denylist covers dense linear layers only, and the MoE layers — where the accuracy concern for an MoE
model actually matters most — still run DeepGemm.

So the warning is truthful about dense layers and silently wrong about MoE.

Why this surfaced now

PR #43477 ("Enable DeepSeek V4 and GLM-5.1 on SM120", shipped v0.24.0) added
is_device_capability_family(120) to deep_gemm.py. Prior to that, the SM100-family gate meant
DeepGemm was never eligible on SM121, so this gap was latent. The FP8 MoE backend priority list
itself is unchanged — DeepGemm simply became eligible and started winning.

Consistent with this, the original accuracy fix #37804 reasoned that "MoE layers are unaffected
(already use FLASHINFER_TRTLLM)" — which holds on SM100 but not on SM121, where
FLASHINFER_TRTLLM is gated out (#43906).

Impact

1. Accuracy — the exact degradation [Bug] DeepGemm E8M0 scale format causes accuracy degradation for Qwen3.5 FP8 on Blackwell #37804 set out to prevent still applies to this model's MoE
   layers on SM120/121.
2. Performance — measurable on our hardware. Forcing Triton MoE (matching what vLLM 0.20.1
   selected) recovered most of a regression vs 0.20.1: median TPOT went 37.3 → 34.9ms at
   concurrency 1 and 43.7 → 39.3ms at concurrency 4 for this model.

Expected behavior

The _DEEPGEMM_BLACKWELL_EXCLUDED_MODEL_TYPES denylist should gate the FP8 MoE backend
selection as well as dense linear layers — i.e. Fp8MoeBackend selection should consult the same
use_deep_gemm/model-type exclusion rather than only the env vars.

Workaround

--moe-backend triton (or VLLM_MOE_USE_DEEP_GEMM=0). We shipped --moe-backend triton.

Related

• [Usage]: vLLM 0.24.0 crashes on startup with Qwen3.6-27B-FP8 on Blackwell SM120 — DeepGemm warmup ignores auto-disable #47169 (open) is the same gap in the warmup path ("DeepGemm warmup ignores auto-disable"),
  reported on SM120 with a GB10/SM121 reproducer in the comments — likely the same fix area.
• [Bug] DeepGemm E8M0 scale format causes accuracy degradation for Qwen3.5 FP8 on Blackwell #37804 (closed) — origin of the denylist.
• Enable DeepSeek V4 and GLM-5.1 on SM120 #43477 — made DeepGemm SM120-family eligible.
• [Bug]: v0.20 latency and throughput regression on MoE models #41306 (open) — maintainer-confirmed MoE backend-selection perf regression of a similar shape.

Note on SM121 specifically

On SM121 the alternatives to Triton for FP8 MoE all appear unusable, which is why --moe-backend triton is the only workaround we found: FLASHINFER_TRTLLM gated to SM100 (#43906), VLLM_CUTLASS has
no SM120/121 CollectiveBuilder (#43507), MARLIN produces corrupt output on GB10 (#49546), HUMMING
crashes at startup (#49554).

[cr-zhao]

working on this

[vladimir-voinea]

Confirmed — independent reproduction on a second GB10 deployment, and I verified the gap is still present on main as of 38a267c.

Environment: single-node NVIDIA DGX Spark (GB10, sm_121a), vllm/vllm-openai:v0.26.0 container (arm64), Qwen/Qwen3.6-35B-A3B-FP8 (blockwise FP8 checkpoint), --speculative-config MTP k=3.

Boot log (excerpted) shows the same contradiction — and that the dense path honors the flag while the MoE path engages the exact E8M0 scale format the denylist exists to avoid:

APIServer:  Auto-disabled DeepGemm for model_type=qwen3_5_moe_text on Blackwell. DeepGemm E8M0 scale
            format causes accuracy degradation for this architecture. Falling back to CUTLASS. [...]
EngineCore: Selected CutlassFp8BlockScaledMMKernel for Fp8LinearMethod        <-- dense: flag honored
EngineCore: Using DEEPGEMM Fp8 MoE backend out of potential backends: [...]   <-- MoE: flag ignored
EngineCore: DeepGEMM E8M0 enabled on current platform.

Code-level confirmation on the v0.26.0 tag:

• Config layer sets quant_config.use_deep_gemm = False and logs the CUTLASS fallback: vllm/config/vllm.py#L997-L1014
• Denylist incl. qwen3_5_moe_text, gated on capability family 100/120: vllm/utils/deep_gemm.py#L27-L46
• Dense path honors it — Fp8LinearMethod reads quant_config.use_deep_gemm: quantization/fp8.py#L292-L295
• MoE path ignores it — select_fp8_moe_backend consults only VLLM_USE_DEEP_GEMM / VLLM_MOE_USE_DEEP_GEMM, and its signature never receives the quant config: fused_moe/oracle/fp8.py#L358-L371. In auto mode DEEPGEMM then outranks VLLM_CUTLASS/TRITON in _get_priority_backends, so it wins wherever it is eligible.
• Fp8MoEMethod calls the oracle at quantization/fp8.py#L527, where self.quant_config is already in scope — so threading the flag into select_fp8_moe_backend looks straightforward.

Still present on main (38a267c): the env-only check is unchanged (fused_moe/oracle/fp8.py#L358-L371, auto-disable now at config/vllm.py#L1033-L1050), and git grep use_deep_gemm -- vllm/model_executor/layers/fused_moe/ still returns no reference to the quant-config flag (only the env-var checks and *_e8m0 helpers).

Since essentially all weight FLOPs in an A3B-class MoE are expert layers, the auto-disable currently protects only the small dense fraction while the startup log claims a full CUTLASS fallback.

Workaround note: the coarse VLLM_USE_DEEP_GEMM=0 (which the warning itself suggests) also works on our deployment — on an excluded model it costs nothing extra, since the dense path has already fallen back to CUTLASS via the flag.

[cr-zhao]

I just found that #47258 already covers this issue, including the missing FP8 MoE backend-selection path. To avoid duplicating the existing work, I’m going to stop working on a separate pr here and follow up on that PR instead.
One implementation detail may be worth considering there: passing the already-resolved quant_config.use_deep_gemm value into select_fp8_moe_backend, rather than re-deriving the decision from the global model type. This would keep the dense and MoE paths aligned on the same resolved configuration.