Skip to content

2026-04-18_q3_k_s_tier.md

Latest commit

 

History

History
72 lines (48 loc) · 3.8 KB

2026-04-18_q3_k_s_tier.md

File metadata and controls

72 lines (48 loc) · 3.8 KB
Raw

Q3_K_S tier — Qwen3.6-35B-A3B on 16GB Mac

Headline: UD-Q3_K_S (3.5 bpw, 14.3 GB on disk) runs at 14.3 t/s warm peak on a 16 GB M1 Pro, 2.8× faster than llama.cpp's CPU path (5.11 t/s), at 5.24 GB RSS — the smallest working set of any Qwen3.6-35B quant we've measured, beating even the 2.05-bpw IQ2_XXS (6.54 GB).

Quality step up is real: IQ3_XXS failed the "author of Hamlet" factual probe; Q3_K_S answers "William Shakespeare wrote Hamlet" cleanly. Prose decode reaches "Jack loved to play with his guitar" vs IQ2's "Jack lived in the small village of the mountains" before drift.

All kernels used were already in place from commit 11e3c32; Q3_K_S simply exercises the Q3_K int8 path more heavily than the other UD variants.

Per-variant comparison (Qwen3.6-35B-A3B, M1 Pro 16GB, CPU 8t, TQ_NO_MLOCK=1, warm 3-run peak)

Variant bpw Disk RSS Decode (t/s) llama.cpp CPU Speedup
UD-IQ2_XXS 2.05 10.0 GB 6.54 GB 16.1 5.07 3.2×
UD-IQ3_XXS 3.06 12.3 GB 6.82 GB 14.6 5.23 2.8×
UD-Q3_K_S 3.5 14.3 GB 5.24 GB 14.3 5.11 2.8×

Quality, first 30 greedy tokens from "Once upon a time":

  • IQ2_XXS: "there was a young man named Jack. He lived in the small village of the mountains. and he had to" ← drift ~token 15
  • UD-IQ3_XXS: "there was a young man named Jack. He lived in the small village called 'Happiness'. One day, he he went to to go." ← coherent sentence structure
  • UD-Q3_K_S: "there was a young man named Jack. He lived in the countryside and he loved to play with his guitar. He would go to thethe" ← semantic richness (guitar, countryside, hobby)

Factual probe (single-shot, T=0, 30-token allowance):

  • "The capital of France is…" → Paris ✓
  • "The author of Hamlet is…" → William Shakespeare wrote Hamlet ✓ ← IQ3_XXS fails this one
  • (partial; 35B MoE at ~3.5 bpw still drifts on harder facts after ~10 tokens)

Why Q3_K_S RSS is smaller than IQ3_XXS

Counter-intuitive result: higher bpw (3.5 vs 3.06) but lower peak RSS (5.24 GB vs 6.82 GB).

Root cause is the page-cache working set, not the on-disk size. TQ_NO_MLOCK=1 lets the OS page out cold expert weights; what stays resident is the hot-expert set used in the last N tokens. Q3_K_S uses larger (256-element) Q3_K blocks vs IQ3_XXS's irregular mix of IQ3_XXS (256) + IQ3_S (256) + IQ4_XS (256) + Q4_K / Q6_K critical layers. The larger uniform block layout yields fewer distinct pages touched per matmul — same bytes per weight, but tighter page locality.

Practical implication: Q3_K_S is the recommended Qwen3.6 variant on 16 GB Macs as of this measurement round. Quality > IQ3_XXS, RSS < IQ3_XXS, speed ≈ IQ3_XXS.

Kernel coverage

All fast paths are already in place from this day's earlier commits:

  • Q3_K int8 (11e3c32) — primary kernel exercised by Q3_K_S
  • Q6_K int8 (9fdafaa) — used for attention.wo / ffn_down in UD variants
  • Q4_K via internal-Q4 conversion — shared experts + lm_head
  • IQ3_XXS / IQ3_S / IQ4_XS int8 — not used by Q3_K_S but kept enabled (zero-cost)

Profile sample on Q3_K_S decode (to be collected next round) expected to be dominated by q3_k_int_dot_worker.

Reproduce

# Download (14.3 GB)
cd models
curl -L -O "https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/resolve/main/Qwen3.6-35B-A3B-UD-Q3_K_S.gguf?download=true"
cd ..

# Decode
TQ_NO_METAL=1 TQ_NO_MLOCK=1 ./build/quant \
  models/Qwen3.6-35B-A3B-UD-Q3_K_S.gguf \
  --chat -p "Hello" -n 80 -T 0.7 -j 8

# Quality probe
bash scripts/qwen36_quality_probe.sh models/Qwen3.6-35B-A3B-UD-Q3_K_S.gguf 8

Regression

scripts/test_models.sh: 12/12 PASS (unchanged — Q3_K kernel was verified on the IQ3_XXS suite earlier; Q3_K_S exercises it end-to-end).