bench

bench/

External benchmark and comparison tooling. Anything that runs outside the Gleam package API lives here — Python and R scripts for statistical analysis, cross-runtime comparisons against PyTorch / NumPy, shell runners, results, and the technical plans behind the numbers.

Gleam-side benchmark entrypoints (the ones invoked via gleam run -m viva_tensor/bench/...) live under dev/viva_tensor/bench/ instead — kept out of src/ so they don't ship in the Hex package.

Layout

bench/
├── README.md           # this file
├── python/             # Python benchmarks + cross-runtime comparisons
│   ├── benchmark.py        # main statistical suite (bootstrap CI, normality, effect size)
│   ├── gpu_benchmark.py    # CUDA peak / kernel-only timing
│   └── compare/            # head-to-head against external runtimes
│       ├── README.md           # how the side-by-side flow works
│       ├── numpy_pytorch.py    # vs PyTorch + NumPy (feeds dev/showdown.gleam)
│       ├── numpy_cpu.py        # CPU-only vs NumPy (writes results/numpy_cpu_baseline.csv)
│       └── side_by_side.py     # prints numpy vs viva table from the two CSVs
├── r/                  # R-based statistical analysis + plots
│   └── analysis.R
├── scripts/            # shell runners
│   └── run_benchmarks.sh
├── results/            # measurement outputs (CSVs, .term, markdowns)
│   ├── matmul_showdown.md      # viva vs PyTorch vs NumPy across shapes / dtypes
│   ├── (pytorch_results.term)  # generated by python/compare/numpy_pytorch.py
│   ├── (numpy_cpu_baseline.csv) # generated by python/compare/numpy_cpu.py
│   └── archive/                # historical baselines + legacy scripts
└── plans/              # technical roadmaps + future work
    ├── INFERENCE_API_PLAN.md   # [DONE] kept for history (W8A16 + per-block-16 shipped)
    ├── CUTLASS_DSL_NOTES.md    # CUTLASS 4 CuTeDSL migration notes
    └── NVFP4_EVT_PLAN.md       # NVFP4 fused dequant + GEMM design

Quick start

# Head-to-head vs PyTorch / NumPy (matches dev/viva_tensor/bench/showdown.gleam)
.bench-venv/bin/python3 bench/python/compare/numpy_pytorch.py
# -> writes bench/results/pytorch_results.term

# CPU-only NumPy baseline
python3 bench/python/compare/numpy_cpu.py
# -> writes bench/results/numpy_cpu_baseline.csv

# Side-by-side table (after running both halves)
python3 bench/python/compare/side_by_side.py

# Full statistical sweep with bootstrap CI
python3 bench/python/benchmark.py

# CUDA peak FLOPS / TOPS
python3 bench/python/gpu_benchmark.py

# Drive everything in one go (CI)
./bench/scripts/run_benchmarks.sh

Reading results

• results/matmul_showdown.md — measured numbers across all backends + shapes, with hardware/software context.
• results/archive/ — older baselines kept for regression tracking.

Plans / design notes

• plans/INFERENCE_API_PLAN.md [DONE] — historical roadmap for FP8 prepack + linear + SwiGLU + sparse. All shipped; kept as context.
• plans/CUTLASS_DSL_NOTES.md — future migration to CUTLASS 4 CuTeDSL.
• plans/NVFP4_EVT_PLAN.md — NVFP4 fused dequant + GEMM (parked, Blackwell-class hardware only).

What changed in this reorg

• compare/ (RESULTS, plans, numpy_pytorch.py) and perf/ (NumPy CPU scripts + CSV) were merged: scripts under python/compare/, outputs under results/, plans under plans/.
• python/benchmark_compare.py was redundant with compare/numpy_pytorch.py and was moved to results/archive/ as legacy reference.

Removed earlier (kept here for the record)

• bench/erlang/ — 22 escript benchmarks superseded by dev/viva_tensor/bench/*.gleam.
• bench/cuda/test_int8_imma.cu — covered by peak.gleam.
• bench/windows/ — .bat runners; project doesn't ship Windows as a first-class target.