TD-032-automate-model-characterization-pipeline.md

id	TD-032
title	Automate model onboarding & characterization (pull→bench→reasoning→report) — and fix the serve lifecycle that blocks it
status	proposed
date	2026-06-23
supersedes
superseded_by
tags	tooling benchmarking reasoning-eval serve-lifecycle automation cli

TD-032: Automate model onboarding & characterization (and fix the serve lifecycle that blocks it)

Context

Onboarding vibethinker-3b (GGUF + an OpenVINO INT4 A/B variant) and adding carbon reasoning showed that the atomic steps are already automated — carbon pull (GGUF download or optimum OV conversion), carbon bench -s (llama-bench tk/s → carbon.toml), carbon reasoning (graded math/logic → bench/runs/<model>/reasoning.json). What is not automated is the pipeline over them and across models — and one defect actively breaks any hands-off multi-model run.

The blocker: serve lifecycle leaks a process

carbon stop / engine.stop kills only the IPEX wrapper; the exec'd llama-server-bin child survives as an orphan — still holding iGPU memory and still LISTENing on :8080. Consequence: the next auto-serve probes /health, the orphan answers healthy, and the run reuses the wrong model. In the reasoning comparison sweep this silently reran gpt-oss-20b's problems against the still-resident qwen3-30b-thinking server (identical token counts gave it away) and forced a manual pkill to recover. bench/run_one.py already does this correctly (start_new_session=True + os.killpg); the serve path in carbon_llm/engines/llamacpp_sycl.py does not. (Footgun for the workaround: pkill -f 'llama-server' also matches any script whose command line contains that string — kill by port fuser -k 8080/tcp or by binary path instead.)

Decision

Make onboard → characterize → compare a first-class, hands-off loop. Three pieces, in order:

#	Piece	What	Where
1 ⭐	Fix serve lifecycle (correctness; do first)	stop kills the process group / the -bin child; auto-serve's reuse-check hard-fails on alias mismatch instead of warn, so a stale server can never corrupt a run	engines/llamacpp_sycl.py, bench/reasoning.py
2	Orchestrated characterization	carbon characterize <model> = pull → bench -s → reasoning → refresh bench/reports/<model>.md; a reasoning leaderboard aggregating reasoning.json across models (extend bench/run_all.sh + bench/report.py, mirroring the speed/chess leaderboard)	new CLI + bench/
3	OV harness parity	the ov-server must report finish="length" on truncation (today it always returns "stop", so OV reasoning truncations read as wrong answers) and honor top_p/top_k/seed for reproducibility	carbon_llm/_ov_server.py

Consequences

• (1) is a correctness fix, not a feature — it is the prerequisite for any automated sweep and removes the manual-pkill workaround that already corrupted one comparison.
• (2) turns "add a model" into one command and makes the cross-model comparison reproducible instead of ad-hoc bash.
• (3) makes engine A/Bs (llama.cpp/IPEX vs OpenVINO) measure the same thing — required before trusting an OV-vs-GGUF reasoning delta.
• carbon models field gaps (surfaced onboarding vibethinker-3b-ov): presence-detection ignored llm-openvino IRs — _model_local only checked the ov-models/<name> dir for image-openvino, so a converted OV LLM showed – ( fixed in cli.py); and there is no OV decode-throughput to display — carbon bench = llama-bench is GGUF-only, so the tk/s column stays — for OV models even though carbon reasoning clocked ~21 tok/s for this one. Piece (2) should also record an OV throughput number (reuse the reasoning-bench wall-clock, or add a small OV micro-bench) so the catalog isn't blank for OV.
• Non-goal: a generic eval framework. Keep the problem set small, verifiable, and in-repo (bench/reasoning.py); this TD is about plumbing the existing pieces together, not building a harness zoo.