AutoResearch-SH

A Self-Healing Reliability Framework for LLM-Generated Research Artifacts

Moving LLM research from "Generate-and-Pray" to "Audit-and-Heal."

---

Key Results

Metric	Value
Pytest	75/75 passed
Benchmark cases	8 (6 benchmark + 2 real)
Precision	0.733
Recall	0.733
F1	0.733
Over-correction rate	0.000
Avg calibration gap	+31.6 → +17.0
Honest claim rate	44% → 100%

Design choice: high-recall conservative auditing. AutoResearch-SH is designed as a high-recall conservative auditor. It favors surfacing unsupported or insufficiently grounded numerical claims over silently accepting them. Precision, recall, F1, and over-correction rate are therefore reported together.

---

Why This Matters

LLM agents can generate plausible research artifacts — but their outputs often contain:

• Fabricated metrics (claimed numbers not produced by code)
• Inflated self-scores (92/100 when independent review gives 62/100)
• Non-reproducible experiments (import errors, hardcoded dummies)
• Weak provenance (numbers with no traceable source)

AutoResearch-SH detects these issues, applies conservative claim-level refinements, and re-audits to confirm improvement. Unlike ordinary LLM-agent demos, it reports benchmark precision, recall, F1, over-correction rate, and ablation results on a seeded benchmark with ground-truth labels.

---

Pipeline Overview

Research Output (manuscript + code)
   ↓
Reviewer — 35%-evidence rubric, default REJECT
   ↓
Experiment Runner — actually executes code.py
   ↓
Provenance Auditor — rules-first trace of every numeric claim to source
   ↓
Headline Recomputer — calibration gap + honest claim rate
   ↓
Refiner — surgical claim downgrades + code-patch suggestions
   ↓
Re-audit — confirms gap narrowed, honest rate rose
   ↓
Dashboard — V1 → V2 comparison + Case Explorer

Every numeric claim is classified as MATCH (byte-traced to source), MISMATCH (different number in source), or UNTRACEABLE (not found in source). The Refiner flags non-MATCH claims with audit annotations, and the re-audit confirms the calibration gap narrowed.

---

Quick Start

# Install dependencies
pip install pydantic streamlit pandas matplotlib numpy

# One-command reproduction of all results
make all

# Or step by step:
PYTHONPATH=src python run_pipeline.py --force    # run pipeline on all 8 cases
PYTHONPATH=src python scripts/evaluate_benchmark.py  # compute precision/recall/F1
PYTHONPATH=src python scripts/run_ablation.py        # ablation study A0→A4
PYTHONPATH=src python scripts/export_paper_assets.py  # export CSV/LaTeX/SVG
PYTHONPATH=src python scripts/freeze_results.py       # freeze results for drift detection

# Launch the dashboard
make dashboard

---

Benchmark Cases

Case	Domain	Seeded Failure Mode
case_000	ML Benchmarking	Conservativeness test — all claims valid, system should not over-correct
case_001	Sim2Real	Fabricated metric — claims 12.4% improvement; code produces 8.1%
case_002	Federated Learning	Broken experiment — ImportError, experiment BLOCKED
case_003	Synthetic Biology	Subscale inflation — subscale sums inflated to 80/100
case_004	Agent Safety	P-hacking — claims 5 seeds ± 0.3 std; code runs 1 seed
case_005	Green AI	Fake baseline — baseline is hardcoded dummy (0.50)

Each case has an expected_issues.json ground-truth file. The benchmark evaluator computes precision, recall, F1, and over-correction rate against these labels. The 2 real-world cases (case_real_001, case_real_002) are included for validation without seeded ground-truth labels.

---

Metrics

Calibration Gap

self_score − reviewer_score. Positive = over-confident. The pipeline narrows this gap.

Honest Claim Rate

(traceable_claims + flagged_claims) / total_claims. A claim is "honest" if it is either byte-traced to source or explicitly flagged as untrusted by the Refiner. This replaces raw traceability, which can be misleading (flagging untraceable claims is honest behavior, not dishonesty). Honest Claim Rate does not imply that all claims are experimentally verified.

Over-Correction Rate

system_incorrectly_modified_non_issues / total_non_issues. A reliability auditor must not only detect unsupported claims, but also avoid damaging already-correct claims. Zero over-correction is a core design goal.

Benchmark Evaluation

Precision / Recall / F1 computed against expected_issues.json ground truth. Each claim is classified as TP (true positive), FP (false positive), TN (true negative), or FN (false negative).

---

Ablation Study

Variant	Avg Gap	Honest Rate	Claim Fixes	Code Patches
A0: Self only	+27.2	N/A	0	0
A1: + Reviewer	+27.2	N/A	0	0
A2: + Runner	+27.2	N/A	0	0
A3: + Auditor	+27.2	32%	0	0
A4: Full Self-Healing	+11.0	100%	26	6

Trend: A0→A4 gap decreases, honest claim rate increases, over-correction stays at zero.

---

Dashboard

The Streamlit dashboard includes:

1. Self-score vs Reviewer-score — with calibration gap arrows
2. Audit Funnel — MATCH / MISMATCH / UNTRACEABLE per case
3. Claim Traceability Table — per-claim audit findings
4. Version Improvement Timeline — V1 → V2 gap and honest rate
5. Case Explorer — per-claim evidence chain with ground truth comparison (TP/FP/TN/FN)

Launch: make dashboard

---

Project Structure

├── run_pipeline.py              # Orchestrator (checkpoint/resume)
├── Makefile                     # One-command reproduction
├── src/auto_research_sh/
│   ├── schemas.py               # Pydantic models for all stages
│   ├── stages.py                # Deterministic reviewer/runner/auditor
│   └── refiner.py               # Self-healing refiner module
├── tests/                       # 75 pytest tests (seed, tables, honest rate, conservativeness, e2e, case_000 regression)
├── benchmarks/                  # 6 seeded cases with expected_issues.json
├── scripts/
│   ├── evaluate_benchmark.py    # Precision/recall/F1 against ground truth
│   ├── run_ablation.py          # A0→A4 ablation study
│   ├── export_paper_assets.py   # Export CSV/LaTeX/SVG
│   └── freeze_results.py        # Freeze results for drift detection
├── dashboard/app.py             # Streamlit 5-section dashboard
├── paper_assets/                # Auto-generated tables, LaTeX, figures
├── frozen_results/              # Frozen result snapshots
├── paper/                       # Paper draft
├── .github/workflows/ci.yml     # CI: test + pipeline + evaluate + ablation
├── LIMITATIONS.md               # Honest limitations
└── RELEASE_CHECKLIST.md         # Release checklist

---

Reproducibility

• make all reproduces all results from scratch
• CI runs on every push: tests + pipeline + evaluation + ablation + export
• frozen_results/ detects data drift between runs
• All pipeline outputs are JSON with Pydantic schema validation
• No LLM API key required (deterministic stages)

---

Limitations

See LIMITATIONS.md for the full list. Key points:

1. Provenance auditing is rules-first and mainly numeric
2. Refiner does not automatically apply code patches (safety choice)
3. Experiment execution is local subprocess (Docker sandbox planned)
4. Benchmark is seeded and small-scale (8 cases)
5. Refiner is deterministic (LLM-backed Refiner planned for P2)

---

Roadmap

P2 — Engineering Hardening

• Docker sandbox execution
• AST-based executable provenance
• RAG provenance over logs / CSV / figures
• LLM-backed Refiner Agent
• Real-world benchmark expansion

---

Citation

If you use AutoResearch-SH, please cite:

@software{autoresearch_sh_2026,
  title={AutoResearch-SH: A Self-Healing Reliability Framework for LLM-Generated Research Artifacts},
  author={AutoResearch-SH Team},
  year={2026},
  version={0.2.0}
}

License

Public good. Use, modify, redistribute. No warranty.