Skip to content

Commit 03861f5

Browse files
docs(benchmarks): separate verified from historical, drop unreproduced claims
BENCHMARKS.md was stamped v2.1.0, presented old micro-benchmarks as current, and footed every table with 'verified via GitHub Actions' — but CI runs tests, it does not benchmark. It also headlined Omega '99.4% retrieval accuracy / drift prediction approved', which no committed harness reproduces. Rewritten to hold the same bar the tool preaches: - Merkle integrity table kept and marked verified (81 tests + hand-confirmed on 3.2.2: drift -> exit 1 fail-closed). - Latency split into a current 3.2.2 end-to-end CLI measurement (with methodology: wall time, dominated by process start-up) and the v2.1.0 pure-scan numbers, clearly labelled historical. - Omega accuracy figures removed as headline results until a runnable harness exists, same as the release notes' anti-overclaim stance. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 83328ba commit 03861f5

1 file changed

Lines changed: 49 additions & 26 deletions

File tree

BENCHMARKS.md

Lines changed: 49 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,36 +1,59 @@
1-
# 📊 Ethernium Performance Benchmarks
2-
**Date:** April 5, 2026
3-
**Version:** v2.1.0 Nexus
1+
# 📊 Chronolith Performance Benchmarks
42

5-
This document presents the technical performance and results of the **Chronolith Framework** under industrial workloads.
3+
**Current version:** 3.2.2
64

7-
## 🏁 Scan & Crystallization Latency
8-
Tests performed on a repository with **512 files** (~10MB total metadata size).
5+
This document separates what is **verified** from what is **historical or not yet
6+
re-measured**, because a drift-detection tool that overstates its own numbers is
7+
exactly the thing it exists to catch.
98

10-
| Profile | Engine | File Count | Latency (Avg) | CPU Overhead |
11-
| :--- | :--- | :--- | :--- | :--- |
12-
| **Initial Scan** | Lite | 500+ | 145ms | < 2% |
13-
| **Incremental Update**| Lite | 1 (Change) | **12ms** | < 0.5% |
14-
| **DNA Audit** | Pro | 500+ | 410ms | < 5% |
15-
| **Deep Parity Check** | Pro | 500+ | 850ms | < 10% |
9+
## 🧬 Cryptographic reliability (Merkle integrity) — verified
1610

17-
## 🧬 Criptographic Reliability (Merkle Integrity)
18-
Tests on bitwise corruption detection.
11+
These behaviours are covered by the test suite (81 Pro tests, green) and were
12+
re-confirmed by hand on 3.2.2: introduce the change, run `chronolith check`,
13+
observe the exit code.
1914

20-
| Test Case | Error Type | Detection Status | Block Action |
15+
| Test case | Error type | Detection | Block action |
2116
| :--- | :--- | :--- | :--- |
22-
| **1-bit alteration** | File Byte Change | **[SUCCESS]** | `exit 1` (HALT) |
23-
| **Metadata drift** | Filename Rename | **[SUCCESS]** | `exit 1` (HALT) |
24-
| **Hidden insertion** | New .md file | **[SUCCESS]** | `exit 1` (HALT) |
25-
| **Permutation** | File reordering | **[SUCCESS]** | `exit 1` (HALT) |
17+
| 1-bit alteration | File byte change | **detected** | `exit 1` (fail-closed halt) |
18+
| Metadata drift | Filename rename | **detected** | `exit 1` (fail-closed halt) |
19+
| Hidden insertion | New `.md` file | **detected** | `exit 1` (fail-closed halt) |
20+
| Permutation | File reordering | **detected** | `exit 1` (fail-closed halt) |
2621

27-
## 🧠 Cognitive Mapping (Omega Exclusive)
28-
Tests on RAG Context Ingestion.
22+
Reproduce any row: edit a tracked file, run `chronolith check`, and check `$?`.
23+
A clean tree exits `0`; any drift exits `1` unless you opt out with
24+
`CHRONOLITH_MODE=permissive`.
2925

30-
| Operation | Scale | Ingestion Time | Retrieval Accuracy |
31-
| :--- | :--- | :--- | :--- |
32-
| **Graph Vector Index**| 1000 nodes | 1.8s | 99.4% |
33-
| **Drift Prediction** | 5 sessions | 0.9s | Approved |
26+
## 🏁 Scan latency
27+
28+
**Current (3.2.2), end-to-end CLI wall time.** Measured on a 21-file markdown
29+
corpus, averaged over 3 warm runs on the maintainer's machine:
30+
31+
| Operation | Corpus | Wall time (avg) |
32+
| :--- | :--- | ---: |
33+
| `chronolith check` (full) | 21 files | ~665 ms |
34+
35+
This is *end-to-end* time and is dominated by Python interpreter start-up and Rich
36+
rendering, not by the Merkle scan itself. It is not comparable to the pure
37+
in-process scan micro-benchmarks below.
38+
39+
**Historical (v2.1.0 Nexus), pure in-process scan time.** Kept for reference; not
40+
re-measured on 3.2.2. Treat as indicative of the scan algorithm's order of
41+
magnitude, not as a current guarantee:
42+
43+
| Profile | Engine | File count | Latency (avg) |
44+
| :--- | :--- | :--- | ---: |
45+
| Initial scan | Lite | 500+ | 145 ms |
46+
| Incremental update | Lite | 1 changed | 12 ms |
47+
| DNA audit | Pro | 500+ | 410 ms |
48+
| Deep parity check | Pro | 500+ | 850 ms |
49+
50+
## 🧠 Cognitive mapping (Omega) — not independently reproduced
51+
52+
The earlier "1.8 s ingestion / 99.4 % retrieval accuracy / drift prediction
53+
approved" figures are **not reproduced in this pass** and are not backed by a
54+
committed measurement harness. They are omitted as headline results until there is
55+
a script anyone can run to regenerate them, the same bar the Merkle table meets.
3456

3557
---
36-
*Results verified via GitHub Actions (Industrial Guardian).*
58+
*The integrity table is exercised by the test suite in CI. The latency figures are
59+
local measurements, not CI-gated — CI runs the tests, it does not benchmark.*

0 commit comments

Comments
 (0)