lme

A naive, auditable, extensible memory system for LongMemEval-V2 (xiaowu0162/LongMemEval-V2): SQLite + brute-force cosine retrieval over chunked web/enterprise-agent trajectories, built around explicit Strategy/Registry/Factory/Decorator/Template-Method patterns so every stage is swappable from config.toml alone. See DECISIONS.md for every judgment call made building it.

Quickstart

# 1. Install (Python 3.11, pinned via .python-version)
uv sync

# 2. Point at an OpenAI-compatible reader backend
export READER_BASE_URL=http://localhost:11434/v1
export READER_MODEL=your-model-name

# 3. Dataset is already fetched to data/longmemeval-v2/ (screenshots
#    excluded by design -- see DECISIONS.md "disk gate"). To refetch:
#    uv run python scripts/fetch_dataset.py --data-root data/longmemeval-v2

# 4. Ingest the pinned 20-question smoke sample (small tier)
uv run python -m lme.ingest --tier small --questions smoke

# 5. Ask one question directly, with an inline trace
uv run python -m lme.query --id <question_id> --verbose --yes-i-will-pay

# 6. Run the full smoke set (writes out/<tag>/hypotheses.jsonl)
uv run python -m lme.run --smoke --tag smoke --yes-i-will-pay

# 7. Watch it live while it runs (separate terminal)
uv run python -m lme.live --tag smoke

# 8. Grade it against the vendored, unmodified LongMemEval-V2 eval dispatch
uv run python -m lme.score --tag smoke

# 9. Compare two tagged runs (e.g. after changing config.toml)
uv run python -m lme.compare out/smoke out/smoke-v2

# 10. Reconstruct the full answer -> context -> hits -> chunks -> steps
#     lineage for one question, straight from the audit log + SQLite store
uv run python -m lme.audit trace --question <question_id>
uv run python -m lme.audit report --run <run_id>

The full 451-question benchmark ("--all")

uv run python -m lme.ingest --tier small --questions all
uv run python -m lme.run --all --tag all --yes-i-will-pay   # prints a token estimate first
uv run python -m lme.score --tag all

run --all is resumable: rerunning it after a crash or Ctrl-C skips whatever already has a successful row in out/all/hypotheses.jsonl.

Architecture

• Typed artifacts (lme/artifacts.py): frozen dataclasses -- Trajectory, Step, Chunk, EmbeddingRecord, Hit, ContextBundle, Answer -- each with a deterministic content-hash id and parent_ids, forming a lineage DAG from an answer back to its source steps.
• Stage interfaces (lme/interfaces.py): one Protocol per concern -- Chunker, Embedder, IndexBuilder, Retriever, ContextAssembler, Reader.
• Strategy + Registry + Factory (lme/registry.py, lme/chunkers|embedders|indexes|retrievers|assemblers|readers/): every strategy self-registers under a string name; PipelineFactory builds the whole pipeline purely from config.toml. Swap an algorithm = one config line. Add one = one new module + one @register(...) call.
• Decorator + Template Method (lme/stage.py): AuditedStage wraps any strategy instance with a uniform validate -> execute -> emit-audit-event lifecycle, so the audit layer can't be forgotten and strategies stay pure.
• Audit/provenance (lme/audit.py, lme/audit_cli.py): append-only JSONL event log per run (app/audit/<run_id>/events.jsonl), a content-addressed blob store for prompts/responses (app/audit/blobs/<hash>), and a run header (config snapshot, git SHA, model revisions). python -m lme.audit trace is the point of the whole system -- see Quickstart.
• Vector-agnostic retrieval (lme/store.py): Retriever.retrieve(question, store) -> list[Hit] where store exposes the chunk table, the embedding matrix, AND the raw trajectory tree. retrievers/pageindex_stub.py is a registered-but-NotImplemented PageIndex strategy documenting the reasoning-based (no-vectors) extension path.
• Confabulation detector (lme/confab.py): a primitive, always-on, gold-answer-blind heuristic flagging answers that are either confident despite empty context or lexically ungrounded in the retrieved context. Not a substitute for lme.score; a fast tripwire in the audit trail.

Layout

• lme/ -- the package (see Architecture above).
• config.toml -- every tunable (chunk size/overlap, embedder params, retrieval top_k, assembler token budget, reader temperature/max_tokens, and which strategy each stage uses). Edit and re-run; hot-loaded.
• data/longmemeval-v2/ -- the dataset (questions/trajectories/haystacks; trajectory_screenshots/ deliberately excluded -- see DECISIONS.md).
• app/ -- runtime artifacts (SQLite store, per-haystack .npy embeddings, smoke sample, audit log, run outputs). Override the root with $LME_APP_DIR.
• vendor/LongMemEval-V2/ -- the official benchmark repo, cloned verbatim, reused (not reimplemented) for dataset parsing/validation and grading (evaluation/qa_eval_metrics.py's eval_from_spec).
• baselines/naive-v0.json -- pinned smoke-run numbers (once the live smoke test has run; see DECISIONS.md).
• tests/ -- uv run pytest.

Known fidelity gaps (see DECISIONS.md for the reasoning)

• Screenshots are recorded (path + trajectory/step lineage) but never fetched or rendered -- this build is text-first by design.
• The two LLM-judged eval functions (llm_abstention_checker, llm_gotchas_checker) reuse READER_MODEL/READER_BASE_URL as the judge rather than the benchmark's own default (gpt-5.2 via OPENAI_API_KEY), since this build has no separate judge credential.
• Token counting in the context assembler is a word-count approximation, not a real tokenizer.
• This build's own lme.run/lme.score produce and grade a hypotheses.jsonl-style output, which is NOT how the real V2 harness works (it drives a memory_modules.Memory subclass in-process and never produces a hypotheses file at all) -- see DECISIONS.md "harness integration contract".
• GraphBuilder's structured entity extractor is a heuristic (URL/action field parsing + regex), not an oracle -- the prior sandbox's PPR result used an oracle extractor and explicitly flags entity-linking quality as the load-bearing constraint in reality. See DECISIONS.md "Graph memory extension".

Experimentation (SVD routing + hybrid delta chunking + entity graph PPR)

Three swappable strategies beyond the naive defaults, each addable from config.toml alone (see DECISIONS.md "SVD/delta extension" and "Graph memory extension" for the architecture and the deviations from the handoff's literal config-section spelling):

• SVDComponentIndex (index = "svd_component", retriever = "svd_component") -- TruncatedSVD + KMeans sub-indexes with query routing (the IVF/nprobe idea, pure-numpy). Fit at index-build time on the haystack's normalized embedding matrix, persisted beside the .npy; query time projects the query through the fitted SVD, ranks clusters by centroid distance, and exhaustively cosine-scores only the routed clusters' members. Tunables:

  • route_top ([retrieval], default 2) -- how many of the top-ranked clusters to scan. route_top >= n_clusters is bit-for-bit equivalent to FlatCosineRetriever (asserted in tests/test_svd_component.py); lower values trade recall for a smaller scan fraction (watch the scan_fraction in the retrieve audit event).
  • n_components, n_clusters ([index], defaults 16, 8) -- SVD dimensionality and cluster count for the fit.
  • min_cluster_scan ([retrieval], default 64) -- floor so a tiny cluster can't starve top_k (the router keeps adding next-best clusters until at least this many members are scanned). The interpretability ledger (chosen clusters + centroid distances, scan fraction, per-cluster build-time purity) is in the index/retrieve audit events' extra field.

• HybridDeltaChunker (chunker = "hybrid_delta") -- wraps StepChunker by composition and emits every step chunk PLUS kind="delta" chunks describing the field-level changes between consecutive observations. Delta chunks are first-class: content-hash ids, parent_ids at BOTH source observations (before and after), a kind="delta" tag in the store and every audit event. No-change steps emit no delta. Tunable:

  • mode ([chunking], default "structured") -- "structured" is a deterministic, free field-level diff (rendered as session {trajectory_id} step {after_state_index}: {field} changed from {old} to {new}); "llm" makes one extraction call per changed step (gated behind LME_DELTA_LLM_ACK=1), producing the same rendered form via the prompt at prompts/delta_extraction.txt. Pair with assembler = "delta_aware" so a delta plus both its parents doesn't burn the token budget: the assembler keeps the delta and at most one parent (the more-relevant), never all three.

• GraphBuilder + GraphPPRRetriever (index = "graph_builder", retriever = "graph_ppr") -- an entity co-occurrence graph built at ingest time, queried by Personalized PageRank. GraphBuilder wraps a base_index (default flat_cosine; set to svd_component in config.allthree.toml) by composition, exactly like HybridDeltaChunker wraps StepChunker -- it still produces the embedding matrix (and SVD fit, if wrapped) every retriever needs, then additionally persists the graph. Tunables:

  • extraction ([index], default "structured") -- "structured" pulls entities at zero model cost from URL host + last path segment and action-field quoted/verb-object phrases; "llm" adds one extraction call per chunk over the thought/observation free text (gated behind LME_GRAPH_LLM_ACK=1, prompt at prompts/entity_extraction.txt). Alias normalization (casefold, strip a/an/the, singular/plural fold, plus the explicit map in graph_aliases.json, persisted per-haystack) applies regardless of extraction mode.
  • base_index ([index], default "flat_cosine") -- which registered IndexBuilder GraphBuilder composes over for the dense artifacts.
  • fallback_retriever ([retrieval], default "flat_cosine") -- used whenever zero query entities match the graph (logged as fallback_fired: true in the retrieve audit event).
  • damping, iterations ([retrieval], defaults 0.85, 20) -- PPR power-iteration parameters; seed_bias_penalty ([retrieval], default 0.5) down-weights a chunk whose only PPR+IEF mass comes from a seed entity itself, so a genuine bridge chunk (positive mass from a non-seed entity PPR actually propagated to) outranks a seed-adjacent distractor. python -m lme.audit graph [--haystack <id>] [--run <run_id>] reports node/edge counts, alias-merge rate, orphaned-entity count, degree distribution, and -- per question -- seeds matched, top PPR nodes, and whether the fallback fired.

Structural memory (dominator analysis over interface-tree representations)

The lme/structural/ subsystem converts recorded interface state -- web DOM, a11y, and COM/UIA trees (the tree type is a MANIFEST label, routed, never inferred from content) -- into faithful graph representations and runs dominator analysis over them, so unavoidability facts become derived, evidence-qualified, retrievable memory:

• Representations (registry group graph_view, config [chunking.structural] views):
  • state_transition -- nodes are interface states (spectral fingerprint clusters of snapshots), edges are observed transitions; a dominator is a mandatory waypoint.
  • containment -- nodes are UI elements unified under the platform's desktop root, edges are observed parent->child containment; a dominator is a gateway container (a modal dialog is a computed dominator over the subtree it blocks, not a heuristic label).
  • causal -- registered design stub (constructible from config, raises NotImplementedError): events + causal precedence, dominator = necessary precondition.
• Analyzers (registry group structural_analyzer, all operating on ANY view through the RootedGraphView protocol -- the same analyzer instance runs over every representation): dominator (waypoint/gateway chains + claim checks), postdominator (inevitability chains + the safety audit), loop (back edges -> natural loops -> nesting forest, classified inner-first with inner loops collapsed before outer classification).
• Derived chunks: every finding is rendered as a first-class kind="derived" chunk, embedded and indexed alongside step/delta chunks, with lineage parent_ids -> graph artifact -> source snapshot (step) ids.

Epistemic rules (enforced -- the renderer raises without evidence): observed graphs are incomplete and a missing edge manufactures a FALSE dominator, so domination is always provisional ("no bypass observed across N trajectories / M edge observations"), while non-domination is proven ("bypass observed in trajectory {id}"). Edge construction is conservative: observed transitions/containments only, each edge carrying EdgeEvidence (snapshot/event ids + observation count).

Config knobs (config.structural.toml):

• [chunking.structural.fingerprint] threshold -- nearest-neighbor distance threshold for snapshot state identity (NetLSD-style Laplacian heat traces, size-normalized, 24 log-spaced time points). Calibrated on the synthetic fixtures (intra-family ~0.011 vs inter-template ~0.026 -> 0.018); it is a TUNABLE, expect to retune on real trees -- watch cluster_count / singleton_rate / merge_distances in lme.audit structural. max_eig_nodes caps the eigendecomposition (larger trees are truncated, counted in the identity report).
• [chunking.structural.dominator] targets / claims -- label patterns whose waypoint chains become chunks; claims are explicit unavoidability hypotheses (held -> provisional chunk, falsified -> proven-bypass chunk).
• [chunking.structural.postdominator] outcomes / irreversible_actions / guard_patterns -- the safety audit: for each (action, outcome) pair, reports guarded (some confirm/dialog node is unavoidable) or unguarded with the concrete bypass edge and witness trajectory.
• [chunking.structural.loop] error_patterns -- a loop is pathological iff one of its OWN nodes matches (inner loops collapsed first, so an inner retry loop's error state never mislabels a healthy outer cycle).
• [assembler] derived_chunk_cap -- max derived chunks per assembled context (guards against structural facts crowding out concrete step evidence); inner picks the wrapped assembler.

Inspect everything (fidelity report, identity-cluster stats, dominator/post-dominator findings with evidence clauses, loop forest):

uv run python -m lme.audit structural              # from the ingested store
uv run python -m lme.audit structural --fixtures   # synthetic demo, no dataset needed

Running the referendum

Seven full config files select a strategy combination each, switched via LME_CONFIG_PATH (no config.toml rewrite between runs). baseline / delta / svd / graph / allthree are the five configs the graph handoff's referendum requires; svddelta is a pre-existing bonus (svd routing + hybrid delta, no graph) kept from the prior extension; structural is the structural-memory extension's tag (dominator-analysis derived chunks over baseline dense retrieval).

tag	chunker	index	retriever	assembler	config file
baseline	step	flat_cosine	flat_cosine	chronological	config.baseline.toml
delta	hybrid_delta	flat_cosine	flat_cosine	delta_aware	config.delta.toml
svd	step	svd_component	svd_component	chronological	config.svd.toml
svddelta	hybrid_delta	svd_component	svd_component	delta_aware	config.svddelta.toml
graph	step	graph_builder (base=flat_cosine)	graph_ppr (fallback=flat_cosine)	chronological	config.graph.toml
allthree	hybrid_delta	graph_builder (base=svd_component)	graph_ppr (fallback=svd_component)	delta_aware	config.allthree.toml
structural	structural (inner=step)	flat_cosine	flat_cosine	derived_capped	config.structural.toml

bash scripts/referendum.sh                     # ingest + score only, skips the paid run unless acked
LME_COST_GATE_ACK=1 bash scripts/referendum.sh  # full run including paid reader calls

referendum.sh delegates to scripts/run_referendum.sh, which loops ingest -> run -> score over all seven tags (printing the structural audit after ingest), then scripts/strategy_report.py renders out/strategy-report.md. The structural hypothesis to check there: derived chunks move the procedure (workflow-knowledge) and errors-gotchas columns; the regression risk is derived-chunk noise crowding the context budget -- derived_chunk_cap is the lever. Equivalent by hand for one config:

for cfg in baseline delta svd svddelta graph allthree structural; do
  LME_CONFIG_PATH=config.$cfg.toml uv run python -m lme.ingest --tier small --questions smoke
  LME_CONFIG_PATH=config.$cfg.toml uv run python -m lme.run --smoke --tag $cfg --yes-i-will-pay
  uv run python -m lme.score --tag $cfg
done
uv run python -m lme.compare out/baseline out/structural

The embedding_cache (keyed by chunk content hash, not haystack) makes re-ingestion across configs cheap: the delta configs re-embed only the new delta chunks, step chunks are cache hits. out/strategy-report.md is the referendum's verdict (per-ability accuracy, latency p50/p95, ingest cost); python -m lme.audit trace --question <dynamic-state qid> shows a delta chunk with dual-parent lineage in a dynamic-state answer chain; python -m lme.audit graph shows the entity graph's build stats and per-question PPR ledger.

Not run in this environment: no data/longmemeval-v2/ and no READER_BASE_URL/READER_MODEL here, so neither ingest nor the paid reader step executed -- see DECISIONS.md "Referendum not run" for exactly what's required to run it for real. All six configs are registered and exercised end-to-end against synthetic fixtures by the test suite instead.