Skip to content

feat(engine): add export_traffic to dump observations as savings-report JSONL#27

Closed
AZERDSQ131 wants to merge 1 commit into
KRLabsOrg:mainfrom
AZERDSQ131:fix/export-observed-traffic-jsonl
Closed

feat(engine): add export_traffic to dump observations as savings-report JSONL#27
AZERDSQ131 wants to merge 1 commit into
KRLabsOrg:mainfrom
AZERDSQ131:fix/export-observed-traffic-jsonl

Conversation

@AZERDSQ131

Copy link
Copy Markdown

Problem

The savings report (rulechef-savings) needs a traffic JSONL file, but there's no helper to produce one. Observation mode (start_observing / add_observation) already captures exactly this data, it just never lands on disk in the traffic format — users had to write their own dump loop.

Fix

Added RuleChef.export_traffic(path). It exports LLM-sourced observations from the buffer as:

  • {"text": ..., "llm_label": ...} for classification output
  • {"text": ..., "llm_entities": [...]} for NER/extraction output (entities or spans key)

Human-sourced examples (add_example, corrections) are skipped — they aren't traffic. Observations missing a text input or a recognized output shape are also skipped rather than raising.

Documented the observe -> export_traffic -> rulechef-savings loop in benchmarks/INSPECTING_RULES.md, including a note that rulechef-savings itself currently only scores the classification (llm_label) shape (NER traffic export is ready for when #19 lands).

Tests

  • Classification and NER export shapes.
  • Human examples are skipped.
  • Observations without text / unrecognized output are skipped (count stays 0, file stays empty).
  • Full round-trip: export_traffic -> rulechef-savings CLI -> assert the generated HTML report (mirrors tests/test_reports.py::test_savings_cli's invocation pattern).

pytest tests/ -q → 316 passed, 1 skipped. ruff check and mypy rulechef/engine.py clean.

Closes #20

…rt JSONL

Observation mode (start_observing / add_observation) already captures
exactly the data rulechef-savings needs, but nothing wrote it to disk
in the traffic format -- users had to write their own dump loop.

Add RuleChef.export_traffic(path): exports LLM-sourced observations
as {text, llm_label} for classification output or {text, llm_entities}
for NER/extraction output. Human-sourced examples and observations
missing a usable text/output shape are skipped.

Closes KRLabsOrg#20
@adaamko

adaamko commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Thanks for picking this up — unfortunately it crossed with #21, which was opened for the same issue (#20) a few days earlier and just merged. Same design, so closing this one. The spans output fallback in yours is a nice touch #21 doesn't have; if you hit a real case that produces spans instead of entities, a small follow-up PR for that is very welcome. Also relevant to your #28: main now has export_traffic, so the NER savings path can be tested end-to-end.

@adaamko adaamko closed this Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Export observed traffic as savings-report JSONL (observe -> rulechef-savings loop)

2 participants