fix(agent): preserve tool_call structure across turns + Markdown->tool_call salvage parser — stop the apr-code text-loop that defeats tool-calling (CCPA m296)#2245
Open
noahgift wants to merge 1 commit into
Conversation
…l_call salvage parser — stop the apr-code text-loop that defeats tool-calling (CCPA m296)
GROUNDED BUG (CCPA m296 distill feasibility spike): the apr-code agentic loop
had a HARNESS bug independent of the model. Even a format-correct model (one
that emits a valid <tool_call>) reverted to 0/N tool_calls across a multi-turn
run because a prior assistant TOOL_CALL turn could be retained / re-rendered as
raw Markdown prose, re-priming prose mode — a self-reinforcing text loop. Until
fixed, ANY fine-tune result is uninterpretable.
WHAT WAS WRONG
- runtime.rs (EndTurn branch) pushed `response.text` verbatim into multi-turn
history as `Message::Assistant(...)`. When the driver's parser failed to
recognize a model's tool-call shape (anything outside the exact <tool_call> /
```json envelope — a bare {"name","input"} object or a ```tool_call/```rust
fence), that turn was scored as inert prose AND its raw tool-call Markdown was
re-injected into the next turn's prompt, eroding tool-calling.
THE FIX (two correctness surfaces)
1. SALVAGE PARSER (realizar.rs `salvage_tool_calls`): when the envelope parser
finds nothing, conservatively recover a tool call from (a) a generically-
fenced block (any language tag or none) whose body is tool-call JSON, or (b)
a bare top-level {"name","input"} JSON object. Only objects with a string
`name` AND an `input` field are salvaged — plain JSON / prose are never
mistaken for tool calls. Salvage events are logged (id `salvage-N`). Applies
to BOTH the embedded RealizarDriver and the apr-serve HTTP path (shared
parser). This recovers the "model almost emitted a tool_call" near-misses.
2. STRUCTURED RETENTION (runtime.rs `retain_assistant_text`): an assistant
turn's text is stripped of lingering <tool_call>/<tool_result> markup before
it enters history, so a tool-using turn is NEVER re-rendered as capability-
breaking raw Markdown that re-primes prose mode. Genuine prose passes through
unchanged; the structured AssistantToolUse/ToolResult messages already carry
that turn's tool semantics (chat_template.rs renders them as the canonical
<tool_call> + <tool_result> envelope — no "### Continue:" prose nudge).
FALSIFIERS (mutation-verified, oracle = the structured/expected render)
- falsify_toolcall_retention_001: next-turn render of a prior tool-call turn
preserves <tool_call>+<tool_result> and contains no "### Continue:" nudge.
- falsify_toolcall_retention_002: history keeps structured tool messages, never
raw <tool_call> markup as Assistant prose.
- falsify_toolcall_retention_003: retain_assistant_text strips residue, keeps
prose.
- test_salvage_* (realizar): recover bare/fenced tool-call JSON; reject plain
JSON / name-without-input; envelope still takes precedence.
- Mutation verified: making retain_assistant_text identity turns 003 RED;
reverting salvage to envelope-only turns the 3 salvage-recovery tests RED.
CONTRACT: contracts/apr-code-toolcall-retention-v1.yaml
(OBLIG-APR-CODE-TOOLCALL-RETENTION) — kind: kernel, 5 single-line cargo-test
falsifier refs. `pv validate` + `pv lint contracts/` PASS (0 errors/0 warnings
on this file).
GREEN: cargo test -p aprender-orchestrate --lib (6514 pass; the lone failure is
agent::tool::mcp_client::test_discover_tools_via_echo — a pre-existing
subprocess/stdio flake unrelated to this change, 1 fail / 3 runs, different
module). clippy -p aprender-orchestrate --all-targets clean (exit 0). fmt clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Grounded bug (CCPA m296)
The apr-code agentic loop had a harness bug independent of the model. Even a format-correct model (one that emits a valid
<tool_call>) reverted to 0/N tool_calls across a multi-turn run because a prior assistant TOOL_CALL turn could be retained / re-rendered as raw Markdown prose, re-priming prose mode — a self-reinforcing text loop. This is a hard prerequisite: until fixed, any fine-tune result is uninterpretable.What was wrong
runtime.rs(theEndTurnbranch) pushedresponse.textverbatim into multi-turn history asMessage::Assistant(...). When the driver's parser failed to recognize a model's tool-call shape — anything outside the exact<tool_call>/```jsonenvelope (a bare{"name","input"}object, or a```tool_call/```rustfence) — that turn was scored as inert prose and its raw tool-call Markdown was re-injected into the next turn's prompt, eroding tool-calling.The fix — two correctness surfaces
Salvage parser (
realizar.rs::salvage_tool_calls): when the envelope parser finds nothing, conservatively recover a tool call from (a) a generically-fenced block (any language tag or none) whose body is tool-call JSON, or (b) a bare top-level{"name","input"}JSON object. Only objects with a stringnameAND aninputfield are salvaged — plain JSON / prose are never mistaken for tool calls. Salvage events are logged (salvage-Nids). Shared parser, so it applies to both the embeddedRealizarDriverand theapr serveHTTP path. This recovers the "model almost emitted a tool_call" near-misses.Structured retention (
runtime.rs::retain_assistant_text): an assistant turn's text is stripped of lingering<tool_call>/<tool_result>markup before it enters history, so a tool-using turn is never re-rendered as capability-breaking raw Markdown that re-primes prose mode. Genuine prose passes through unchanged; the structuredAssistantToolUse/ToolResultmessages already carry that turn's tool semantics (chat_template.rsrenders them as the canonical<tool_call>+<tool_result>envelope — and there is no### Continue:prose nudge after a tool turn).Falsifiers (mutation-verified — oracle = the structured/expected render)
falsify_toolcall_retention_001— next-turn render of a prior tool-call turn preserves<tool_call>+<tool_result>and contains no### Continue:nudge.falsify_toolcall_retention_002— history keeps the structured tool messages, never raw<tool_call>markup as Assistant prose.falsify_toolcall_retention_003—retain_assistant_textstrips residue, keeps prose.test_salvage_*(realizar) — recover bare/fenced tool-call JSON; reject plain JSON and name-without-input; envelope still takes precedence.Mutation results: making
retain_assistant_textan identity turns003RED; reverting salvage to envelope-only turns the 3 salvage-recovery tests RED. (Perfeedback_contracts_ratchet_not_radar.)Contract
contracts/apr-code-toolcall-retention-v1.yaml(OBLIG-APR-CODE-TOOLCALL-RETENTION) —kind: kernel, 5 single-linecargo testfalsifier refs.pv validate+pv lint contracts/PASS (0 errors / 0 warnings on this file).Green
cargo test -p aprender-orchestrate --lib: 6514 pass. The lone failure isagent::tool::mcp_client::test_discover_tools_via_echo— a pre-existing subprocess/stdio flake unrelated to this change (1 fail / 3 runs, different module; passes on pristine origin/main and 2/3 with this change).cargo clippy -p aprender-orchestrate --all-targets: clean (exit 0).cargo fmt: clean.🤖 Generated with Claude Code