bug(sync): near-tip AwaitUtxo timeout is treated as an invalid block, causing a sync restart loop

[oxarbitrage]

Description

When a block's transparent input spends an output from a very recent block that is not yet committed to the state, the verifier's AwaitUtxo lookup can time out. Since #10810 the timeout is correctly classified as TransactionError::TransparentInputNotFound (in the AwaitUtxo arm of zebra-consensus/src/transaction.rs) instead of InternalDowncastError — but the syncer then treats that transient condition as a consensus-invalid block.

should_restart_sync() (zebrad/src/components/sync.rs) has no arm for a verify-side transient NotFound: the error falls into the catch-all and triggers a full sync restart. Under a sustained near-tip stall, each restart re-downloads the same range, hits the same not-yet-committed parent, times out again, and restarts again — a restart loop that keeps the node pinned just below the tip.

The catch-all's own diagnostic confirms this error class is reaching a place it was never meant to: because the debug string contains NotFound, it logs "a BlockDownloadVerifyError that should have been filtered out was detected, which possibly indicates a programming error" (see the existing TODOs referencing #2908 and #2909 at that site).

Observed

All 4 cached-state GCP jobs (rpc-submit-block, lwd-rpc-send-tx, lwd-grpc-wallet, lwd-sync-update) timed out together on the 2026-07-31 main run:

https://github.com/ZcashFoundation/zebra/actions/runs/30641561945

Log signature: AwaitUtxo timeout → TransparentInputNotFound → "error downloading and verifying block" WARN → sync restart, repeating.

Verified unfixed on main at 392e4b5 (2026-08-01): no TransparentInputNotFound arm in should_restart_sync(), and no open PR touches the classification.

Possible fixes

• Add a should_restart_sync() arm treating Invalid { TransparentInputNotFound } as transient: drop the block and continue (the syncer will re-request it), restarting only after N consecutive occurrences — mirroring the existing download-side NotFound arm. Note TransparentInputNotFound is ambiguous (transient timeout vs. genuinely spending a nonexistent output), so it should not be treated as unconditionally transient.
• Alternatively, distinguish "timed out waiting for UTXO" from "input genuinely not found" in the error type, so the syncer can be precise (the typed-error work Improve VerifyBlockError::Commit typing, so we don't accidentally break syncer error handling #2908 asks for; related to refactor(network): add typed NotFound check on SharedPeerError to replace string-grep pattern #10691).

Related

• Investigate sync restarts at chain tip #10192 — open investigation of sync restarts at the chain tip; this issue identifies one concrete cause (the duplicate-block cause suspected there was already fixed by Commit-stage duplicate block errors are misclassified as state errors, causing spurious syncer restarts #10915).
• fix(consensus): handle Elapsed timeout in AwaitUtxo to prevent near-tip sync stall #10810 — fixed the error classification at the verifier (merged 2026-06-25); this issue is about the syncer's response to it.
• Improve VerifyBlockError::Commit typing, so we don't accidentally break syncer error handling #2908 — typed verifier errors; the structural fix. The catch-all's TODO already points there.
• Research and potentially replace stateful lightwalletd tests with regtest #9941 — migrating the affected CI jobs to regtest removes the CI exposure, but this behavior affects any production node near tip and needs fixing regardless.
• fix(network): increase the near-tip tolerance for stall detection #11122, fix(sync): download singleton FindBlocks responses #11165 — adjacent near-tip sync fixes (stall tolerance, singleton FindBlocks); neither addresses the restart classification.

[jiehuo100net]

I looked into this and found that #11132 is the same defect seen from the other side — both
reports land on the same match in should_restart_sync().

TransparentInputNotFound reaches the catch-all arm, which does two things to it:

• line 1378 returns true, restarting the sync — the loop described here;
• line 1369 matches the substring "NotFound" against the error's debug output and logs it as a
  suspected downcast bug — which is exactly the false positive conradoplg reported in bug: sync error checking mismatch #11132,
  with Invalid { error: Block { source: Transaction(TransparentInputNotFound) } } at Mainnet
  heights 3,427,629 and 3,427,920.

So a single typed arm fixes both: matched by type, the error no longer reaches the catch-all, so
neither the restart nor the spurious log happens. #11132 asks for precisely that ("this substring
match is the thing that needs replacing with a type check").

The error path is fully typed end to end, so no downcast or string matching is needed:

BlockDownloadVerifyError::Invalid { error: RouterError }
  -> RouterError::Block { source: VerifyBlockError }
    -> VerifyBlockError::Transaction(TransactionError::TransparentInputNotFound)

VerifyBlockError::Transaction has a single construction site, zebra-consensus/src/block.rs:321
in SemanticBlockVerifier::call. The checkpoint verifier cannot produce it — RouterError::Checkpoint
carries a VerifyCheckpointError — so that side is always false.

One design question before I open a PR. I have a working implementation of the typed arm, with
regression tests covering both this issue and the spurious log from #11132 — but there is one point
in your description I would rather not decide on your behalf.

You note that TransparentInputNotFound is ambiguous and should not be treated as unconditionally
transient, and suggest restarting after N consecutive occurrences. The existing download-side
NotFound arm just above is unconditional, and its TODO asks the same question ("restart after a
certain number of NotFound errors?"), so mirroring it exactly means staying unconditional — which
is what I currently have.

Adding a consecutive-occurrence counter is a real change in shape rather than a new arm:
should_restart_sync() and handle_response() are both associated functions with no self, so
the counter would have to live on ChainSync and both would become methods — five call sites in
total, three for handle_response and two for should_restart_sync.

Which would you prefer — the unconditional typed arm that mirrors the existing one, or the
counter? Happy to do either, or to leave it with you if you'd rather take it yourselves.

(I also looked at distinguishing the timeout from a genuine miss in the error type, as your second
option suggests. TransactionError is not #[non_exhaustive], so adding a variant is a semver
break for zebra-consensus — seems better left to the typed-error work in #2908.)