Research and potentially replace stateful `lightwalletd` tests with regtest

[oxarbitrage]

Currently, Zebra’s CI runs several integration tests related to lightwalletd. Many of these tests require nodes that are close to the chain tip, which makes them unreliable, slow, and difficult to maintain over time.

Using regtest mode could significantly reduce sync times and improve reliability, while also lowering ongoing maintenance costs.

Goal

Explore replacing the current stateful lightwalletd tests with tests that run in regtest mode.

Specifically, we want to deprecate and remove:

• zebrad/tests/common/lightwalletd.rs
• The other lightwalletd test files
• The relevant sections in acceptance.rs

Options

• Integrate lightwalletd into the Python QA framework. Similar to what we already did with zallet.
• Replace with Rust integration tests in regtest mode.

[oxarbitrage]

Research done — conclusion: adapt the wallet-interface coverage to regtest against Zebra's native gRPC server (#10953), then delete the stateful GCP jobs. Of the two options in the issue description, this lands on "Rust integration tests in regtest mode", with one twist: most tests shouldn't target the external lightwalletd binary at all.

Why now

All four cached-state GCP jobs (rpc-submit-block, lwd-rpc-send-tx, lwd-grpc-wallet, lwd-sync-update) timed out together on the 2026-07-31 main run (https://github.com/ZcashFoundation/zebra/actions/runs/30641561945). The flakiness is structural: each test first syncs a cached mainnet state to the live tip over real peers before testing anything, so any near-tip sync stall fails the whole suite. Since NU6.3 activation the lightwalletd jobs also fail deterministically — the pinned Go binary can't parse the new transaction format (see comment below).

Key insight

These tests conflate two concerns:

1. The wallet-facing gRPC interface returns correct data — the valuable coverage.
2. The external Go lightwalletd daemon remains compatible with Zebra — cross-project compatibility.

PR #10953 implements all 18 CompactTxStreamer methods natively in zebra-rpc, and was manually verified on Regtest by mining blocks via generate and exercising 12 methods with a tonic client. That manual flow is exactly the automated test we need: concern (1) becomes testable on Regtest with no lightwalletd binary, no cached LWD state, and no mainnet. Concern (2) is better served by scheduled/downstream compatibility testing (e.g. zcash/integration-tests) than by required PR gates.

Plan

Track A — mining RPCs. Independent of #10953; can start immediately.

1. Delete rpc-submit-block. Regtest coverage is equal or stronger: common/regtest.rs::submit_blocks_test (200-block getblocktemplate→submitblock loop with typed Accepted assertions), the rejected_block_* two-process tests, and the NU6 funding-stream tests' typed reject assertions for invalid coinbase values — the stateful test never asserted rejections at all.
2. Port rpc-get-block-template to regtest, then delete the stateful job. Its unique coverage is the dynamic long-poll behavior (submit_old: false when a template is invalidated) — no other e2e test covers this; zebra-rpc has only static snapshots. The regtest version triggers invalidation deterministically: start a long-poll request → submit a block from a second client → assert the long poll returns promptly with submit_old: false and a new template that validates in proposal mode. A few deterministic iterations replace the stateful test's 1000 samples of random mainnet events. Everything else it does is already covered on regtest (nu6_3_block_template_proposal exercises proposal mode).
3. Remove the submit block and get block template GCP jobs once 1–2 land.

Track B — lightwalletd wallet interface. Depends on #10953 merging (tracked by #11158).

4. Add a regtest integration test for the native CompactTxStreamer (replaces lwd-grpc-wallet): mine a chain with Sapling/Orchard coinbase outputs (following regtest_coinbase), then assert GetLatestBlock, GetBlock/GetBlockRange, GetTreeState, GetTaddressBalance/GetTaddressTxids/GetAddressUtxos (expected values computed from the self-mined chain), and GetLightdInfo. Accepted loss: GetSubtreeRoots needs 2^16 notes per subtree — not reproducible on any in-test chain.
5. Delete lwd-sync-update without direct replacement: it tests the Go daemon's ingestion loop converging to Zebra's tip; the native server reads state in-process, so there is no equivalent to test. If a compat canary is wanted during the transition, a scheduled non-required job can keep it.
6. Adapt lwd-rpc-send-tx to the native server: SendTransaction + GetMempoolTx/GetMempoolStream plumbing can run on regtest with transparent txs now; the meaningful shielded-mempool assertion needs funded shielded transactions, which no in-tree wallet can produce — reuse the zcashd sidecar pattern from zebrad/tests/common/zcashd_compat/tx_flow.rs (110 funding blocks + z_sendmany, already running in the weekly zcashd-compat workflow) for the shielded case.
7. Remove the lightwalletd GCP jobs and the cached-LWD-state disk maintenance once 4–6 land.

Sequencing

The tracks are independent. Track A needs no prerequisites and can start now. Track B's critical path is the #10953 review; its urgency increased since the lightwalletd jobs now fail deterministically on every scheduled run (11h of GCP time each) until either the pinned binary is updated or the jobs are removed.

---

Edited 2026-08-01: restructured into two independent tracks and added rpc-get-block-template (port, not delete — long-poll coverage gap) after analyzing the 07-31 scheduled-run failures; details in the comments below. Original plan had four jobs in a single #10953-gated sequence.

[oxarbitrage]

Two more data points from the 2026-07-31 scheduled stateful run, both reinforcing this migration (neither is a node bug — zebrad was at tip and healthy in both jobs):

• lwd-sync-full fails deterministically since NU6.3 activation: the CI lightwalletd binary (2024-04-24 build, 2fb5ccd7) can't parse the NU6.3 transaction format, so it retries the activation block 3,428,143 (version group ID 3632576152 must be 0x26A7270A) forever until the 11h harness timeout. It will fail every run until the pinned image is updated — exactly the recurring "keep the external Go binary's protocol knowledge current" cost that testing against Zebra's native gRPC server (feat(rpc)!: add lightwalletd-compatible gRPC server implementing CompactTxStreamer #10953 / Serve light clients directly: implement the lightwalletd CompactTxStreamer gRPC interface in Zebra #11158) eliminates.

• stateful::rpc::rpc_get_block_template timed out on a startup ordering race: check_sync_logs_until expects "finished state checkpoint validation" then "activating mempool", but with a hot near-tip cached state the mempool activates (~1s) before checkpoint validation finishes (~7s), so the ordered matcher waits 1h for a line that already went by. Rather than patching the race, I'm adding this test to the migration list (plan updated above): the internal miner already drives getblocktemplate on regtest, and submit_blocks_test exercises a 200-block getblocktemplate→submitblock loop, so native regtest coverage largely exists.