devops: reshape GCP integration CI into weekly maintenance plus recovery

[alchemydc]

Priority: P1 design issue
Related: #9941, #10737

Motivation

The GCP tier is producing little trustworthy signal: among 33 sampled push/scheduled runs, only one succeeded, 22 failed, and 10 were cancelled. The initial analysis (gist) correctly rejected Zakura's deletion of this coverage but mischaracterized the current trigger shape.

Full Zebra Mainnet/Testnet syncs already run weekly, with additional runs when the corresponding tip disk is missing. Full lightwalletd sync likewise runs weekly or when its disk is missing. The expensive per-main work is mainly:

• building and publishing the CI image;
• syncing cached tip/checkpoint state forward;
• generating checkpoints;
• running cached-state Zebra RPC and lightwalletd checks.

These concerns should not share one health statistic or one all-or-nothing workflow graph. Zebra needs weekly maintenance and immediate disk-format recovery, while deterministic functional tests should run in the normal PR lane wherever equivalent coverage exists.

Proposed change

1. Classify the current jobs

Create a checked-in or issue-attached inventory with one row per top-level GCP job:

• maintenance/artifact: creates or updates Zebra/lightwalletd disks or checkpoint artifacts;
• public-network smoke: validates behavior that inherently needs current public-chain state or peers;
• functional: validates an RPC or component behavior that can potentially run on local/regtest state;
• orchestration: build, disk lookup, aggregation, cleanup, or reporting.

For each row, record trigger, prerequisites, typical duration/cost, whether it creates an artifact, current pass/fail/skip behavior, equivalent PR coverage, and proposed destination.

2. Make main pushes a lightweight recovery preflight

On relevant main pushes:

1. Check the state database format and availability of compatible checkpoint, tip, and lightwalletd images before building the CI image.
2. If all required disks exist, finish without running the maintenance fleet.
3. If a required disk is missing or incompatible, build once and launch the minimum recovery chain needed to recreate it.
4. Keep workflow_dispatch for explicit maintainer recovery and diagnosis.

Do not make the expensive Docker build an unconditional prerequisite for discovering that no recovery work is needed.

3. Run maintenance weekly

On the Friday schedule, run the agreed maintenance set, including full syncs, update checks, checkpoint generation, and artifact publication. Preserve the dependency order required to generate missing disks. Define one owner and one recovery-aware tracking issue per independently actionable failure class rather than one permanently red umbrella issue.

Reporting must open/update on failure and comment/close on recovery. Expected skipped jobs must not fail aggregation or count against the green rate.

4. Audit functional coverage before moving it

Audit rpc-get-block, rpc-get-block-template, rpc-submit-block, and the lightwalletd jobs against existing integration/regtest tests. In particular:

• existing regtest block-template/submission coverage may already overlap the cached-state GCP jobs;
• the cached-state getblock test validates a historical Mainnet block and cannot be declared equivalent to a generic regtest getblock check without a coverage argument;
• Research and potentially replace stateful lightwalletd tests with regtest #9941 covers research into replacing stateful lightwalletd/public-network checkpoint coverage, not a pre-approved migration of all three RPC jobs.

Retire or demote a GCP functional job only after the inventory names the deterministic replacement test and explains any semantic difference. If no equivalent exists, keep the GCP job in the weekly lane or file a separate implementation issue for the replacement.

5. Remove expected PR noise after coverage migration

Once no blocking behavior depends on the run-stateful-tests label, remove the broad pull_request trigger from this workflow and use workflow_dispatch for targeted PR/branch diagnosis. Until then, report PR skips separately and do not include them in reliability rates.

Acceptance criteria

• Every GCP job is classified with trigger, artifact, cost, coverage, and destination.
• A normal relevant main push checks disk compatibility without building an image or launching VMs when no recovery is needed.
• A simulated missing/incompatible disk launches the minimum recovery chain automatically.
• Weekly maintenance preserves disk and checkpoint production required by release operations.
• No RPC/lightwalletd job is removed without a named deterministic replacement or explicit team acceptance of the coverage loss.
• Failure issues open/update and close on recovery.
• Reliability dashboards exclude expected PR skips and separate push, schedule, manual, and recovery events.
• Four consecutive eligible weekly/recovery runs complete successfully before the redesign is considered stable.

---

Part of the Improve CI program (#11196)

Drafted with Claude Code from the comparative analysis and the reviewed plan; independently reviewed and corrected with OpenAI Codex. Maintainers remain responsible for validating measurements and approving coverage tradeoffs.