Long integration tests don't survive zfnd-runners recycling, and there's no retry

[oxarbitrage]

A manually dispatched full sync on 2026-08-09 ran for 12h19m and was then killed by the runner disappearing, losing the whole run.

Run 31320701999, job 93264624735:

2026-08-09T15:50:36Z  Starting 1 test across 25 binaries (1283 tests skipped)
2026-08-10T04:09:05Z  ##[error]The runner has received a shutdown signal. This can happen
                      when the runner service is stopped, or a manually started runner is canceled.
2026-08-10T04:09:06Z  ##[error]The operation was canceled.

This was not a timeout — sync-full-mainnet sets is_long_test: true, giving it a 7200-minute (5-day) limit. It was not GCP capacity either. The job runs on runs-on: zfnd-runners, and the runner went away mid-test.

For reference, the last successful sync-full-mainnet (2026-07-31) took 20 hours, so any recycling inside a 20-hour window destroys the run. There is no retry or resume: the GCP instance is torn down and the result is lost.

The is_long_test input is documented as "Does this test need multiple run jobs? (Does it run longer than 6 hours?)", which suggests splitting long tests across jobs was already contemplated — that would also bound the loss from a single recycle.

Combined with #11226, full sync has not completed successfully since 2026-07-31.

Possible directions

• Prevent zfnd-runners from recycling while a job is in flight, or drain rather than kill.
• Split long tests across multiple jobs, as is_long_test hints at, so a recycle costs one segment.
• Retry the job automatically when it ends with a runner shutdown rather than a test result.