Orphaned GCP instance blocks the full-sync cron, and the error is reported as a capacity problem

[oxarbitrage]

The Friday full-sync cron failed on 2026-08-07 after 22 minutes because a leftover instance blocked creation. No full sync has succeeded since 2026-07-31.

Run 31177507471, job 92867258967:

ERROR: (gcloud.compute.instances.create-with-container) Could not fetch resource:
 - The resource 'projects/zfnd-dev-zebra/zones/us-east1-c/instances/
   sync-full-mainnet-main-3dd9756' already exists

Three separate problems:

1. Cleanup and the cron are misordered, so one collision costs a week. Instance names are <test>-<branch>-<sha>, so a leftover instance collides with the next run that resolves to the same name. "Delete GCP resources" runs daily at ~07:20 and the weekly full-sync cron runs Fridays at ~12:16 — so an instance that survives the morning cleanup blocks that day's cron, and is only removed the following morning, by which point the cron has failed and will not run again for seven days.

This is a race, not a leak: the instance above was removed by a later cleanup and no longer exists. But the sync it blocked is still lost.

2. The diagnostic is wrong, and expensively so. After the create failed, the job logged:

::error::All candidate zones (...) remain out of capacity after 2 sweeps

It was not a capacity problem. Anyone investigating starts by looking at quota and zone availability, which is the wrong place entirely.

3. The error path is broken. The failed create falls through to gcloud compute ssh --zone with an empty zone argument, producing argument --zone: expected one argument — a second error that masks the first.

Suggested fixes

• Delete a pre-existing instance of the same name before creating, or make the name unique per run. Either removes the race regardless of cleanup timing.
• Only report "out of capacity" when the failure was actually a capacity error; otherwise surface the real gcloud message.
• Stop the post-create steps when creation failed, instead of running them with empty arguments.

Unrelated stale resources in zfnd-dev-zebra

Found while investigating. Listed for triage rather than as part of this bug — some may be deliberate:

Resource	Created	State
zebrad-miner-with-disks-5469302-testnet	2024-10-04	RUNNING (~22 months)
full-sync-tmp-test-vm	2025-07-04	terminated
zebrad-cache-replay-test-testnet (400 GB)	—	unattached disk
zebrad-cache-v2-1-0-179aaa4-testnet (400 GB)	—	unattached disk

The zebrad-main-* instances in that project are active CI nodes and are not included above.

[oxarbitrage]

Updated the description after checking the project directly.

Correction: I originally wrote that the daily cleanup "doesn't target these" instances. That was wrong — sync-full-mainnet-main-3dd9756 has since been removed and no longer exists. The bug is a race, not a leak: the cleanup runs daily at ~07:20 and the full-sync cron at ~12:16 on Fridays, so an instance that survives the morning cleanup blocks that day's cron and is only removed the next morning — by which time the run has failed and won't retry for a week. That makes fixes 1 (delete-before-create, or unique names) the real fix, since it removes the race regardless of cleanup timing.

Also added an inventory of unrelated stale resources in zfnd-dev-zebra spotted while looking — including an instance running since 2024-10-04 and 800 GB of unattached disk. Listed for triage, not as part of this bug; some may well be deliberate.

[oxarbitrage]

Related but distinct: #11232.

A manually dispatched full sync on 2026-08-09 got past instance creation fine, ran for 12h19m, and was then killed by the zfnd-runners runner being recycled mid-job — a different failure mode with an independent fix, so I filed it separately rather than folding it in here.

Between the two, sync-full-mainnet has not completed successfully since 2026-07-31.