fix(runner): exit when the parent worker dies — orphaned runners pin memory and RDMA state#2205
Open
aidiffuser wants to merge 1 commit into
Open
fix(runner): exit when the parent worker dies — orphaned runners pin memory and RDMA state#2205aidiffuser wants to merge 1 commit into
aidiffuser wants to merge 1 commit into
Conversation
…hundreds of GB A hard-killed worker (SIGKILL, crash) silently orphans its runner child (reparented to pid 1), which keeps all Metal/unified-memory allocations and RDMA queue pairs alive indefinitely. The failure is hard to diagnose: the orphan's RSS looks tiny (a few GB) while its real physical footprint can be hundreds of GB, so the node reports low available RAM, placements start failing with "likely does not fit", and replacement runners crash-loop on the held queue pairs (jaccl: Changing queue pair to RTR failed with errno 16). Observed repeatedly on a 2-node tensor-parallel deployment: a ~465 GB model's orphaned runner showed 6 GB RSS but a 313 GB footprint (per macOS footprint(1)). Fix: a small daemon watchdog thread in the runner polls the parent pid and hard-exits when it changes; there is nothing to clean up gracefully once the parent is gone. Escape hatch: EXO_DISABLE_ORPHAN_WATCHDOG=1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
aidiffuser
force-pushed
the
upstream-orphan-watchdog
branch
from
394f6fc to
78cc99b
Compare
EternaPeptix
added a commit
to EternaPeptix/exo
that referenced
this pull request
Jul 6, 2026
…ry + RDMA Cherry-pick of exo-explore#2205 (aidiffuser). When the parent exo worker is hard-killed (SIGKILL, crash, or our `launchctl kickstart -k` recovery path), its multiprocessing runner child is silently reparented to pid 1 and keeps living — holding all Metal/unified-memory model allocations and the multi-node RDMA queue pairs. The orphan is nasty to diagnose: its RSS looks small while the real physical footprint is huge (PR exo-explore#2205 observed 6 GB RSS vs 313 GB footprint on a 465 GB model). Two downstream consequences, both of which we've seen on this cluster: - the node reports low ramAvailable, so placements fail with "does not fit" even though the model would fit fine; - replacement runners crash-loop on the still-held queue pairs ([jaccl] Changing queue pair to RTR failed / Recv failed) while the link itself looks healthy. This is plausibly a root cause of the stale-QP state that our RDMA watchdog was built to recover from: if the orphan never releases its QPs, the next runner can't establish its own, and the QP layer looks "stale" from the new runner's perspective. Fix: a small daemon thread in the runner records the parent pid at startup, polls os.getppid() every 5s, and hard-exits (os._exit(1)) when it changes. Once the parent is gone there is nothing to clean up gracefully — exiting is what releases the memory and the queue pairs. Escape hatch: EXO_DISABLE_ORPHAN_WATCHDOG=1. Tests (test_orphan_watchdog.py, 3 cases): the disable escape-hatch, that the watchdog thread starts when enabled, and that entrypoint() invokes it. The full kill->reparent->exit flow depends on OS reparenting semantics that don't reproduce reliably from pytest (a launchd-managed runner reparents to pid 1, a pytest-spawned child does not), so the end-to-end kill test was removed in favor of deterministic coverage of the wiring. 86 tests pass on both 512S1 and 512S2.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
If the main exo process dies hard (SIGKILL, crash,
pkillin a restart script), its model-runner child is silently reparented to pid 1 and lives on holding everything: all Metal/unified-memory allocations and — for multi-node instances — the RDMA queue pairs.The failure is nasty to diagnose because the orphan hides: its RSS looks tiny while the real physical footprint stays huge. Observed on a 2-node tensor-parallel deployment serving a ~465 GB model: the orphaned runner showed 6 GB RSS but a 313 GB footprint (macOS
footprint(1)). Consequences:ramAvailable, so placements fail with 503 "likely does not fit" for models that fit fine;[jaccl] Changing queue pair to RTR failed with errno 16on the orphaned node,Recv failed errno=2on its peer) while ping and the link look healthy.Diagnosis fingerprint, for anyone who lands here from a search:
ps -axo pid,ppid,command | awk '$2==1 && /spawn_main/'.Fix
A small daemon watchdog thread in the runner records the parent pid at startup, polls
os.getppid()every 5 s, and hard-exits (os._exit(1)) when it changes. Once the parent is gone there is nothing to clean up gracefully — exiting is what releases the memory and the queue pairs (verified: memory returns within seconds of killing an orphan manually).Escape hatch:
EXO_DISABLE_ORPHAN_WATCHDOG=1for anyone intentionally debugging a runner that outlives its worker.🤖 Generated with Claude Code