Skip to content

feat(engine,producer): verified interleaved parallel drawElement streaming (opt-in)#2056

Open
vanceingalls wants to merge 2 commits into
de2-14-subtimeline-failfastfrom
de2-15-parallel-de-verified
Open

feat(engine,producer): verified interleaved parallel drawElement streaming (opt-in)#2056
vanceingalls wants to merge 2 commits into
de2-14-subtimeline-failfastfrom
de2-15-parallel-de-verified

Conversation

@vanceingalls

@vanceingalls vanceingalls commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #2045 (sub-composition timeline fail-fast). Step 2 of the DE engagement plan (plans/drawelement-fast-capture/2026-07-08-parallel-de-engagement-plan.md); step 1 was the #2026 inversion.

Why

clamp:parallel exists because parallel capture had no runtime self-verification. This PR builds the verified path: multi-worker drawElement through the streaming encoder with the full safety net riding along — opt-in via HF_DE_PARALLEL_STREAM=true, default routing (including the #2026 inversion) untouched.

Mechanism

  • Interleaved distribution (distributeFramesInterleaved, WorkerTask.frameStride): worker i takes frames i, i+N, i+2N… — seek-based capture makes stride free, and the ordered writer's reorder window shrinks from totalFrames/N to N. (Contiguous chunks serialize workers behind the writer — the original stall.)
  • Depth-2 pipelined worker-encode produce in the parallel worker loop, same shape as the sequential loop.
  • Verification threaded: the drain guard is extracted to createDrainFrameGuard (session-parameterized) — every parallel frame gets the same blank-guard + PSNR self-verify as the sequential drain, against its owning worker's pre-injection ground truth (all sessions arm identical sample indices).
  • FrameReorderBuffer.abort(err): a failed worker rejects all parked/future waiters. Without it, a verification failure in one worker deadlocks its peers forever (the pool awaits ALL workers before surfacing errors) — found by the verify-trip test, unit-tested.
  • Typed error preservation: DrawElementVerificationError survives the pool's error-string flattening so the orchestrator's verify-retry fires.
  • Dedup stride hazard fixed: lastEncodeResult reuse now requires every frame in (lastEncodeResultFrame, i] to be predicted-static — sequential behavior unchanged, stride-N reuse safe.
  • Separate browser processes per worker under the flag: co-tenant pages starve non-active pages of BeginFrames on the paint-wait path (measured 86s vs 30s on a 3,245-frame rAF comp).

Validation

  • Happy path W3: verify samples pass across workers (4×∞ on the 2,381f comp); parallel output vs single-worker DE = 59.3 dB (encode noise floor) — interleave + dedup-stride produce identical pixels.
  • Verify-trip (marginal comp, HF_DE_VERIFY_MIN_DB=45): fails at frame 649 (32.2 dB), peers abort instead of deadlocking, render retries via parallel screenshot, RENDER_OK in 42.6s.
  • Canary suite 7/7 with the flag off; orchestrator tests 99/99; engine 909 pass (14 pre-existing main failures, stash-A/B verified); new reorder-buffer abort tests.

Perf — clean benchmark complete, VERDICT: promote

36-run benchmark (4 comps × {single, par2, par3} × 3 reps, interleaved order) on a quiet machine (load1 held 4–6 throughout, hard-fail load gate, 0 gate timeouts). 4/4 verify + 0 fallback on every run, max within-cell drift 8.3% (no monotonic drift — confirms the machine was actually quiet).

comp frames par2/single par3/single par3 spread
0768f038 (GSAP graphics) 2,381 1.13× 1.36× 0.7%
398c0655 (rAF, high-variance) 3,245 1.16× 1.29× 7.7%
07856d80 (39% static/dedup-heavy) 3,600 1.07× 1.16× 4.9%
13ce38c6 (915f crossover probe) 915 1.16× 1.27× 5.6%

2/3 comps ≥2,000 frames clear the pre-agreed 1.25× bar on totals with spread <20%; the dedup-heavy comp doesn't (1.16×), which is mechanically expected — static-frame dedup already skips the capture work parallelism would split. No comp anywhere shows par3 < single; par2 is monotonically weaker than par3 on every comp (real parallel scaling, not noise).

Merging this as opt-in (HF_DE_PARALLEL_STREAM=true, default routing unchanged). Next: a separate default-off router PR preferring verified parallel DE over the #2026 single-worker inversion above a measured frame threshold, then a 2-week telemetry soak (revert rate ≤2%, de_verify_min_db p10 stable) before any default flip.

🤖 Generated with Claude Code

vanceingalls and others added 2 commits July 8, 2026 00:25
…aming (opt-in)

Step 2 of the DE engagement plan: multi-worker drawElement capture through
the streaming encoder, with the full runtime self-verification net riding
along — the confinement rule that kept the parallel clamp in place is now
satisfied on this path. Opt-in via HF_DE_PARALLEL_STREAM=true; default
routing (including the #2026 single-worker inversion) is unchanged.

Mechanism:
- distributeFramesInterleaved + WorkerTask.frameStride: worker i captures
  frames i, i+N, i+2N... — seek-based capture makes stride free and the
  ordered writer's reorder window shrinks from totalFrames/N to N (contiguous
  chunks serialize workers behind the writer).
- Depth-2 pipelined worker-encode produce in the parallel worker loop (the
  same shape as the sequential loop; frame k's in-page encode overlaps
  k+stride's produce). HF_DE_PAR_DEBUG=1 traces the first frames per worker.
- Drain guard extracted to createDrainFrameGuard (session-parameterized):
  every parallel frame gets the SAME blank-guard + PSNR self-verify as the
  sequential drain, against its owning worker's pre-injection ground truth
  (all sessions arm identical sample indices from
  CaptureOptions.compositionDurationSeconds).
- FrameReorderBuffer.abort(err): a failed worker (e.g. verification error)
  rejects all parked and future waiters — without this, peers park forever
  in waitForFrame and the pool (which awaits ALL workers before surfacing
  errors) deadlocks. Found by the verify-trip test; unit-tested.
- The typed DrawElementVerificationError is preserved past the pool's
  error-string flattening so the orchestrator's verify-retry recognizes it.
- Static-dedup stride hazard fixed: lastEncodeResult reuse now requires EVERY
  frame in (lastEncodeResultFrame, i] to be predicted-static (sequential
  capture reduces to the old has(i) check).
- Workers get separate browser PROCESSES under the flag: pages co-tenant in
  one browser starve non-active pages of BeginFrames on the paint-wait path
  (measured 86s vs 30s on a 3,245-frame rAF comp).

Validation:
- Happy path W3: verify samples pass across workers (4x inf on the 2,381f
  comp), output vs single-worker DE = 59.3dB (encode noise floor) — the
  interleave + dedup-stride produce identical pixels.
- Verify-trip (marginal comp + HF_DE_VERIFY_MIN_DB=45): fails at frame 649
  (32.2dB < 45), peers abort instead of deadlocking, whole render retries
  via parallel screenshot, RENDER_OK in 42.6s.
- Canary suite 7/7 with the flag off (default paths untouched); producer
  orchestrator tests 99/99; engine suite 909 passed (14 pre-existing main
  failures, stash-A/B verified); reorder-buffer abort unit tests.

Perf note: capture-only parallel speedup measured 1.38x (W2) / 1.52x (W3)
over single-worker DE in the spike; end-to-end numbers on this machine are
currently noisy (separate-browser init overhead + bench load) — clean
benchmarks before any default routing change. The flag stays explicit
opt-in; promoting it into the router replaces the #2026 W=1 pin for the
same cohort.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant