fix: deterministic election recovery for restarted and equal-clock nodes (#2197)

[oliver0006]

Motivation

Fixes #2197 — both reported wedges.

1. Restarted master invisible forever: election_clock is process-volatile, and peers silently dropped any message below their clock. A master restarting from clock 0 into a cluster at clock N campaigned into the void: nobody answered, so it could never learn the cluster clock, and recovery depended on an incidental connection event.
2. Dead round: an equal-clock candidacy arriving while no campaign was active was appended to _candidates — a list that only a running campaign ever evaluates (max(candidates) at timeout). The candidacy sat there forever.

Changes

• Reply to stale messages (message.clock < self.clock): instead of dropping silently, the receiver answers with its current status. This is the missing piece of the classic bully algorithm: the stale campaigner receives a higher-clock message, adopts the cluster clock through the existing clock > self.clock path, and rejoins the election deterministically. Replies are bounded (one per stale message received) and can never ping-pong, since staleness is strictly one-sided.
• Track campaign liveness (_campaigns_started/_campaigns_finished, incremented at scheduling time and in the campaign's finally): an equal-clock candidacy arriving with no active campaign now starts one, so the round resolves instead of dying in the list.

Why It Works

Both fixes stay at the protocol level — no persistence, no new message types, no changes to the ordering semantics of ElectionMessage. A restarted node converges in one round-trip: its stale broadcast triggers a status reply, the reply carries the cluster clock, and the normal higher-clock path takes over from there. The issue's Additional context mentions a durable on-disk election clock as the fix deployed in your fork — that approach also covers full-cluster restarts and would compose cleanly with this change; happy to discuss whether it's wanted upstream as a follow-up (@rogerstom-lgtm).

Note on an existing test: test_ignores_older_messages asserted the silent-drop behavior this issue reports as a bug. It's updated to assert the new contract: an older message still never changes the clock or starts a round, but the sender now gets a status reply.

Test Plan

Manual Testing

Hardware: Fedora Linux. Ran the new regression tests against the unfixed election.py: 6 fail (both unit repros, the two-node convergence test, and the updated stale-message contract, across both async backends). Against the fix: all 20 pass.

Automated Testing

Following the unit-level repro shapes suggested in the issue:

• test_equal_clock_candidacy_without_active_campaign_starts_a_round — an equal-clock candidacy from a better candidate, arriving after the previous round resolved, must produce an ElectionResult electing it (used to die in the list).
• test_stale_message_gets_current_status_reply — a node campaigning at clock 1 while the cluster is at 5 must receive a status reply carrying clock 5 (used to get silence).
• test_restarted_master_rejoins_cluster_deterministically — end-to-end: two wired Election instances, survivor at clock 5, fresh node booting from clock 0; the restarted node must reach the cluster clock and both nodes must converge on a single session, with no connection event involved. Asserts convergence rather than the winner, since round outcome depends on seniority tie-breaking that predates this change.

Full uv run pytest src → 345 passed. uv run basedpyright 0 errors; ruff check and ruff format --check pass.

---

Disclosure: co-written by a human (@oliver0006) directing the work and Claude AI (Fable 5) writing code, following AGENTS.md and .clauderules.

🤖 Generated with Claude Code