Release primary lock on election timeout

[munakoiso]

Bug summary

During a failover election, if the winning replica timed out while waiting for the election manager to release its lock (epoch_manager), it would throw ElectionTimeout without first releasing the leader (PRIMARY_LOCK_PATH) ZooKeeper lock it had already acquired. This left the cluster in a broken state: the "winner" replica held the leader lock in ZK but never actually promoted itself, and no other replica could start a new election because the lock appeared occupied.

Root cause

In _participate_in_election(), after successfully acquiring PRIMARY_LOCK_PATH, the code waits for the manager's epoch_manager lock to be released (a signal that the manager has finished its cleanup). If this wait times out (because the manager lost its ZK connection and its session had not yet expired), ElectionTimeout was raised without releasing PRIMARY_LOCK_PATH:

# Before fix — lock is leaked on timeout
if not self._zk.try_acquire_lock(self._zk.PRIMARY_LOCK_PATH, timeout=self._timeout):
    return False
if not self._await_lock_holder_fits(...):  # ← may time out
    raise ElectionTimeout               # ← PRIMARY_LOCK_PATH never released!

The exception was caught higher up in _make_election() which logged the error and returned False (no promote). Meanwhile, subsequent calls to make_election() would immediately bail out on seeing the orphaned leader lock:

if self._zk.get_current_lock_holder(self._zk.PRIMARY_LOCK_PATH):
    return False  # lock held by the stuck winner → no new election possible

The cluster stayed without a primary until the ZK session of the stuck replica eventually expired and the lock was auto-released.

Observed scenario (from faultstorm test logs)

1. pg3 (primary) killed.
2. pg2 acquires epoch_manager, becomes election manager. Its local PG is also dead, so it votes with lsn=0.
3. pg1 becomes a participant, votes with lsn=1426231568. Gets elected winner.
4. pg1 acquires leader lock (06:24:50).
5. pg2 loses ZK connection (06:24:52) — its epoch_manager session lock is still held because the ZK session has not expired yet.
6. pg1's 5-second election_timeout waiting for epoch_manager to become empty expires at 06:24:55.
7. ElectionTimeout is thrown, leader lock is not released.
8. pg1 stays as a replica, cluster has no primary.

Fix

Release PRIMARY_LOCK_PATH explicitly

This allows the next iteration to start a fresh election round without being blocked by an orphaned lock.