Prevent shutdown if a DRBD resrource is out of sync

[nebula-it]

The Setup

The storage class is setup with linstor.csi.linbit.com/placementCount: "2". Nodes w01 and w02 are nodes backing this pool, w03 does not back this pool but gets auto added as TieBreaker

╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName                             ┊ Node    ┊ Layers       ┊ Usage  ┊ Conns ┊      State ┊ CreatedOn           ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ pvc-0b2a2a38-e993-4398-ba5e-a41d33831351 ┊ w01  ┊ DRBD,STORAGE ┊ InUse  ┊ Ok    ┊   UpToDate ┊ 2025-03-22 08:34:02 ┊
┊ pvc-0b2a2a38-e993-4398-ba5e-a41d33831351 ┊ w02  ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-03-22 08:34:00 ┊
┊ pvc-0b2a2a38-e993-4398-ba5e-a41d33831351 ┊ w03  ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ TieBreaker ┊ 2025-03-22 08:34:01 ┊

The Issue

When the nodes w01 and w02 are rebooted in quick succession (e.g for updates), following scenario happens:

1. Node w01 just came online after a reboot, the DRBD resources on it are OutofDate and its syncing those from w02
2. While this is going, we rebooted the node w02, now it went down without completing the sync
3. Now both nodes are OutofDate and fail to complete the resync.

The OS on nodes is Talos, so drbd-shutdown-guard is not implemented. I wonder, if its possible to set a high graceful shutdown time for satellite pods and there is mechanism (maybe preStop hook) that ensures there is no active sync going on at the moment before it shuts down the pod.

This is tangentially related to #860

[WanzenBug]

This is an interesting case. I believe in this situation, DRBD should still come up as UpToDate on w02, but perhaps there is a bug that prevents that.

[nebula-it]

I've had this happened more than once. Mostly when doing node OS upgrades.
I'll spin up a test cluster and try to reproduce this for further testing.