NodePublishVolume fails with "failed to set source device readwrite" when DRBD Primary role is stuck on a node other than the one CSI publishes to

[vooodoo10]

What happened

Multiple newly-provisioned PVCs (created via volumeBindingMode: Immediate StorageClasses) got stuck in ContainerCreating with:

rpc error: code = Internal desc = NodePublishVolume failed for <pvc>: failed to set source device readwrite: exit status 1

drbdadm status <resource> on each node shows the DRBD role is Primary (with open:no — unused) on a node that is different from the node the pod was actually scheduled to and that NodePublishVolume is targeting. Example:

# on node az2-01 (stale primary, open:no):
pvc-ddadb066-... role:Primary
  disk:UpToDate open:no
  az2-02 role:Secondary
  az2-03 role:Secondary

# pod is actually scheduled on az2-03, which is Secondary and can't promote
# because az2-01 already holds Primary.

This reproduces on plain (non-topology-restricted) replicated-style StorageClasses, so it isn't specific to any topology/replicasOnSame constraint — it looks like a race between initial volume formatting (which transiently promotes some node to Primary) and the pod's eventual scheduled node, with no handoff/demotion of the stale Primary afterward.

Expected behavior

NodePublishVolume on the pod's target node should either wait for / trigger demotion of a stale, unused (open:no) Primary elsewhere, or the CSI driver should not leave an unused Primary behind after formatting.

Steps to reproduce

1. Create a StorageClass with volumeBindingMode: Immediate and a DRBD-backed resource group (layerList: "drbd storage", autoPlace >= 2).
2. Create a PVC + pod (e.g. a StatefulSet where the pod initially fails to schedule and retries — FailedScheduling: 0/N nodes are available: pod has unbound immediate PersistentVolumeClaims — before finally landing).
3. Observe the pod stuck in ContainerCreating, FailedMount with "failed to set source device readwrite".
4. Compare drbdadm status <resource> across all nodes hosting a replica — one will show role:Primary disk:UpToDate open:no on a node other than the pod's node.

Workaround

drbdadm secondary <resource> on the node holding the stale, unused (open:no) Primary. Kubelet's mount retry then succeeds.

Attachments

See accompanying diagnostic bundle: DRBD status output across nodes, linstor resource/volume list, linstor-csi node logs, PVC/PV manifests, StorageClass manifest, and full events for the affected pods.

Environment

• Kubernetes v1.35.6 (Talos Linux, 3-node cluster)
• piraeus-operator v2.10.8
• linstor-controller (piraeus-server) v1.33.3
• linstor-csi (piraeus-csi) v1.11.3
• DRBD kernel module 9.3.2, drbd-utils GIT 54d5a64

00-affected-resources.txt
drbd-status.txt
linstor-resource-state.txt
original-error.log

[WanzenBug]

Are there any error reports on the LINSTOR side? LINSTOR temporarily needs to set the resource primary to start the sync. If it fails to set the resource secondary afterwards, there should be error reports pointing at a cause.

In my test set up, I cannot reproduce this issue as described.

[vooodoo10]

The root cause is StorageException: Failed to become secondary again after creating filesystem logged in DrbdLayer.java:988 on DrbdAdm.java:830.
After LINSTOR promotes the resource to Primary to create the filesystem, drbdadm secondary fails with exit code 11 and the following kernel message:

State change failed: (-12) Device is held open by someone /dev/drbd1002 open_cnt:1, writable:1; list of openers follows drbd1002 opened by init (pid 1514) at 2026-06-03 10:55:09.454
The device is opened by init (pid 1514) the moment the DRBD block device appears — before LINSTOR finishes the promote→filesystem→demote sequence. LINSTOR logs the StorageException and aborts, leaving the replica permanently stuck in Primary role on that satellite node. Any pod later scheduled to a different node then fails to mount with: failed to set source device readwrite: exit status 1 because it cannot promote its own replica while another is already Primary.
This pattern has repeated across all 3 satellites (June 3, June 7, June 16, July 12 2026). The only recovery is manually running drbdadm secondary on the satellite holding the stale Primary.

When LINSTOR temporarily promotes a DRBD resource to Primary to create the filesystem (runPostInitializationIfNeeded in DrbdLayer.java:988), it then calls drbdadm secondary <resource> to demote it back — but the kernel rejects with:
State change failed: (-12) Device is held open by someone
/dev/drbd1002 open_cnt:1, writable:1; list of openers follows
drbd1002 opened by init (pid 1514)

[WanzenBug]

Hmm, I guess this is a udev problem. DRBD has some special code to prevent a udev from interfering with demotion, but I would guess that assume a "regular" systemd-based system, so it does not work on Talos.