Merge to main: Merge to soperator-release-4.1: SCHED-1041: Pre-create GPU lib placeholders so a failing nvidia-container-cli cannot break the cluster

[github-actions]

Merge back to main of the Pull Request #2731 by @theyoprst

Original PR Description

Merge back to soperator-release-4.1 of the Pull Request #2703 by @theyoprst

Original PR Description

Problem

At cold cluster bring-up all workers run nvidia-container-cli against the shared jail at the same moment. The CLI creates 0-byte placeholder files on the shared jail and bind-mounts the host driver libs over them. Confirmed in 7 e2e reproductions:

1. The workers race to create the same files. The loser fails (mount error: file creation failed: .../gsp_ga10x.bin: file exists), its container restarts, and the solo re-run succeeds — this worker always recovers by accident.
2. Before dying, the failing CLI deletes every 0-byte placeholder it had mounted — including the ones under the other worker's live bind mounts. That worker's kernel detaches its mounts, and it stays GPU-broken for the pod's lifetime: prepull-container-image fails the bring-up, or multi-node GPU jobs fail later with file too short.

The second half is not limited to bring-up races: any CLI failure on any worker at any time deletes the placeholders under the whole fleet's mounts.

Solution

Pre-create the placeholders as small non-empty marker files before the CLI runs (precreate_gpu_placeholders in complement_jail.sh):

• the file set comes from nvidia-container-cli list at runtime, so driver upgrades are handled automatically; the /dev and /run entries resolve to objects already created by the kernel and system daemons, so the existence check skips them;
• on the first bring-up the missing files are created under a cross-node flock, with the host file's mode; on every later pod start everything already exists and no lock is taken;
• the CLI then finds every file present and skips creation — no race, no crash;
• the CLI's failure cleanup only deletes empty files, so a failing CLI can no longer break the other workers, whatever the failure cause.

The CLI result is verified instead of trusted: every mount the CLI created must stay attached and show real content by path (by size — a placeholder passes plain existence checks), whatever file set the enabled driver capabilities select. The libnvidia-ml.so.1 soname is additionally verified after ldconfig (repaired by a local ldconfig run if missing) when NVML is mounted. On any failure the container exits so kubelet restarts it.

The two places that treated "empty file" as "placeholder" (the CPU-worker libdummy mounts and the populate_jail cleanup) now match by size and versioned lib name (lib*.so.* smaller than 64 bytes), covering both old 0-byte and new marker placeholders while leaving the jail image's other small files (ncurses linker scripts, glibc stub archives) alone.

Testing

• shellcheck + smoke tests with stubbed CLI/flock/stat: cold and warm paths, node-local filtering, failure exits, the size predicates.
• three green e2e cold bring-ups (28922118210, 28952699476, 28973074103, SB_MAN_H200) together exercised every pre-create path: sole creator, all-present gate, and a worker that queued on the lock mid-creation and found everything present on the re-check. Both CLIs skipped all creations in each run, no anomalies. The [nvml] log telemetry stays for verifying this in production.

Release Notes

Fix: GPU driver lib placeholders in the shared jail are now pre-created as non-empty files, so concurrent nvidia-container-cli runs never race to create them and a failing run can no longer delete them from under the other workers' bind mounts — fixing intermittent NVML load failures ("cannot open shared object file" / "file too short") on multi-worker bring-up.

---

You cannot skip this merge, but if you really don't want these changes (conflicts or doesn't make sense), just comment /revert-last-pr, and wait for the revert to come, then merge this PR (even if 0 changes).

---

You cannot skip this merge, but if you really don't want these changes (conflicts or doesn't make sense), just comment /revert-last-pr, and wait for the revert to come, then merge this PR (even if 0 changes).