SCHED-1897: Add guard to avoid calling get_node_info out of hc_program for undrain…

[fabrizio2210]

Problem

Passive checks can generate unnecessary Slurm RPCs from job-coupled contexts.

In particular, boot_disk_full is configured with on_ok: undrain and runs in both prolog and hc_program. Although the check itself only performs a local disk-usage check, check_runner.py evaluates the on_ok: undrain path after a successful prolog run. This calls get_node_info(), which executes scontrol show node ... --json.

As a result, every successful boot_disk_full execution in prolog can produce an unnecessary scontrol show node call. This adds per-job/per-node RPC load to Slurm and was identified as one of the main contributors in SCHED-1897.

Solution

Restrict on_ok recovery actions to the hc_program context.

undrain and uncomment are intended to be issued only by periodic health checks, not by prolog or epilog. With this change, successful checks in prolog/epilog no longer evaluate the undrain/uncomment path and therefore no longer call get_node_info() just to decide whether a node should be resumed or uncommented.

Failure handling remains unchanged: checks can still drain or comment nodes on failure according to their configuration.

Testing

Tested with local check_runner_test.py.

Release Notes

Fix: Reduced Slurm RPC load from passive checks by preventing prolog/epilog success paths from evaluating undrain/uncomment actions.

The intended auto-recovery behavior remains available from hc_program; no user-facing configuration changes or migrations are required.

[fabrizio2210]

This is the stats of running 100 jobs on 2 nodes (all the jobs were scheduled on one node).

Count	Context	Check	Command class
100	prolog	boot_disk_full	show node
100	prolog	alloc_mem_used	show job
3	hc_program	boot_disk_full	show node
3	hc_program		show node
1	hc_program	job_tmpfs_delete_leftover	listjobs

After the fix, same amount of jobs:

Count	Context	Check	Command class
100	prolog	alloc_mem_used	show job
7	hc_program		show node
1	hc_program	job_tmpfs_delete_leftover	listjobs

It means 48% less RPC calls.