Skip to content

runner: track per-sandbox host-side file descriptor usage #4995

Description

@mu-hashmi

Is your feature request related to a problem? Please describe.

The runner reports per-sandbox CPU, memory, and disk, but nothing about file descriptor usage of the host-side processes backing a sandbox. File-handle exhaustion in a sandbox's filesystem path is invisible until workloads start failing inside the guest, and the in-guest indicators (ulimit -n, /proc/sys/fs/file-nr) do not reflect the host-side budget actually being consumed. Operators have no way to see pressure building, correlate it to a sandbox, or alert before impact.

Describe the solution you'd like

A per-sandbox open-fd gauge collected by the runner alongside the existing per-sandbox resource metrics: sum of open descriptors across the sandbox's associated host processes (container cgroup members, plus runtime helper processes where the configured container runtime spawns them), plus the effective limit for those processes so usage can be expressed as a percentage. A configurable warning threshold (default around 70%) should emit a log/event suitable for alerting.

Acceptance criteria:

  • Runner exposes per-sandbox open-fd count and effective limit
  • Works for the default runtime; runtime helper processes are attributed to the owning sandbox where present
  • Threshold crossing emits an operator-visible event with sandbox ID
  • Negligible collection overhead at fleet scale (sampled, not per-request)

Describe alternatives you've considered

In-guest monitoring (cannot see host-side budgets, which is precisely the blind spot). Host-global fd monitoring (no per-sandbox attribution, so no actionable alert).

Additional context

Write-heavy small-file workloads (dependency installs are the common benign case) can drive host-side fd consumption far beyond what guest indicators suggest. This metric is the difference between a capacity alert and a customer-reported outage.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions