feat(runner): track per-sandbox host-side file descriptor usage by mu-hashmi · Pull Request #5001 · daytonaio/daytona

mu-hashmi · 2026-06-11T00:52:38Z

TL;DR: the runner now samples per-sandbox host-side file-descriptor usage on its existing 5s tick and exposes Prometheus gauges plus a rate-limited operator warning - zero API surface change.

Description

During the fd-exhaustion incident, host-side fd budgets were exhausted while guest-side indicators looked fine; operators had no per-sandbox visibility. This PR adds it:

Sampling rides the existing allocated-resources tick: docker init PID -> cgroup members (v1/v2) -> per-process fd counts (dirent count, no per-fd reads) + soft RLIMIT_NOFILE.
Gauges on the runner's existing protected /metrics endpoint: sandbox_open_fds, sandbox_fd_limit (0 = every sampled process unlimited; dashboards must guard division), sandbox_fd_usage_percent (worst-process ratio - RLIMIT_NOFILE is per-process, so the worst process is what actually hits EMFILE), labeled by sandbox_id.
Warning log when worst-process usage crosses SANDBOX_FD_USAGE_WARNING_THRESHOLD_PERCENT (default 70, validated 1-100), with a hysteresis band and 10m re-warn interval; recovery logs once; label sets and tracker state are pruned when sandboxes disappear.
Transient sampling failures are NOT treated as disappearance: warn-tracker state survives error ticks (no series flapping or re-warn spam); gauges are absent for the failed tick only.
Best-effort attribution of runtime helper processes (shim descendants outside the container cgroup) into a separate sandbox_runtime_helper_open_fds gauge; failures degrade to absent metrics, never errors.
Deliberately no OpenAPI change: both spec-typed metrics consumers are untouched, so no client regen. Promoting fd data into the runner healthcheck is a follow-up decision.

Known limits: nested-cgroup sandboxes undercount (members enumerated non-recursively); reading /proc/<pid>/fd requires the privileges the runner already runs with.

Review guide

Suggested order:

apps/runner/internal/metrics/fd.go - sampling math, limit parsing, worst-process computation, warn tracker (risk center: warn-state lifecycle across failure ticks and the hysteresis floor, reachable across the full 1-100 threshold range).
apps/runner/internal/metrics/collector.go - tick integration, presence-before-sampling, helper attribution gate.
Tests (fake /proc + cgroup trees).

No generated files.

Commit map

Commit	What	Why it exists
`b988909`	sample per-sandbox fd usage in collector	implementation (plan)
`b197e0a`	threshold warning with hysteresis	implementation (plan)
`c5aa131`	best-effort helper-process attribution	implementation (plan)
`2df2ffe`	warn state survives transient failures; hysteresis floor; undercount fix	Copilot + cubic review (3 valid findings)

Related PRs

Independent. Sibling fd-incident PRs: #5002 (degradedReason surfacing), #5003 (daemon shell resolution).

Validation

11 unit tests against fake /proc + cgroup trees (sum/worst-process math, unlimited handling, cgroup v1/v2 resolution, hysteresis incl. threshold=3 clear, re-warn interval, prune, transient-failure survival, undercount case, helper attribution) - run on linux in a container (apps/runner does not build on darwin; netlink dependency).
GOOS=linux go build ./..., gofmt clean, golangci-lint v2.6.2 zero issues, go work sync no drift.
All CI checks green on 2df2ffe. (Earlier red runs on this PR were the golangci schema-download flake, tracked in ci: golangci-lint config verify fetches a remote schema and flakes the entire lint matrix #5007 - never a lint finding.)

Closes #4995

Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>

nx-cloud · 2026-06-11T00:54:31Z

View your CI Pipeline Execution ↗ for commit 2df2ffe

Command	Status	Duration	Result
`nx e2e:cleanup daytona-e2e`	✅ Succeeded	<1s	View ↗
`nx run-many --target=test:e2e --all --nxBail=true`	✅ Succeeded	6m 46s	View ↗
`nx e2e daytona-e2e`	✅ Succeeded	56s	View ↗
`nx run-many --target=build --projects=api,runne...`	✅ Succeeded	5s	View ↗

💡 Verify your cache is correct by running tasks in a sandbox. Read docs ↗

☁️ Nx Cloud last updated this comment at 2026-06-11 04:10:16 UTC

Copilot

Pull request overview

This PR adds per-sandbox host-side file descriptor (FD) telemetry to the runner’s Prometheus /metrics endpoint, sampled on the existing allocated-resources snapshot tick, and includes operator warning logs when per-sandbox FD usage crosses a configurable threshold.

Changes:

Introduces new per-sandbox Prometheus gauges for open FDs, FD soft limit (RLIMIT_NOFILE), usage percent, and runtime-helper FD attribution.
Implements Linux-only FD sampling by mapping container init PID → container cgroup members → per-process FD counts + RLIMIT_NOFILE, plus best-effort runtime-helper attribution.
Adds config wiring for SANDBOX_FD_USAGE_WARNING_THRESHOLD_PERCENT and warning hysteresis/re-warn tracking + pruning.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
apps/runner/pkg/common/metrics.go	Adds new per-sandbox FD-related Prometheus gauge vectors.
apps/runner/internal/metrics/fd.go	Implements proc/cgroup-based FD sampling, runtime-helper attribution, and warning state tracking.
apps/runner/internal/metrics/fd_test.go	Adds unit tests for FD sampling logic, cgroup resolution, helper attribution, and warning hysteresis/rewarn/prune behavior.
apps/runner/internal/metrics/collector.go	Wires FD sampling into the allocated-resources snapshot loop; updates container inspect usage and adds cleanup/prune handling.
apps/runner/go.mod	Promotes `github.com/prometheus/procfs` to a direct dependency.
apps/runner/cmd/runner/main.go	Passes the configured warning threshold into the metrics collector.
apps/runner/cmd/runner/config/config.go	Adds envconfig/validation for the FD usage warning threshold percent.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

mu-hashmi · 2026-06-11T20:03:02Z

+	sandboxId := strings.TrimPrefix(containerJSON.Name, "/")
+
+	usage, err := sampleSandboxFdUsage(procRoot, cgroupRoot, containerJSON.State.Pid)
 	if err != nil {
-		return 0, 0, 0, err
+		c.log.DebugContext(ctx, "Failed to sample sandbox fd usage", "sandbox_id", sandboxId, "error", err)
+		return
+	}
+
+	seen[sandboxId] = struct{}{}
+	common.SandboxOpenFds.WithLabelValues(sandboxId).Set(float64(usage.OpenFds))


Fixed in 2df2ffe: presence is recorded before sampling, so a transient sampling failure no longer prunes warn state; gauges are dropped for the failed tick only.

cubic-dev-ai

3 issues found across 7 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

…d fix edge cases Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>

mu-hashmi added 3 commits June 10, 2026 17:46

feat(runner): sample per-sandbox host-side fd usage in metrics collector

b988909

Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>

feat(runner): warn when sandbox fd usage crosses configurable threshold

b197e0a

Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>

feat(runner): best-effort fd attribution for runtime helper processes

c5aa131

Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>

mu-hashmi marked this pull request as ready for review June 11, 2026 03:32

Copilot AI review requested due to automatic review settings June 11, 2026 03:32

Copilot started reviewing on behalf of mu-hashmi June 11, 2026 03:32 View session

Copilot AI reviewed Jun 11, 2026

View reviewed changes

cubic-dev-ai Bot reviewed Jun 11, 2026

View reviewed changes

Comment thread apps/runner/internal/metrics/collector.go

Comment thread apps/runner/internal/metrics/fd.go Outdated

Comment thread apps/runner/internal/metrics/fd.go

fix(runner): keep fd warn state across transient sampling failures an…

2df2ffe

…d fix edge cases Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>

This was referenced Jun 11, 2026

feat(api,runner): surface sandbox file-descriptor exhaustion as degradedReason #5002

Open

fix(daemon): resolve the shell in-process and cache successful resolution #5003

Open

mu-hashmi requested review from brunogrbavac and idagelic June 11, 2026 21:22

vedranjukic force-pushed the main branch 2 times, most recently from 2c15a7f to b40f732 Compare June 23, 2026 14:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(runner): track per-sandbox host-side file descriptor usage - #5001

feat(runner): track per-sandbox host-side file descriptor usage#5001
mu-hashmi wants to merge 4 commits into
mainfrom
feat/sandbox-fd-telemetry

mu-hashmi commented Jun 11, 2026 •

edited

Loading

Uh oh!

nx-cloud Bot commented Jun 11, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

mu-hashmi Jun 11, 2026

Uh oh!

cubic-dev-ai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

mu-hashmi commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Review guide

Commit map

Related PRs

Validation

Uh oh!

nx-cloud Bot commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

mu-hashmi Jun 11, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mu-hashmi commented Jun 11, 2026 •

edited

Loading

nx-cloud Bot commented Jun 11, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading