Durable Emitter - TTL budget metric

[tarcisiozf]

This pull request refactors how the durable emitter tracks and reports queue expiry pressure metrics. Instead of counting the number of events near expiry, it now measures the "TTL budget"—the remaining time before the oldest pending event expires. This simplifies the code and provides a more direct signal for alerting on dead-letter queue (DLQ) pressure.

Metrics and Observability Improvements:

• Replaced the NearTTLCount metric (number of events near expiry) with a new TTLBudget metric, which tracks the remaining TTL headroom for the oldest pending event. This is now reported as queue.ttl_budget_seconds.
• Updated the DurableQueueStats struct and the ObserveDurableQueue interface to remove the nearExpiryLead parameter and field, and instead use the new TTLBudget field.
• Updated metric recording and gauge definitions to support the new TTL budget metric, including changes to metric names and types.
• Simplified the queue statistics observation logic in all stores (PgDurableEventStore, MemDurableEventStore, and metric wrappers) to remove all logic related to "near expiry" windows and focus only on TTL budget calculation.

[Copilot]

Pull request overview

This PR refactors DurableEmitter queue-expiry pressure observability by replacing the “near expiry count” signal with a more direct “TTL budget” signal (time remaining before the oldest pending event reaches EventTTL). This flows through the observer interface, store implementations, and metric instruments.

Changes:

• Replaced NearTTLCount / NearExpiryLead-based observation with TTLBudget (EventTTL - OldestPendingAge) in DurableQueueStats and ObserveDurableQueue.
• Updated Postgres + in-memory observer implementations and the emitter metrics loop to use the new observer signature.
• Renamed/retyped the gauge from durable_emitter.queue.near_ttl (int) to durable_emitter.queue.ttl_budget_seconds (float).

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pkg/durableemitter/store.go	Computes TTLBudget from the oldest pending event instead of querying a near-expiry count.
pkg/durableemitter/observable_store.go	Updates DurableQueueStats and DurableQueueObserver interface to use TTLBudget and removes nearExpiryLead.
pkg/durableemitter/durable_emitter.go	Removes near-expiry lead plumbing and updates metrics loop to call the new observer signature.
pkg/durableemitter/durable_emitter_test.go	Updates test in-memory store observer to compute TTLBudget.
pkg/durableemitter/durable_emitter_metrics.go	Introduces the durable_emitter.queue.ttl_budget_seconds gauge and records it from observed stats.

Comments suppressed due to low confidence (1)

pkg/durableemitter/durable_emitter.go:788

• When PollInterval is unset/zero, metricsLoop currently defaults to polling every 500ms, but DurableEmitterMetricsConfig.PollInterval is documented as "Zero = 10s". This mismatch is likely to surprise callers and can significantly increase DB load due to ObserveDurableQueue being called 2x/sec by default. Consider aligning the implementation with the documented 10s default (or update the doc if 500ms is intended).

	mc := d.cfg.Metrics
	poll := mc.PollInterval
	if poll <= 0 {
		poll = 500 * time.Millisecond
	}

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.