admission: export bypassed requests as a metric#171657
Open
shreyasganesh0 wants to merge 1 commit into
Open
Conversation
The IO load listener logs the per-interval count of requests that bypass admission control (below-raft writes that were not subject to replication admission control) in its [accounting] log line, but this count was never exported as a metric. Operators investigating overload escalations had to read logs to find it. The existing admission.granter.io_tokens_bypassed.kv counter tracks bypassed tokens, not the request count. Add an admission.granter.bypassed_requests.kv counter, incremented every adjustment interval with the same per-interval bypassed-request count that is logged. A growing value indicates replication writes from nodes that do not have replication admission control enabled, which is a useful signal that a version upgrade would relieve the overload. Epic: none Fixes: cockroachdb#137408 Release note (ops change): Added the admission.granter.bypassed_requests.kv metric, which counts requests that bypassed admission control (for example, below-raft writes not subject to replication admission control).
|
Thank you for contributing to CockroachDB. Please ensure you have followed the guidelines for creating a PR. My owl senses detect your PR is good for review. Please keep an eye out for any test failures in CI. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The IO load listener logs the per-interval count of requests that bypass admission control (below-raft writes that were not subject to replication admission control) in its
[accounting]log line, but this count was never exported as a metric. Operators investigating overload escalations had to read logs to find it. The existingadmission.granter.io_tokens_bypassed.kvcounter tracks bypassed tokens, not the request count.This adds an
admission.granter.bypassed_requests.kvcounter, incremented every adjustment interval with the same per-interval bypassed-request count that is logged. A growing value indicates replication writes from nodes that do not have replication admission control enabled, which is a useful signal that a version upgrade would relieve the overload.Open question (carried over from the issue): aggregate request count only, or would you also like the bypassed write/ingest bytes exported alongside? Happy to add that if wanted.
Epic: none
Fixes: #137408