Mute ydb/tests/functional/secrets/test_old_secrets_usage.py.test_existing_objects_are_ok_if_old_secret_creation_is_disabled in main

[ydbot]

Mute:
ydb/tests/functional/secrets/test_old_secrets_usage.py.test_existing_objects_are_ok_if_old_secret_creation_is_disabled

Branch:
main

Build type:
relwithdebinfo

Owner: TEAM:@ydb-platform/security

About this issue

• Auto-created by the mute workflow.
• Bot closes when all listed tests are unmuted or no longer exist.
• Close as Completed to trigger fast-unmute.
• mute_rules.md — full details.

Summary history:
test_old_secrets_usage.py.test_existing_objects_are_ok_if_old_secret_creation_is_disabled: Flaky last 1 days, at 2026-07-24:2026-07-27: success_rate 77.8%, p-7, f-2, m-0, s-0, total-9

Test run history: link

More info in dashboard

[ydbot]

LLM instructions

Classify: TEST_ISSUE | YDB_ISSUE | TEST_INFRA_ISSUE
- YDB_ISSUE: product bug under ydb/core, ydb/library, etc.
- TEST_ISSUE: test logic, stale refs in helpers/workloads, missing waits. Fixture/harness PR
  that exposed a latent test bug → TEST_ISSUE; Responsible = fix path, not PR author.
- TEST_INFRA_ISSUE: shared harness/runner/CI broken for many unrelated tests. One directory /
  one test pattern → usually TEST_ISSUE, not TEST_INFRA_ISSUE.

Find root cause and introducing commit/PR; severity HIGH/MEDIUM/LOW; propose a fix.
In Root Cause list Suspect PR(s) most-likely-first (#12345 + why; tie to code/regression window).

Summary history (success_rate, p-*, f-*, total-*, dates): 50% often = runs before vs after a
merge, not a timing race. Compare failing tests with sibling tests (same fixture, still pass).

After fixture stop_driver(): "Driver was stopped" / "Topic client closed" → check stale driver
snapshot in workload/helpers before suggesting retries, sleeps, or graceful-shutdown revert.

VERIFY/SANITIZER — use status_description in exactly 3 lines:
  VERIFY failed (<ISO_UTC>): / ydb/.../file:line / assertion text (no stack frames, no +0x...)

Responsible (NOT issue-body Owner: — that is mute assignee only):
Pick team for the root-cause fix (first match):
  a) Fix path → .github/TESTOWNERS longest prefix (wins if PR exposed latent test bug)
  b) YDB_ISSUE → TESTOWNERS for broken ydb/... path
  c) Harness itself wrong → introducing PR author team
  d) Pure CI/Actions, no product commit → area/engineering
Map slug → area/... via .github/config/owner_area_mapping.json
Rules: root cause > Owner; commit from A exposing bug in B → B fixes; use area/queryprocessor not area/@ydb-platform/...

Reply exactly:
## Analysis
### Classification:
#### Error type: <TYPE>
#### Severity: <LEVEL>
#### Responsible: area/<team>
### Root Cause
<what broke>
Suspect PR(s), ordered by causality (most likely first):
1. #<pr> — <why>
### Suggested Fix
<fix or investigation>
### Summary
<2-4 sentences: failure, root cause, who fixes, next step>

Links to recent failed test runs

test	status	type	last_run	history	run	stderr	stdout	log	logsdir
test_old_secrets_usage.py.test_existing_objects_are_ok_if_old_secret_creation_is_disabled	failure	N/A	2026-07-27 13:49:01 UTC	link	link	n/a	n/a	link	link

[robot-vibe-db]

Analysis

Classification:

Error type: TEST_ISSUE

Severity: LOW

Responsible: area/security

Root Cause

The test fails with HTTP 500 Internal Server Error when calling viewer/json/describe on /Root/eds-old-secret immediately after cluster restart. The assert_path_ready method (line ~320 in the test file) performs a single HTTP GET to the viewer/monitoring endpoint without any retry/wait logic:

def assert_path_ready(self, path):
    description = self.viewer_describe(path)
    assert description["Status"] == "StatusSuccess", description
    assert description["PathDescription"]["Self"]["CreateFinished"] is True, description
    return description

After restart_cluster, the gRPC driver becomes ready (with a 120s wait), but the HTTP viewer/monitoring subsystem may still be initializing — returning 500 before it finishes loading scheme object descriptions. This is a classic race condition: the driver readiness does not guarantee viewer endpoint readiness.

Notably, the same test file already implements the correct pattern in wait_object_error_state, which uses wait_for(ready, timeout_seconds=60, step_seconds=1) with retries. The assert_path_ready method lacks this retry logic.

The 77.8% success rate (7/9 runs) is consistent with a timing-dependent race — most of the time the viewer endpoint initializes fast enough, but occasionally it doesn't.

Suspect PR(s), ordered by causality (most likely first):

1. More support for disabling old secrets creation and usage #47612 — Introduced assert_path_ready, viewer_describe, and the post-restart viewer endpoint checks in test_existing_objects_are_ok_if_old_secret_creation_is_disabled. This PR significantly expanded the test to create replication/transfer objects and verify them via viewer HTTP endpoints after cluster restart, but did not add retry logic for these calls.

Suggested Fix

Wrap the assert_path_ready logic in a wait_for retry loop, similar to the existing wait_object_error_state pattern. For example:

def assert_path_ready(self, path):
    def ready():
        try:
            description = self.viewer_describe(path)
        except requests.exceptions.RequestException:
            return False
        if description.get("Status") != "StatusSuccess":
            return False
        if not description.get("PathDescription", {}).get("Self", {}).get("CreateFinished"):
            return False
        return True

    if not wait_for(ready, timeout_seconds=60, step_seconds=1):
        description = self.viewer_describe(path)
        raise AssertionError(
            f"Object {path} didn't become ready, got: {description}"
        )

This catches transient HTTP 500 errors during cluster startup and retries until the viewer endpoint is fully initialized. The same fix should be applied defensively for test_existing_objects_can_migrate_to_new_secrets_if_old_secret_creation_is_disabled, which also calls assert_path_ready after restart_cluster.

Summary

The flaky test is caused by a race condition in assert_path_ready introduced in PR #47612 — it queries the viewer HTTP endpoint immediately after cluster restart without retry logic, but the viewer subsystem may still be initializing (returning HTTP 500) even though the gRPC driver is ready. This is a test-level timing issue, not a product bug. The fix is to add wait_for retry logic to assert_path_ready, following the pattern already used by wait_object_error_state in the same file. The security team (TESTOWNERS for /ydb/tests/functional/secrets) should apply this fix.

[robot-vibe-db]

Full analysis log

Analysis performed by claude, claude-opus-4-6.