NKikimr::TBlobStorageGroupInfo::Parse

[maximyurchuk]

0 EscapeC(TBasicString<char, std::__y1::char_traits<char>> const&) () util/generic/ptr.h +592
1 NKikimr::TBlobStorageGroupInfo::Parse(NKikimrBlobStorage::TGroupInfo const&, NKikimr::TEncryptionKey const*, IOutputStream*) () contrib/ydb/core/blobstorage/groupinfo/blobstorage_groupinfo.cpp +745
2 NKikimr::NBlobDepot::TBlobDepot::Handle(TAutoPtr<NActors::TEventHandle<NKikimr::TEvBlobDepot::TEvApplyConfig>, TDelete>) () contrib/ydb/core/blob_depot/op_apply_config.cpp +74
3 NKikimr::NBlobDepot::TBlobDepot::StateWork(TAutoPtr<NActors::IEventHandle, TDelete>&) () contrib/ydb/core/blob_depot/blob_depot.cpp +124
4 NActors::IActor::Receive(TAutoPtr<NActors::IEventHandle, TDelete>&) () contrib/ydb/library/actors/core/actor.cpp +394
5 NActors::TExecutorThread::Execute(NActors::TMailbox*, bool, long) () contrib/ydb/library/actors/core/executor_thread.cpp +279
6 NActors::TExecutorThread::ProcessExecutorPool()::$_0::operator()(NActors::TMailbox*, bool, long) const () contrib/ydb/library/actors/core/executor_thread.cpp +468
7 NActors::TExecutorThread::ProcessExecutorPool() () contrib/ydb/library/actors/core/executor_thread.cpp +520
8 NActors::TExecutorThread::ThreadProc() () contrib/ydb/library/actors/core/executor_thread.cpp +546
9 (anonymous namespace)::TPosixThread::ThreadProxy(void*) () util/system/thread.cpp +245
10 ?? () libpthread.so.0
11 ?? () libc.so.6

see http://st/KIKIMR-26410

[maximyurchuk]

Posted by an AI agent (Claude Code) on behalf of @maximyurchuk. The analysis below is static code reading against upstream/main @ dc4bf7f plus branch-containment checks; it has not been reproduced in a test. Please verify before acting.

Root cause: nullptr dereference in TBlobStorageGroupInfo::Parse while formatting the error message

Crash site

ydb/core/blobstorage/groupinfo/blobstorage_groupinfo.cpp#L744-L747 — matches frame 1 of the stack (+745):

if (!key || !*key) {
    lcp = ELCP_KEY_NOT_LOADED;      // <-- key == nullptr lands here
} else if (mainKeyId != key->Id) {
    ...
}
if (err) {
    *err << "LifeCyclePhase# " << lcp << " Key.Id# \"" << EscapeC(key->Id)   // <-- L745: key is nullptr
        << "\" Key.Version# " << key->Version << " MainKey.Id# \"" << EscapeC(mainKeyId)
        << ...;
}

The !key branch handles a missing key correctly, but the if (err) block that follows reads key->Id and key->Version unconditionally. Id is a TString at a non-zero offset inside TEncryptionKey, so the fault happens inside EscapeC when it reads the string's data pointer — hence frame 0 EscapeC(...) at util/generic/ptr.h +592 (inlined TString internals).

Trigger conditions, all required: EncryptionMode != EEM_NONE and LifeCyclePhase == ELCP_IN_USE and key == nullptr and err != nullptr. The stack alone proves the group was encrypted and IN_USE, since L745 is the only EscapeC call in Parse reachable from that path.

This is not debug-only: the segfault precedes the Y_DEBUG_ABORT_UNLESS and happens in release builds too.

Why it only showed up on 26-3

Parse is byte-for-byte identical in stable-26-2-1 and stable-26-3-1 — the latent bug is old. What changed is the caller: 26-3 introduced the first call site with exactly that argument combination.

Commit 6340984ef68 — "Add support for recommissioning in BlobDepot (WIP)" (#35185), 2026-03-03. Containment: stable-26-2 no, stable-26-2-1 no, prestable-26-3 / stable-26-3 / stable-26-3-1 / main yes.

It added both halves of the trigger:

1. BSC now serializes GroupInfo into TEvApplyConfig — virtual_group.cpp#L896:

SerializeGroupInfo(ev->Record.MutableGroupInfo(), *group, Self->StoragePools);

and SerializeGroupInfo sets EncryptionMode / LifeCyclePhase / MainKeyId unconditionally (config.cpp#L1316-L1321).

2. BlobDepot now parses it — op_apply_config.cpp#L72-L76:

if (record.HasGroupInfo()) {
    TStringStream err;
    GroupInfo = TBlobStorageGroupInfo::Parse(record.GetGroupInfo(), nullptr, &err);  // key=nullptr, err!=nullptr
    Y_DEBUG_ABORT_UNLESS(GroupInfo);
}

Before 26-3, HasGroupInfo() was always false, so Parse was never called here.

Failure chain

BSC (26-3) sends TEvApplyConfig carrying the GroupInfo of an encrypted virtual group → the BlobDepot tablet parses it without a key → SIGSEGV kills the whole dynamic node process, not just the tablet → every other tablet on that node dies with it (DataShard, ColumnShard, PersQueue, …) → the node restarts, BSC re-sends TEvApplyConfig, and it crashes again. That produces the observed tight crash loop (hundreds of restarts within minutes) and the cascade of "system tablet is unresponsive" across affected databases. Only databases whose BlobDepot group sits on an encrypted storage pool are affected, which is why the blast radius is a subset of the cluster.

The parsed value is never used

TBlobDepot::GroupInfo (blob_depot_tablet.h#L274) is only ever assigned at op_apply_config.cpp:74 and checked in the debug assert — it is read nowhere else. It is groundwork from the WIP recommissioning PR, so the call that is crashing production carries no functional value in 26-3.

Same latent bug at two other call sites

Both pass key = nullptr with a non-null err and would crash on an encrypted group in ELCP_IN_USE:

• ydb/core/mind/bscontroller/impl.h#L2541 — TStaticGroupInfo, inside BSC itself
• ydb/core/blobstorage/nodewarden/distconf_invoke_static_group.cpp#L47

Static groups are normally unencrypted, which is why these have never fired.

Suggested fix

Hotfix for 26-3 (stops the crash loop): drop the if (record.HasGroupInfo()) block in op_apply_config.cpp — it is dead code. Minimal alternative, if the block should stay: pass nullptr instead of &err.

Root fix (main + backport): make the error formatting in Parse null-safe — when !key, print something like Key# nullptr instead of touching key->Id / key->Version. This also covers the BSC and distconf call sites. The bug is present unchanged in main.

[maximyurchuk]

#35185 (review) -- вот тут какие-то проблемы, может релейтед

[mregrock]

#48076
#48077

[serbel324]

both PRs are merged