[Bug] Copying Admin Storage Prevents Retention of Message Counter State

[zaid-google]

Describe the bug

This issue is a problem with regards to how the test harness manages its admin_storage.json file, and as a result its message counter data. The problem was found when running the tests mentioned below.

This was found when running TC-ACE-1.6 and TC-SC-5.2 on the same commissioning window (i.e. run one of them with commissioning, and then the other without a factory reset). This can also happen when either of these 2 tests are run 2 times in a row (again, with no factory reset).

The tests fails on the second run, because of unexpected values for the event checks. In step 20a and 16b of TC_ACE_1.6 and TC_SC_5.2 respectively, the test expects certain values for a groupcast testing event, but instead a message replay event is sent back. This happens because of how the test harness is handling admin_storage.json between test runs.

The following code is called on the first test run:
https://github.com/project-chip/certification-tool-backend/blob/main/test_collections/matter/sdk_tests/support/python_testing/models/utils.py#L202

And then this code is called on the second test run:
https://github.com/project-chip/certification-tool-backend/blob/main/test_collections/matter/sdk_tests/support/python_testing/models/utils.py#L303
Showing that it will copy a version of the admin_storage.json file to the container. The admin_storage.json file is snapshotted once after commissioning then copied to the container before every test where commissioning is not re-done. This file contains information on the state of the message counter to determine what the next valid message counter should be for an outgoing message. These message counters are supposed to be monotonically increasing, as when the counters are repeated the message is marked as a duplicate. Keys such as g/gdc and g/gcc which are required for these monotonically increasing message counters are reset backwards, which then causes message replays. At the end of the first test run, the updated values in admin_storage.json are not brought back before the second test run.

Before running either TC_ACE_1.6 or TC_SC_5.2, the DUT MUST be factory reset in order to avoid this problem. Otherwise, these tests may fail due to message duplication causing different event data than expected.

Steps to reproduce the behavior

No response

Expected behavior

No response

Log files

No response

PICS file

No response

Screenshots

No response

Environment

No response

Additional Information

No response

[antonio-amjr]

Hi @zaid-google,

The diagnostic from the description as the storage is not saving the counter state makes sense.
However, I'm not able to verify here a solution I implemented because I'm having a hard time running these group cast steps
I'm running this scenario with the Matter 1.6 MVE release (v2.15.1-beta1+summer2026) and using the sample app chip-all-clusters-app as DUT, but I couldn't reproduce this probably because of some configuration since a lot of steps are skipped as Not-Applicable.

Can you do me a favor @zaid-google and try the following Backend branch from my fork and verify if the solution works?
The branch is the following: https://github.com/antonio-amjr/certification-tool-backend/tree/fix/reuse_of_admin_storage.

If that doesn't work, please walk me through on which app (if possible with sample apps) and how to config and reproduce this problem.
Also, please share the Test Harness logs, and system environment information (e.g. running and sharing the output of $ /home/ubuntu/certification-tool/scripts/th/th-doctor.sh --complete).

[zaid-google]

Hi @antonio-amjr ,

To reproduce the issue, it would be better if you can use the all-devices as opposed to all-clusters. You can start the app with the following app arguments: --groupcast --device on-off-light:1. Be sure to then run the tests (TC_ACE_1_6 or TC_SC_5_2) with the endpoint argument being endpoint 1 (as the all devices app will start as an on/off light on endpoint 1). I've attached the logs from running TC_ACE_1_6 2 times (as described in the issue), where the first run is passing and the second run is failing. You can see in these logs repeated message counters between the 2 runs. I also added the system information logs from the test harness you requested.

I'm a bit confused by the branch you sent, it shows it's 100+ commits behind master, should this not be built on top of the latest changes? Please try running the tests again with the all devices app and let me know if you can reproduce

th_system_info.log
test_run_ace_1_6_first_run_2026-07-29T09-32-40.log
test_run_ace_1_6_rerun_2026-07-29T09-33-58.log

[antonio-amjr]

Hi @zaid-google

To reproduce the issue, it would be better if you can use the all-devices as opposed to all-clusters. You can start the app with the following app arguments: --groupcast --device on-off-light:1. Be sure to then run the tests (TC_ACE_1_6 or TC_SC_5_2) with the endpoint argument being endpoint 1 (as the all devices app will start as an on/off light on endpoint 1). I've attached the logs from running TC_ACE_1_6 2 times (as described in the issue), where the first run is passing and the second run is failing. You can see in these logs repeated message counters between the 2 runs. I also added the system information logs from the test harness you requested.

Thanks for those informations and the app suggestion. I'll try later again soon with all-devices and the configurations you mentioned.

---

I'm a bit confused by the branch you sent, it shows it's 100+ commits behind master, should this not be built on top of the latest changes? Please try running the tests again with the all devices app and let me know if you can reproduce

Sorry, I'm confused about this branch comment as well, I'm not sure why the master comparison, since there's no master at Backend repo, only main (maybe looking at SDK repo?). Please let me clarify to confirm we're at the same page.

Looking at your th_system_info.log file I confirmed you're using the TH v2.15.1-beta1+summer2026 version, that points to the Backend with HASH cb44885. If you verify the commits from the branch I shared, you'll see as the below image:

By the image we see my commit is in the top of the release commit.

So in this case you should try it out stoping the Backend container (you could use the script /home/ubuntu/certification-tool/scripts/stop.sh), update the Backend with my change (you could use a cherry-pick the hash f12de50a9f232dc546a87da4729a6092045ba2c0 from my fork or any other method), then start the containers again (/home/ubuntu/certification-tool/scripts/start.sh`).

Let me know if that helps or if I misunderstood anything.

[Pradeep-grl]

@zaid-google , Could you please check and share the feedback? We need to fix the issue before August 14, 2026.

[antonio-amjr]

Using the above suggested sample app (all-devices) along with the arguments of group cast I was able to reproduce this behavior. By reusing the commissioning window of the all-device from the former execution, the GroupcastTesting event fails.

I'm trying out against my possible solution shared before. If positive I'll make the PR in sequence and share here.