Description
Our exactly-once coverage in ExactlyOnceTest.java is thin. The happy-path tests (checkTotalsEqual) just verify totals with no faults, and the only fault we inject (checkSpottyNetwork) is a full ClickHouse Cloud service restart via the Cloud API - which is coarse, slow, flaky, and Cloud-only. It doesn't exercise the failure modes that actually stress the exactly-once state machine: a component dying in the middle of a batch, between the BEFORE and AFTER Keeper flags.
We should build a proper chaos test that repeatedly injects mid-processing failures while data is flowing, then asserts on resume that we land exactly once (no duplicates, no loss — i.e. count() == uniqExact() and total matches what was produced). Specifically, it should cover:
- Killing ClickHouse mid-insert — kill/restart individual cluster nodes (ch0/ch1/ch2, including the node hosting Keeper state) while inserts are in flight, against the local Docker cluster from
ClickHouseCluster, not just a Cloud service restart. This validates that we reconstruct the identical block and dedup handles the retry, and that Keeper state survives a node going down.
- Killing the Confluent/Connect instance mid-processing — kill the Connect worker (or the task) while a batch is being processed, before offsets/flags are committed, then bring it back and confirm it resumes from a consistent Keeper state with no dupes or loss. Today, we only do a graceful restart of the Connector; we want a hard kill mid-flight.
Ideally, these run as a randomized/looping "monkey" that injects faults at random points across many rounds, with enough records in flight that failures land mid-batch. It should run against the local Docker cluster so it's part of normal CI rather than requiring Cloud credentials.
This is closely related to #742 (designing a better, non-Cloud-restart way to simulate network interruptions — e.g. Toxiproxy, which we already use in ClickHouseSinkConnectorIntegrationTest). The two should share the same fault-injection approach: #742 covers network blips, and this covers hard component kills. Whatever helper we land for one should be reusable for the other.
Description
Our exactly-once coverage in
ExactlyOnceTest.javais thin. The happy-path tests (checkTotalsEqual) just verify totals with no faults, and the only fault we inject (checkSpottyNetwork) is a full ClickHouse Cloud service restart via the Cloud API - which is coarse, slow, flaky, and Cloud-only. It doesn't exercise the failure modes that actually stress the exactly-once state machine: a component dying in the middle of a batch, between the BEFORE and AFTER Keeper flags.We should build a proper chaos test that repeatedly injects mid-processing failures while data is flowing, then asserts on resume that we land exactly once (no duplicates, no loss — i.e. count() == uniqExact() and total matches what was produced). Specifically, it should cover:
ClickHouseCluster, not just a Cloud service restart. This validates that we reconstruct the identical block and dedup handles the retry, and that Keeper state survives a node going down.Ideally, these run as a randomized/looping "monkey" that injects faults at random points across many rounds, with enough records in flight that failures land mid-batch. It should run against the local Docker cluster so it's part of normal CI rather than requiring Cloud credentials.
This is closely related to #742 (designing a better, non-Cloud-restart way to simulate network interruptions — e.g. Toxiproxy, which we already use in
ClickHouseSinkConnectorIntegrationTest). The two should share the same fault-injection approach: #742 covers network blips, and this covers hard component kills. Whatever helper we land for one should be reusable for the other.