test(cluster): fix flaky barman-plugin-migration upgrading assertion#1
Open
aqeelat wants to merge 40 commits into
Open
test(cluster): fix flaky barman-plugin-migration upgrading assertion#1aqeelat wants to merge 40 commits into
aqeelat wants to merge 40 commits into
Conversation
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
…uster Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
…only Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
…ions Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
…ing assertion The 'Switch the cluster to Barman CNPG-I' step ran 'helm upgrade --wait' then asserted the transient 'Upgrading cluster' phase. Because --wait blocks until the release has fully converged, the rolling upgrade is already complete by the time the assertion polls, so status.phase is back to 'Cluster in healthy state' and the assert times out. Dropping --wait lets the assertion observe the 'Upgrading cluster' window (the plugin sidecar addition forces a rolling restart), while the subsequent healthy assertion still gates the next step.
The migration from in-tree barmanObjectStore to the CNPG-I plugin triggers a rolling upgrade, but status.phase 'Upgrading cluster' is only set transiently (sub-second) before transitioning to 'Waiting for the instances to become active'. Asserting it after 'helm upgrade --wait' (or a sleep) always misses the window, so the assertion never passed. The operator's own e2e tests model rollouts as a phase sequence (PhaseUpgrade -> PhaseWaitingForInstancesToBeActive) and emit an 'UpgradingInstance' event per rolled instance (cluster_upgrade.go:upgradePod). Verify the rollout deterministically by checking the persistent UpgradingInstance event after the upgrade completes, instead of polling a transient phase.
…heck helm upgrade --wait returns immediately (~0.5s) for CNPG Cluster CRs because helm cannot assess CRD readiness; the operator reconciles asynchronously. The previous one-shot kubectl events check ran before the rollout started, so it never found the event. Use a chainsaw 'assert' on the Event resource instead. Chainsaw's assert polls until the UpgradingInstance event is emitted (proving the rollout started), then the existing healthy assertion polls until readyInstances=2 (proving it finished). This makes the rollout detection deterministic regardless of operator reconcile latency.
The post-migration backup completes in ~10s on CI, faster than chainsaw can poll, so the 'status.phase: running' assertion always timed out (the backup was already 'completed'). This is the same transient-state class of failure as the migration phase assertion. The 'running' check is redundant with the 'completed' assertion in the next step (which also covers the backup-failed case), so remove it. The backup's success is verified by the 'completed' assertion after the checkpoint.
The data-test job referenced 'recovery-backup-pitr-superuser' (the helm release name) but CNPG names the superuser secret after the cluster, 'recovery-backup-pitr-cluster-superuser'. The kubelet couldn't find the secret and the pod stayed in CreateContainerConfigError. This matches the working data-write job, which correctly references 'barman-plugin-migration-cluster-superuser'.
Keep coverage instead of removing the assertion. The post-migration backup completes in ~10s, so 'status.phase: running' was never observable via polling. Replace it with a polling assert on the persistent backup 'Starting' Event (backup_controller.go:337 emits it on the Backup resource), mirroring the migration step's UpgradingInstance event approach.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI test run for a proposed fix to cloudnative-pg#924.
Drops
--waitfrom the migrationhelm upgradeso the transientUpgrading clusterphase is observable. See upstream PR cloudnative-pg#924.Throwaway repo — will be deleted after the run.