Skip to content

Add CRR cascaded tests #2447

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
SylvainSenechal wants to merge 2 commits into
base: development/2.16
Choose a base branch
from
Open

Add CRR cascaded tests #2447

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
50ca7f7
move replication tests into their own folder
SylvainSenechal Jun 24, 2026
d87c34e
Add tests for Crr Cascade feature
SylvainSenechal Jun 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/scripts/end2end/configure-e2e.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,12 @@ CRR_SOURCE_LOCATION_NAME=${CRR_SOURCE_LOCATION_NAME} \
CRR_DESTINATION_LOCATION_NAME=${CRR_DESTINATION_LOCATION_NAME} \
CRR_SOURCE_ACCOUNT_NAME=${CRR_SOURCE_ACCOUNT_NAME} \
CRR_DESTINATION_ACCOUNT_NAME=${CRR_DESTINATION_ACCOUNT_NAME} \
CRR_LOCATION_A_NAME=${CRR_LOCATION_A_NAME} \
CRR_LOCATION_B_NAME=${CRR_LOCATION_B_NAME} \
CRR_LOCATION_C_NAME=${CRR_LOCATION_C_NAME} \
CRR_ACCOUNT_A_NAME=${CRR_ACCOUNT_A_NAME} \
CRR_ACCOUNT_B_NAME=${CRR_ACCOUNT_B_NAME} \
CRR_ACCOUNT_C_NAME=${CRR_ACCOUNT_C_NAME} \
CRR_ROLE_NAME=${CRR_ROLE_NAME} \
DEPLOY_CRR_LOCATIONS=${DEPLOY_CRR_LOCATIONS} \
yarn --cwd "$FUNCTIONAL_TESTS_DIR" initializeFunctionalTestsResources
Expand Down
29 changes: 29 additions & 0 deletions .github/scripts/end2end/setup-e2e-env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,9 @@ export ZENKO_SESSION_TOKEN=$(kubectl get secret ${ZENKO_NAME}-account-zenko -o j
# CRR account credentials
_src_secret="${ZENKO_NAME}-account-${CRR_SOURCE_ACCOUNT_NAME:-crr-source-account}"
_dst_secret="${ZENKO_NAME}-account-${CRR_DESTINATION_ACCOUNT_NAME:-crr-destination-account}"
_crr_a_secret="${ZENKO_NAME}-account-${CRR_ACCOUNT_A_NAME:-crr-account-a}"
_crr_b_secret="${ZENKO_NAME}-account-${CRR_ACCOUNT_B_NAME:-crr-account-b}"
_crr_c_secret="${ZENKO_NAME}-account-${CRR_ACCOUNT_C_NAME:-crr-account-c}"

SOURCE_ACCESS_KEY=$(kubectl get secret "${_src_secret}" -o jsonpath='{.data.AccessKeyId}' | base64 -d)
SOURCE_SECRET_KEY=$(kubectl get secret "${_src_secret}" -o jsonpath='{.data.SecretAccessKey}' | base64 -d)
Expand All @@ -68,6 +71,24 @@ DESTINATION_SESSION_TOKEN=$(kubectl get secret "${_dst_secret}" -o jsonpath='{.d
DESTINATION_ACCOUNT_ID=$(kubectl get secret "${_dst_secret}" -o jsonpath='{.data.AccountId}' | base64 -d)
export CRR_DESTINATION_INFO="{\"AccessKeyId\":\"${DESTINATION_ACCESS_KEY}\",\"SecretAccessKey\":\"${DESTINATION_SECRET_KEY}\",\"SessionToken\":\"${DESTINATION_SESSION_TOKEN}\",\"AccountId\":\"${DESTINATION_ACCOUNT_ID}\"}"

CRR_A_ACCESS_KEY=$(kubectl get secret "${_crr_a_secret}" -o jsonpath='{.data.AccessKeyId}' | base64 -d)
CRR_A_SECRET_KEY=$(kubectl get secret "${_crr_a_secret}" -o jsonpath='{.data.SecretAccessKey}' | base64 -d)
CRR_A_SESSION_TOKEN=$(kubectl get secret "${_crr_a_secret}" -o jsonpath='{.data.SessionToken}' | base64 -d)
CRR_A_ACCOUNT_ID=$(kubectl get secret "${_crr_a_secret}" -o jsonpath='{.data.AccountId}' | base64 -d)
export CRR_INFO_A="{\"AccessKeyId\":\"${CRR_A_ACCESS_KEY}\",\"SecretAccessKey\":\"${CRR_A_SECRET_KEY}\",\"SessionToken\":\"${CRR_A_SESSION_TOKEN}\",\"AccountId\":\"${CRR_A_ACCOUNT_ID}\"}"

CRR_B_ACCESS_KEY=$(kubectl get secret "${_crr_b_secret}" -o jsonpath='{.data.AccessKeyId}' | base64 -d)
CRR_B_SECRET_KEY=$(kubectl get secret "${_crr_b_secret}" -o jsonpath='{.data.SecretAccessKey}' | base64 -d)
CRR_B_SESSION_TOKEN=$(kubectl get secret "${_crr_b_secret}" -o jsonpath='{.data.SessionToken}' | base64 -d)
CRR_B_ACCOUNT_ID=$(kubectl get secret "${_crr_b_secret}" -o jsonpath='{.data.AccountId}' | base64 -d)
export CRR_INFO_B="{\"AccessKeyId\":\"${CRR_B_ACCESS_KEY}\",\"SecretAccessKey\":\"${CRR_B_SECRET_KEY}\",\"SessionToken\":\"${CRR_B_SESSION_TOKEN}\",\"AccountId\":\"${CRR_B_ACCOUNT_ID}\"}"

CRR_C_ACCESS_KEY=$(kubectl get secret "${_crr_c_secret}" -o jsonpath='{.data.AccessKeyId}' | base64 -d)
CRR_C_SECRET_KEY=$(kubectl get secret "${_crr_c_secret}" -o jsonpath='{.data.SecretAccessKey}' | base64 -d)
CRR_C_SESSION_TOKEN=$(kubectl get secret "${_crr_c_secret}" -o jsonpath='{.data.SessionToken}' | base64 -d)
CRR_C_ACCOUNT_ID=$(kubectl get secret "${_crr_c_secret}" -o jsonpath='{.data.AccountId}' | base64 -d)
export CRR_INFO_C="{\"AccessKeyId\":\"${CRR_C_ACCESS_KEY}\",\"SecretAccessKey\":\"${CRR_C_SECRET_KEY}\",\"SessionToken\":\"${CRR_C_SESSION_TOKEN}\",\"AccountId\":\"${CRR_C_ACCOUNT_ID}\"}"

# --- 6. Keycloak test user ---
export KEYCLOAK_TEST_USER="${OIDC_USERNAME}-norights"
export KEYCLOAK_TEST_PASSWORD=${OIDC_PASSWORD}
Expand Down Expand Up @@ -103,6 +124,8 @@ export RING_S3C_BACKEND_SOURCE_NON_VERSIONED_LOCATION
export RING_S3C_INGESTION_SRC_NON_VERSIONED_BUCKET_NAME
export RING_S3C_INGESTION_NON_VERSIONED_OBJECT_COUNT_PER_TYPE
export CRR_SOURCE_LOCATION_NAME CRR_DESTINATION_LOCATION_NAME CRR_ROLE_NAME
export CRR_LOCATION_A_NAME CRR_LOCATION_B_NAME CRR_LOCATION_C_NAME
export CRR_INFO_A CRR_INFO_B CRR_INFO_C
export MOCHA_FILE=${MOCHA_FILE:-_reports/test-results-[hash].xml}

# Ensure test results dir exists for Mocha JUnit reporter
Expand Down Expand Up @@ -399,8 +422,14 @@ if [ -n "${GITHUB_ENV:-}" ]; then # Don't do it for Codespace
echo "RING_S3C_ENDPOINT=$RING_S3C_ENDPOINT" >> "$GITHUB_ENV"
echo "CRR_SOURCE_LOCATION_NAME=$CRR_SOURCE_LOCATION_NAME" >> "$GITHUB_ENV"
echo "CRR_DESTINATION_LOCATION_NAME=$CRR_DESTINATION_LOCATION_NAME" >> "$GITHUB_ENV"
echo "CRR_LOCATION_A_NAME=$CRR_LOCATION_A_NAME" >> "$GITHUB_ENV"
echo "CRR_LOCATION_B_NAME=$CRR_LOCATION_B_NAME" >> "$GITHUB_ENV"
echo "CRR_LOCATION_C_NAME=$CRR_LOCATION_C_NAME" >> "$GITHUB_ENV"
echo "CRR_ROLE_NAME=$CRR_ROLE_NAME" >> "$GITHUB_ENV"
# JSON vars need the heredoc delimiter syntax (values contain '=')
printf 'CRR_SOURCE_INFO<<EOF\n%s\nEOF\n' "$CRR_SOURCE_INFO" >> "$GITHUB_ENV"
printf 'CRR_DESTINATION_INFO<<EOF\n%s\nEOF\n' "$CRR_DESTINATION_INFO" >> "$GITHUB_ENV"
printf 'CRR_INFO_A<<EOF\n%s\nEOF\n' "$CRR_INFO_A" >> "$GITHUB_ENV"
printf 'CRR_INFO_B<<EOF\n%s\nEOF\n' "$CRR_INFO_B" >> "$GITHUB_ENV"
printf 'CRR_INFO_C<<EOF\n%s\nEOF\n' "$CRR_INFO_C" >> "$GITHUB_ENV"
fi
6 changes: 6 additions & 0 deletions .github/workflows/end2end.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,12 @@ env:
CRR_DESTINATION_LOCATION_NAME: crr-destination-location
CRR_SOURCE_ACCOUNT_NAME: crr-source-account
CRR_DESTINATION_ACCOUNT_NAME: crr-destination-account
CRR_LOCATION_A_NAME: crr-location-a
CRR_LOCATION_B_NAME: crr-location-b
CRR_LOCATION_C_NAME: crr-location-c
CRR_ACCOUNT_A_NAME: crr-account-a
CRR_ACCOUNT_B_NAME: crr-account-b
CRR_ACCOUNT_C_NAME: crr-account-c
CRR_ROLE_NAME: crr-role
# Enable this for Ring tests
ENABLE_RING_TESTS: "false"
Expand Down
56 changes: 56 additions & 0 deletions tests/functional/ctst/features/replication/crrCascade.feature
View file
Open in desktop

@SylvainSenechal SylvainSenechal Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Realising here I have 3 scenarios, and each new scenario pretty much covers the case of the previous scenario so in theory we could just run the last one to test the feature, I think still good to have a few scenarios

Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
Feature: CRR Cascade Replication

Cross-Region Replication Cascade allows replication of an object from
a first location to carry on through multiple locations without
stopping at first destination.

@2.15.0

@delthas delthas Jul 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All those new tests should be marked 2.16 since that's where CRR will land.

@PreMerge
@ReplicationTest
@CRRCascade
Scenario: Cascade replication nominal case : A -> B -> C
Given cascade replication accounts are registered
And a versioned bucket exists in location "crr-location-a"
And a versioned bucket exists in location "crr-location-b"
And a versioned bucket exists in location "crr-location-c"
And replication is configured from location "crr-location-b" to "crr-location-c"
And replication is configured from location "crr-location-a" to "crr-location-b"
When an object "cascade-obj" is put in location "crr-location-a"
Then the object should replicate to location "crr-location-b" within 300 seconds
And the object should replicate to location "crr-location-c" within 300 seconds

@francoisferrand francoisferrand Jun 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should also check the "final" replication status on each location : SUCCESS / REPLICA / REPLICA ?

@SylvainSenechal SylvainSenechal Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I'm gonna add a step at the end to check all the status

When I wait 15 seconds
Then the cascade replication states should be settled
Comment on lines +21 to +22

@francoisferrand francoisferrand Jul 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead/in addition to verify things are "settled" (which may pass by luck, in a dynamic system) : should/coudl we try to "count" replications? i.e. use metrics/logs/kafka topic to verify no extra replication was triggered...

(not sure if possible, i.e. how to observe this specifically when other tests may be running...)


@2.15.0
@PreMerge
@ReplicationTest
@CRRCascade
Scenario: Cascade replication with loop : A -> B -> C -> A
Given cascade replication accounts are registered
And a versioned bucket exists in location "crr-location-a"
And a versioned bucket exists in location "crr-location-b"
And a versioned bucket exists in location "crr-location-c"
And replication is configured from location "crr-location-a" to "crr-location-b"
And replication is configured from location "crr-location-b" to "crr-location-c"
And replication is configured from location "crr-location-c" to "crr-location-a"
When an object "cascade-loop-obj" is put in location "crr-location-a"
Then the object should replicate to location "crr-location-b" within 300 seconds
And the object should replicate to location "crr-location-c" within 300 seconds
And the object at location "crr-location-a" should never have replication status PENDING within 30 seconds

@francoisferrand francoisferrand Jun 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should also check the "final" replication status on each location : SUCCESS / REPLICA / REPLICA ?

@francoisferrand francoisferrand Jun 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we have more complex scenario, where we write on multiple points in the loop?
may not be able to control how he operations race to ensure a specific scenario, but worth adding to ensure the system is "eventually" consistent and stable?

@SylvainSenechal SylvainSenechal Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I came up with a more complex scenario which I think is decent :

  • Loop config
  • Concurrent writes on all location, writing a specific random metadata marker
    => Assert that they all converge on the same metadataMarker

Tbh, still not the perfect test but I think worth doing

When I wait 15 seconds
Then the cascade replication states should be settled

@2.15.0
@PreMerge
@ReplicationTest
@CRRCascade
Scenario: Multiple writes across a loop converge to the last write
Given cascade replication accounts are registered
And a versioned bucket exists in location "crr-location-a"
And a versioned bucket exists in location "crr-location-b"
And a versioned bucket exists in location "crr-location-c"
And replication is configured from location "crr-location-b" to "crr-location-c"
And replication is configured from location "crr-location-c" to "crr-location-a"
And replication is configured from location "crr-location-a" to "crr-location-b"
When the object "cascade-convergence-obj" is concurrently written 10 times to every cascade location
Then all cascade locations should converge to the same metadata marker within 300 seconds
0 ...st/features/crrReplicationS3utils.feature → ...replication/crrReplicationS3utils.feature
View file
Open in desktop
File renamed without changes.
0 ...ctional/ctst/features/replication.feature → .../features/replication/replication.feature
View file
Open in desktop
File renamed without changes.
Loading
Loading