Skip to content

Add Cucumber tests for multi-destination CRR #2451

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
maeldonn wants to merge 1 commit into
base: improvement/ZENKO-5263
Choose a base branch
from
Open

Add Cucumber tests for multi-destination CRR #2451

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion tests/functional/ctst/common/common.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ export async function cleanS3Bucket(
world.deleteKeyFromCommand('key');
world.deleteKeyFromCommand('versionId');
}
await S3.deleteBucketReplication(world.getCommandParameters()).catch(() => {});
await S3.deleteBucketLifecycle(world.getCommandParameters());
await S3.deleteBucket(world.getCommandParameters());
}
Expand Down Expand Up @@ -274,7 +275,7 @@ Given('a transition workflow to {string} location', async function (this: Zenko,

Given('a replication configuration to {string} location',
async function (this: Zenko, replicationLocation: string) {
this.addToSaved('replicationLocation', replicationLocation);
this.addToSaved('replicationLocations', [replicationLocation]);
await putBucketReplication.call(this, this.getSaved<string>('bucketName'), replicationLocation);
});

Expand Down
6 changes: 3 additions & 3 deletions tests/functional/ctst/features/replication/crrCascade.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Feature: CRR Cascade Replication
@ReplicationTest
@CRRCascade
Scenario: Cascade replication nominal case : A -> B -> C
Given cascade replication accounts are registered
Given CRR replication accounts are registered
And a versioned bucket exists in location "crr-location-a"
And a versioned bucket exists in location "crr-location-b"
And a versioned bucket exists in location "crr-location-c"
Expand All @@ -26,7 +26,7 @@ Feature: CRR Cascade Replication
@ReplicationTest
@CRRCascade
Scenario: Cascade replication with loop : A -> B -> C -> A
Given cascade replication accounts are registered
Given CRR replication accounts are registered
And a versioned bucket exists in location "crr-location-a"
And a versioned bucket exists in location "crr-location-b"
And a versioned bucket exists in location "crr-location-c"
Expand All @@ -45,7 +45,7 @@ Feature: CRR Cascade Replication
@ReplicationTest
@CRRCascade
Scenario: Multiple writes across a loop converge to the last write
Given cascade replication accounts are registered
Given CRR replication accounts are registered
And a versioned bucket exists in location "crr-location-a"
And a versioned bucket exists in location "crr-location-b"
And a versioned bucket exists in location "crr-location-c"
Expand Down
4 changes: 2 additions & 2 deletions tests/functional/ctst/features/replication/crrReplicationS3utils.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Feature: Replication
And a replication configuration to "awsbackendmismatch" location
When the job to replicate existing objects with status "NEW" is executed
Then the object replication should "succeed" within 300 seconds
And the replicated object should be the same as the source object
And the replicated object should match the source on every configured destination

@2.12.0
@PreMerge
Expand All @@ -35,4 +35,4 @@ Feature: Replication
When the destination bucket on the location is created again
And the job to replicate existing objects with status "FAILED" is executed
Then the object replication should "succeed" within 300 seconds
And the replicated object should be the same as the source object
And the replicated object should match the source on every configured destination
164 changes: 154 additions & 10 deletions tests/functional/ctst/features/replication/replication.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,26 +1,170 @@
Feature: Live Object Replication
This feature tests live replication of objects across different sizes,
verifying that oplog location stripping does not affect replication
correctness.

The objectRefreshSizeThresholdMB is set to 2 in the Zenko CR, meaning
objects >= 2 MiB have their location stripped from the MongoDB change
stream event. The replication consumer detects this and re-fetches
metadata from the source before replicating.
Feature: Bucket Replication
End-to-end coverage of Zenko's bucket replication, covering both
cloud destinations (AWS / GCP / Azure backends) and CRR loopback
destinations (replication between Zenko accounts).

@2.14.0
@PreMerge
@ReplicationTest
# Targets the location-stripping threshold: objectRefreshSizeThresholdMB
# is set to 2 in the Zenko CR, so objects >= 2 MiB have their location
# stripped from the MongoDB change stream event. The replication consumer
# detects this and re-fetches metadata from the source before replicating.
Scenario Outline: Object of <objectDescription> replicates with location stripping threshold
Given an existing bucket "repl-strip-<sizeBytes>" "with" versioning, "without" ObjectLock "without" retention mode
And a replication configuration to "awsbackendmismatch" location
And 1 objects "repl-strip-obj" of size <sizeBytes> bytes
Then the object replication should "succeed" within 300 seconds
And the replicated object should be the same as the source object
And the replicated object should match the source on every configured destination

Examples:
| sizeBytes | objectDescription |
| 0 | 0 bytes (zero-byte) |
| 1048576 | 1 MiB (below threshold) |
| 2097152 | 2 MiB (at threshold, stripped) |
| 4194304 | 4 MiB (above threshold, stripped) |

@2.16.0
@PreMerge
@ReplicationTest
# Multi-destination replication: a single replication configuration
# carries several rules targeting distinct destinations, optionally with
# overlapping prefixes resolved by Priority. The legacy V1 format and
# the non-standard comma-separated StorageClass form remain supported
# for backward compatibility and round-trip unchanged through
# getBucketReplication.
Scenario: V2 replication fans out to two cloud destinations
Given an existing bucket "multi-cloud-2" "with" versioning, "without" ObjectLock "without" retention mode
And a valid multi-destination replication configuration with rules:
| id | prefix | priority | location |
Comment thread
SylvainSenechal marked this conversation as resolved.
| rule-a | | 1 | awsbackendmismatch |
| rule-b | | 2 | awsbackend |
And 1 objects "multi-cloud-obj" of size 1024 bytes
Then the object replication should "succeed" within 300 seconds
And the replicated object should match the source on every configured destination

@2.16.0
@PreMerge
@ReplicationTest
Scenario: V2 replication fans out to a mix of cloud and CRR destinations
Given CRR replication accounts are registered
And a versioned bucket exists in location "crr-location-a"
And a versioned bucket exists in location "crr-location-b"
And an existing bucket "multi-fanout" "with" versioning, "without" ObjectLock "without" retention mode
And a valid multi-destination replication configuration with rules:
| id | prefix | priority | location |
| rule-a | | 1 | awsbackendmismatch |
| rule-b | | 2 | awsbackend |
| rule-c | | 3 | crr-location-a |
| rule-d | | 4 | crr-location-b |
And 1 objects "multi-fanout-obj" of size 1024 bytes
Then the object replication should "succeed" within 300 seconds
And the replicated object should match the source on every configured destination

@2.16.0
@PreMerge
@ReplicationTest
Scenario: CRR + cloud mixed destination replication
Given CRR replication accounts are registered
And a versioned bucket exists in location "crr-location-a"
And an existing bucket "multi-mixed" "with" versioning, "without" ObjectLock "without" retention mode
And a valid multi-destination replication configuration with rules:
| id | prefix | priority | location |
| rule-c | | 1 | crr-location-a |
| rule-x | | 2 | awsbackendmismatch |
And 1 objects "mixed-obj" of size 1024 bytes
Then the object replication should "succeed" within 300 seconds
And the replicated object should match the source on every configured destination

@2.16.0
@PreMerge
@ReplicationTest
Scenario: V2 rules with overlapping prefixes to different destinations all apply
Given an existing bucket "multi-overlap" "with" versioning, "without" ObjectLock "without" retention mode
And a valid multi-destination replication configuration with rules:
| id | prefix | priority | location |
| broad | | 1 | awsbackendmismatch |
| narrow | docs- | 2 | awsbackend |
And 1 objects "docs-report" of size 1024 bytes
Then the object replication should "succeed" within 300 seconds
And the replicated object should match the source on every configured destination

@2.16.0
@PreMerge
@ReplicationTest
Scenario: Disabled rule does not replicate
Given an existing bucket "multi-disabled" "with" versioning, "without" ObjectLock "without" retention mode
And a valid multi-destination replication configuration with rules:
| id | prefix | priority | status | location |
| active | | 1 | Enabled | awsbackendmismatch |
| off | | 2 | Disabled | awsbackend |
And 1 objects "disabled-obj" of size 1024 bytes
Then the object replication should "succeed" within 300 seconds
And the object should not be replicated to "awsbackend"
And the replicated object on "awsbackendmismatch" should match the source

@2.16.0
@PreMerge
@ReplicationTest
Scenario: Overlap on same destination without distinct priorities is rejected
Given an existing bucket "multi-bad-overlap" "with" versioning, "without" ObjectLock "without" retention mode
And an invalid multi-destination replication configuration with rules:
| id | prefix | priority | location |
| dup-a | | 1 | awsbackendmismatch |
| dup-b | docs-| 1 | awsbackendmismatch |
Then the replication configuration request should be rejected with "InvalidRequest"

@2.16.0
@PreMerge
@ReplicationTest
Scenario: Same destination overlap with distinct priorities is accepted
Given an existing bucket "multi-priority-dedup" "with" versioning, "without" ObjectLock "without" retention mode
And a valid multi-destination replication configuration with rules:
| id | prefix | priority | location |
| lo | | 1 | awsbackendmismatch |
| hi | docs-| 2 | awsbackendmismatch |
And 1 objects "docs-winner" of size 1024 bytes
Then the object replication should "succeed" within 300 seconds
And the replicated object on "awsbackendmismatch" should match the source

@2.16.0
@PreMerge
@ReplicationTest
Scenario: V1 configuration replicates and round-trips through getBucketReplication
Given an existing bucket "multi-v1-compat" "with" versioning, "without" ObjectLock "without" retention mode
And a valid multi-destination replication configuration with rules:
| id | prefix | location |
| v1-only | | awsbackendmismatch |
And 1 objects "v1-obj" of size 1024 bytes
Then the object replication should "succeed" within 300 seconds
And the replicated object on "awsbackendmismatch" should match the source
And getBucketReplication should return a "V1" configuration with 1 rules

@2.16.0
@PreMerge
@ReplicationTest
Scenario: Legacy comma-separated StorageClass replicates to multiple sites
Given an existing bucket "multi-legacy-comma" "with" versioning, "without" ObjectLock "without" retention mode
And a legacy comma-StorageClass replication configuration to locations "awsbackendmismatch,awsbackend"
And 1 objects "legacy-obj" of size 1024 bytes
Then the object replication should "succeed" within 300 seconds
And the replicated object should match the source on every configured destination
And getBucketReplication should return a "V1" configuration with 1 rules
And getBucketReplication rule 0 should have StorageClass "awsbackendmismatch,awsbackend"

@2.16.0
@PreMerge
@ReplicationTest
Scenario: Two CRR destinations replicate independently
Given CRR replication accounts are registered
And a versioned bucket exists in location "crr-location-a"
And a versioned bucket exists in location "crr-location-b"
And an existing bucket "multi-crr-pair" "with" versioning, "without" ObjectLock "without" retention mode
And a valid multi-destination replication configuration with rules:
| id | prefix | priority | location |
| crr-a | | 1 | crr-location-a |
| crr-b | | 2 | crr-location-b |
And 1 objects "crr-pair-obj" of size 1024 bytes
Then the object replication should "succeed" within 300 seconds
And the replicated object on "crr-location-a" should match the source
And the replicated object on "crr-location-b" should match the source
35 changes: 4 additions & 31 deletions tests/functional/ctst/steps/crrCascade.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,46 +11,19 @@ import assert from 'assert';
import { Identity, IdentityEnum, Utils } from 'cli-testing';
import Zenko from 'world/Zenko';

interface CRRAccountInfo {
export interface CRRAccountInfo {
AccessKeyId: string;
SecretAccessKey: string;
SessionToken?: string;
AccountId: string;
}

Given('cascade replication accounts are registered', function (this: Zenko) {
const roleName = process.env.CRR_ROLE_NAME;
assert.ok(roleName, 'CRR_ROLE_NAME must be set');

const locationNames = [
process.env.CRR_LOCATION_A_NAME,
process.env.CRR_LOCATION_B_NAME,
process.env.CRR_LOCATION_C_NAME,
];
assert.ok(locationNames.every(Boolean), 'CRR_LOCATION_A/B/C_NAME must be set');

const infoByLocation: Record<string, CRRAccountInfo> = {};
for (const [i, loc] of locationNames.entries()) {
const varName = `CRR_INFO_${['A', 'B', 'C'][i]}`;
const raw = process.env[varName];
assert.ok(raw, `${varName} must be set`);
const info = JSON.parse(raw) as CRRAccountInfo;
Identity.addIdentity(IdentityEnum.ACCOUNT, loc!, {
accessKeyId: info.AccessKeyId,
secretAccessKey: info.SecretAccessKey,
sessionToken: info.SessionToken,
});
infoByLocation[loc!] = info;
}

this.addToSaved('cascadeInfoByLocation', infoByLocation);
this.addToSaved('cascadeRoleName', roleName);
this.addToSaved('cascadeBuckets', {} as Record<string, string>);
});

Given('a versioned bucket exists in location {string}', async function (this: Zenko, location: string) {
const bucket = `cascade-${Utils.randomString().toLowerCase()}`;
Identity.useIdentity(IdentityEnum.ACCOUNT, location);
// Persist the identity so the default After-hook cleanup targets the
// same account that owns the buckets created here (and by later steps).
this.addToSaved('accountName', location);
const client = this.createS3Client();
await client.send(new CreateBucketCommand({ Bucket: bucket }));
await client.send(new PutBucketVersioningCommand({
Expand Down
Loading
Loading