Replicate Existing Objects in your Amazon S3 Buckets with Amazon S3 Batch Replication

2.1 From your list of S3 buckets, choose your source S3 bucket. The console takes you to the S3 bucket landing page.

2.2 On the S3 bucket landing page, you can review the Objects, Properties, Permissions, Metrics, Management, and Access Points for the selected S3 bucket.

On the Management tab, under Replication rules, select Create replication rule.

2.3 Enter a Replication rule name and make sure Enabled is selected under the Status section. If the replication rule is disabled, it will not run.

NOTE: Amazon S3 attempts to replicate objects according to all replication rules. However, if there are two or more rules with the same destination bucket, then objects are replicated according to the rule with the highest priority. The lower the number, the higher the priority. You can edit the priority of each replication rule on the replication configuration page.  

2.4 Narrow the scope of replication by defining a Filter type (Prefix or Tags), or choose to replicate the entire bucket. For example, if you want to only replicate objects that include the prefix Finance, specify that scope. For more information on filtering objects for replication, refer to the documentation on specifying a filter in the S3 User Guide.

2.5 Choose the destination bucket you created by selecting the Browse S3 button and entering the entire bucket name.

You can't create a new S3 bucket during the replication setup process.

2.6 When creating new replication rules from the same source bucket, make sure that the AWS Identity and Access Management (IAM) role associated with this configuration has sufficient permissions to write new objects in the new destination bucket. You can choose to create a new IAM role or select an existing IAM role with the right set of permissions. For more information, see the documentation on setting up permissions for S3 Replication.

2.7 (Optional) If your objects are encrypted with Amazon S3-managed encryption keys (SSE-S3) or AWS Key Management Service (AWS KMS), specify the encryption options. S3 Replication supports SSE-S3 (default encryption), AWS KMS server-side encryption (SSE-KMS), and server-side encryption with customer-provided keys (SSE-C). If you choose AWS KMS encryption, provide the AWS KMS keys to decrypt in the source bucket and re-encrypt in the destination bucket. To save on AWS KMS costs, you can enable Amazon S3 Bucket Keys.

2.8 (Optional) Choose an S3 storage class for your replicated objects at the destination bucket. Consider choosing lower cost storage classes as appropriate for your workloads. For example, you can choose Intelligent-Tiering to optimize storage costs for data with unpredictable or changing access patterns, Glacier Instant Retrieval if your replicated objects will be infrequently accessed, but need to be retrieved in milliseconds, or Glacier Deep Archive to archive data that rarely needs to be accessed. For more information, refer to using Amazon S3 storage classes.

2.9 Choose any additional replication options you need:

• Replication Time Control (RTC): S3 RTC helps you meet compliance and business requirements because it provides an SLA of 15 minutes to replicate 99.99% of your objects. You can enable S3 RTC along with S3 CRR and S3 SRR. Replication metrics and notifications are enabled by default.

• Replication metrics and notifications: For non-RTC rules, you have the option to select Replication metrics and notifications, which provides detailed metrics to track minute-by-minute progress of bytes pending, operations pending, operations failed, and replication latency for the replication rule.

• Delete marker replication: Selecting Delete marker replication means deletes on the source bucket will be replicated to the destination bucket. This should be enabled if you want to keep the source and destination buckets in sync, but not if the goal is to protect against accidental or malicious deletes.

• Replica modifications sync: To establish two-way replication between two S3 buckets, create bidirectional replication rules (A to B, and B to A) and enable Replica modification sync for the replication rules in both the source and destination S3 buckets. This will help you to keep object metadata such as tags, ACLs, and Object Lock settings in sync between replicas and source objects.

S3 RTC, Replication metrics and notifications, and Replica modification sync are not supported while replicating existing objects with S3 Batch Replication.

When you have configured the replication, choose Save.  

2.10 When you create the first rule in a new replication configuration for your S3 bucket or add a new destination AWS Region to an existing configuration, you have the option to enable existing object replication for that replication rule. To replicate existing objects, choose, Yes, replicate existing objects, and then choose Submit.

The console takes you to the Create Batch Operations job page.

3.1 Set Job run options. If you want the S3 Batch Replication job to run immediately, you can choose Automatically run the job when it’s ready. If you want to wait to run the job when it’s ready, you can save the Batch Operations manifest to review the list of objects to be replicated.

3.2 Set Batch Operations manifest options. The Amazon S3 generated manifest file uses the same source bucket, prefix, and tags as your replication configuration to list all eligible versions of your objects for replication. We recommend to always choose Save Batch Operations manifest so that you can review the object list before replication begins. You can save the manifest in the same or a different AWS Account, but the manifest file must be stored in the same AWS Region as the source bucket.

In this example, we chose the “aws-s3-tutorial-batch-replication-manifest-destination” bucket to save the manifest file, which is in the same AWS Account as the source bucket.

3.3 For additional security, encrypt the manifest file using Amazon S3 managed keys (SSE-S3) or with AWS Key Management Service key (SS3-KMS).

3.4 As long as S3 Batch Operations successfully processes at least one object, Amazon S3 generates a completion report after the batch replication job completes, fails, or is cancelled. The completion report contains additional information for each task, including the object key name and version, status, error codes, and descriptions of any errors. We recommend choosing Generate completion report for All tasks so that you can review the status of all objects replicating with this job. For examples of completion reports, see Examples: S3 Batch Operations completion reports.

3.5 Make sure that the IAM role associated with this Batch Replication job has sufficient permissions to perform S3 Batch Operations on your behalf. For more information, see the documentation on configuring IAM policies for Batch Replication and granting permissions for Amazon S3 Batch Operations.

Review the configuration, and select Save.

You are redirected to the Batch Operations home page.

3.6 Select the Job ID of your new job to review the job configuration. You can also track the status of the Batch Replication job.

4.1 On the left navigation pane of the console home page, choose Batch Operations, and then choose Create job.

4.2 On the Create job page, select the AWS Region where you want to create your Batch Replication job. You must create the job in the same AWS Region in which the source S3 bucket is located.

4.3 Provide the list of objects to replicate. You can add a user-generated manifest in the form of an Amazon S3 inventory report or a CSV file. The manifest needs to have all of the object versions that need to be replicated. Amazon S3 can also generate a manifest for you using the existing S3 Replication configuration on the source bucket.

NOTE: In this example, we chose Create manifest using S3 Replication configuration to let Amazon S3 generate a manifest on our behalf and chose “aws-s3-replication-tutorial-source-bucket” as the source bucket. If you choose to let Amazon S3 generate a manifest for you, you will also see additional filters such as object creation date and replication status to reduce the scope of the job.

       4.4 (Optional) If you chose to save the Batch Operations manifest, encrypt your manifest file using Amazon S3 managed keys (SSE-S3) or using the AWS Key Management Service key (SSE-KMS) for additional security and access control.

• If you do not specify an encryption mode, Amazon S3 will use the default encryption settings on the manifest destination bucket to encrypt the manifest file.

• If there is no default encryption enabled on destination bucket, Amazon S3 will use SSE-S3 to encrypt the manifest file.

 

 

4.5 Choose Next to go to the Choose operation page.

4.6 If you chose Create manifest using S3 Replication Configuration on the previous page, the only Operation option is Replicate. This is because replication is the only operation that is permitted while using an S3 generated manifest. Select Replicate, and then choose Next.

4.7 Configure additional options:

• Enter a Description to best define the purpose of the job.
• Select a Priority to indicate the relative priority of this job to others running in your account. A higher number indicates higher priority. For example, a job with Priority 2 will be prioritized over a job with priority 1. S3 Batch Operations prioritizes jobs according to priority numbers, but strict ordering isn't guaranteed. Therefore, you shouldn't use job priorities to ensure that any one job starts or finishes before any other job. If you need to ensure strict ordering, wait until one job has finished before starting the next.
  

4.8 Choose whether you want to generate a completion report. 

4.9 Choose a valid Batch Operations IAM role to grant Amazon S3 permissions to perform actions on your behalf.

You must also attach a Batch Replication IAM policy to the Batch Operations IAM role. To create a valid IAM role and policy see, configuring IAM policies for Batch Replication.

4.10 Add Job tags to your Batch Replication job, and then choose Next to review your job configuration.

 

4.11 On the Review page, choose Edit to make changes, then choose Next to save your changes and return to the Review page.

When your job is ready, choose Create job.  

 

4.12 After the Batch Replication job is created, Batch Operations processes the manifest. If successful, it will change the job status to Awaiting your confirmation to run. You must confirm the details of the job before it can run.

When the job is successful, a banner displays at the top of the Batch Operations page.

5.1 From your list of S3 buckets, choose the S3 bucket that you want to configure as your source for replication.

The console takes you to the S3 bucket landing page.

5.2 Review the Objects, Properties, Permissions, Metrics, Management, and Access Points for the selected S3 bucket.

5.3 On the Management tab, under Replication rules, select View replication configuration.

5.4 On the replication configuration home page for your source bucket, choose Create replication job to go to the Create job page for S3 Batch Operations. Repeat the previous steps to create a Batch Replication job from the existing replication configuration.