Object Storage - Master asynchronous replication across your buckets

Introduction

Object replication is a powerful feature that facilitates the automatic and asynchronous replication of objects within a source bucket to one or several destination buckets. This capability is crucial for maintaining data consistency, availability, and redundancy across different storage locations.

Destination buckets can reside within a single region or be dispersed across multiple regions, tailored to your specific requirements. This flexibility allows for strategic data placement and management across global infrastructure networks.

Objectives

This guide aims to equip you with the knowledge and skills to:

• Set up object replication: Learn to configure object replication across buckets for automated data duplication from a source to one or more destinations.
• Enhance data availability: Understand how object replication improves data resilience by creating copies in different regions or storage zones.
• Achieve compliance: Explore how replication aids in meeting regulatory requirements for data geo-redundancy and backup.
• Reduce storage costs: Discover strategies to lower storage expenses by replicating data to more cost-effective storage classes.
• Facilitate data sharing: See how object replication streamlines data sharing and synchronization across teams, boosting operational efficiency.

Requirements

• Cloud storage account: An active account with access to cloud storage services that support object replication.
• Buckets configured: At least two buckets set up within your cloud storage account, designated as the source and destination.
• Data backup: A recent backup of your data, particularly important if setting up replication for existing data to avoid accidental loss.
• Understanding of storage classes: Familiarity with the various storage classes offered by your cloud service, along with their cost and performance implications.
• Familiarity with cloud storage policies: Knowledge of the policies and permissions necessary for performing object replication.
• CLI tools or management console access: The ability to use your cloud provider's command line interface (CLI) tools or management console to set up and manage replication rules.
• Versioning enabled: Versioning must be activated on your buckets if required by your cloud service for object replication.
• Object Storage User: An Object Storage user account already created within your project.
• AWS CLI configuration: The AWS CLI installed and configured on your system. For a guide on configuring the CLI, refer to the OVHcloud documentation on "Getting started with Object Storage".

Instructions

Key use cases for object replication

• Exact object copies with metadata replication: Replication is not just about duplicating the object; it includes the replication of all associated metadata (e.g. object creation time, version ID, etc.). This ensures that the replicas are true copies of the source objects, maintaining integrity and consistency for critical applications.

• Data synchronization across teams: This facilitates seamless synchronization of data across various teams, enhancing collaboration and data sharing based on predefined access controls and policies. It is crucial to note that while data synchronization is a significant advantage, storage options and configurations must be carefully managed to ensure they meet the specific needs of each team in terms of access and security.

• Cost-effective data storage management: Organizations need to explore alternative strategies to optimize their backup and storage costs, considering the current limitations related to data replication. By default, replicas are created using the source object’s storage class. If you need replicas in a different storage class, set Destination.StorageClass (where supported by the destination region and storage class availability). Nevertheless, organizations can still optimize their storage management by carefully assessing their needs and selecting the most suitable storage class from the outset to balance cost and performance without compromising data availability or durability.

• Enhanced data resiliency across regions: Enhance your data protection strategies by replicating critical data across multiple geographical regions. This increases resiliency against data loss and ensures business continuity in the face of regional disruptions.

• Reduced latency for global access: Positioning your data closer to your end users minimizes access latency and improves the overall user experience. Replication allows for strategic data placement in OVHcloud regions nearest to your customer base.

• Efficiency boost for computational workloads: By bringing your data closer to your OVHcloud compute resources, replication enhances the efficiency and performance of your workloads, facilitating faster data processing and analysis.

• Compliance and regulatory fulfillment: Many compliance frameworks mandate that data be stored at a considerable distance from the primary location or require multiple copies of critical data. Object replication simplifies the process of meeting these requirements by enabling automatic replication across vast distances and into multiple storage mediums.

Implementing object replication ensures not only the safety and availability of your data but also enhances operational efficiency and compliance posture.

What is asynchronous replication?

Basic concepts

At its core, the asynchronous replication is designed to facilitate several key operations in the management and safeguarding of your data. This includes the following actions:

• Exact replica creation

• Replicate data within the same region

• Replicate data to a different region

• Replicate data to two different regions

What is replicated and what is not

The following table provides the default behavior of the OVHcloud Object Storage Asynchronous Replication feature:

¹: To replicate objects that were uploaded before the creation of the replication configuration, use Batch Replication. Learn more about configuring Batch Replication at Replicating existing objects.

²: Learn how to activate the replication of delete markers at Delete marker replication.

Replication configuration

A replication configuration is defined through a set of rules within a JSON file. This file is uploaded and applied to the source bucket, detailing how objects are to be replicated. Each replication rule defines:

• A unique rule ID to identify the rule.
• Rule priority to determine the order of execution when multiple rules exist.
• Destination bucket where the replicated objects will be stored.
• Objects to be replicated: By default, all objects are eligible for replication. However, you can specify a subset of objects by filtering them with a prefix and/or tags.

Replication rule structure

The basic structure of a replication rule within the configuration JSON file is as follows:

{
  "Role": "arn:aws:iam::<project_id>:role/s3-replication",
  "Rules": [
    {
      "ID": "string",
      "Priority": integer,
      "Filter": {
        "Prefix": "string",
        "Tag": {
          "Key": "string",
          "Value": "string"
        },
        "And": {
          "Prefix": "string",
          "Tags": [
            {
              "Key": "string",
              "Value": "string"
            }
          ]
        }
      },
      "Status": "Enabled"|"Disabled",
      "Destination": {
        "Bucket": "arn:aws:s3:::<destination_bucket_name>",
        "StorageClass": "STANDARD"|"STANDARD_IA"|"EXPRESS_ONEZONE"
      },
      "DeleteMarkerReplication": {
        "Status": "Enabled"|"Disabled"
      }
    }
  ]
}

Delete marker replication

Understanding delete markers

When a delete object operation is performed on an object in a versioning-enabled bucket, it does not delete the object permanently but it creates a delete marker on the object. This delete marker becomes the latest and current version of the object with a new version ID.

A delete marker has the following properties:

• A key and version ID like any other object.
• It does not have data associated with it, thus it does not retrieve anything from a GET request (you get a 404 error).
• By default, it is not displayed in the Control Panel UI anymore.
• The only operation you can use on a delete marker is DELETE, and only the bucket owner can issue such a request.

To permanently delete an object, you have to specify the version ID in your DELETE object request.

Replicate delete markers

However, you can still replicate delete markers by adding the DeleteMarkerReplication element to your replication configuration rule. DeleteMarkerReplication specifies if delete markers should or should not be replicated (when versioning is enabled, a delete operation is performed on an object it does not actually delete the object but it flags it with a delete marker).

{
  "Role": "arn:aws:iam::<project_id>:role/s3-replication",
  "Rules": [
    {
      "ID": "ruleId",
      "Status": "Enabled",
      "Priority": 1,
      "Filter": {},
      "Destination": {
        "Bucket": "arn:aws:s3:::destination-bucket"
      },
      "DeleteMarkerReplication": {
        "Status": "Enabled"
      }
    }
  ]
}

Checking the replication status

The replication status can be used to determine the status of an object that is being replicated. To get the replication status of an object, you can use the head-object command via the AWS CLI:

aws s3api head-object --bucket <source_bucket_name> --key <object_key>
{
  "LastModified": "Fri, 15 Mar 2024 10:18:15 GMT",
  "ContentLength": 3481,
  "ETag": "\"417947d3634d4645e05ca9e875f5b202\"",
  "VersionId": "17104978950.04081",
  "ContentType": "binary/octet-stream",
  "Metadata": {},
  "StorageClass": "STANDARD",
  "ReplicationStatus": "COMPLETED"
}

The ReplicationStatus attribute can have the following values:

Replication between buckets with object lock enabled

Object Lock can be used with replication to enable automatic copying of locked objects across buckets. For replicated objects, the object lock configuration of the source bucket will be used in the destination bucket. However, if you upload an object directly to the destination bucket (outside of the replication process), it will use the lock configuration of the destination bucket.

Replicating existing objects

By default, the Asynchronous Replication feature does not replicate objects uploaded before the setup of a replication configuration i.e existing objects. While Asynchronous Replication continuously and automatically replicates new objects across OVHcloud Object Storage buckets, Batch Replication occurs on demand on existing objects.

You can get started with Batch Replication by creating a new Batch replication job that will get executed on your source bucket.

Special considerations

Before creating your first job, please take into account the following considerations:

• Your source bucket and destination(s) bucket(s) must have versioning enabled.
• Your source bucket must have an existing replication configuration set up, as Batch Replication will create a job that will try to apply the existing replication configuration to ALL objects of the source bucket that have NOT been replicated yet.
• If you have a Lifecycle policy configured for your bucket, we recommend disabling your lifecycle rules while the Batch Replication job is active to ensure maximum consistency between buckets and data synchronization.
• You cannot create another Batch Replication job when there is a running job, this limitation helps us to protect our infrastructures from malicious and/or abusive uses.
• Batch replication does NOT support objects that are stored in the Cold Archive storage class.
• There are no SLAs on the job time to completion.

Checking the Batch Replication job status

Getting started with Batch Replication

    POST /cloud/project/{serviceName}/region/{regionName}/storage/{name}/job/replication HTTP/1.1
    -H "accept: application/json"\
    -H "authorization: Bearer {auth_token}"

{
    "id": "{job_id}"
}

Examples of replication configurations

Simple replication between 2 buckets

{
  "Role": "arn:aws:iam::<your_project_id>:role/s3-replication",
  "Rules": [
    {
      "ID": "ruleId",
      "Status": "Enabled",
      "Priority": 1,
      "Filter": { },
      "DeleteMarkerReplication": { "Status": "Disabled" },
      "Destination": {
        "Bucket": "arn:aws:s3:::destination-bucket"
      }
    }
  ]
}

This configuration will replicate all objects (indicated by the empty Filter field) to the bucket destination-bucket.

Replication of delete markers

{
  "Role": "arn:aws:iam::<your_project_id>:role/s3-replication",
  "Rules": [
    {
      "ID": "ruleId",
      "Status": "Enabled",
      "Priority": 1,
      "Filter" : {
        "Prefix": "backup"
      },
      "Destination": {
        "Bucket": "arn:aws:s3:::destination-bucket"
      },
      "DeleteMarkerReplication": { "Status": "Enabled" }
    }
  ]
}

This configuration will replicate all objects that have the prefix "backup" to the bucket destination-bucket. Additionally, we indicate that deletion operations in the source bucket should be also replicated.

Replicating source to multiple regions

{
  "Role": "arn:aws:iam::<your_project_id>:role/s3-replication",
  "Rules": [
    {
      "ID": "rule1",
      "Status": "Enabled",
      "Priority": 1,
      "Filter": {},
      "Destination": {
        "Bucket": "arn:aws:s3:::region1-destination-bucket"
      },
      "DeleteMarkerReplication": {
        "Status": "Disabled"
      }
    },
    {
      "ID": "rule2",
      "Status": "Enabled",
      "Priority": 2,
      "Filter": {},
      "Destination": {
        "Bucket": "arn:aws:s3:::region2-destination-bucket"
      },
      "DeleteMarkerReplication": {
        "Status": "Disabled"
      }
    }
  ]
}

Suppose the source bucket, region1-destination-bucket and region2-destination-bucket are 3 buckets in 3 OVHcloud regions, this configuration will allow you to back up all objects in the source bucket to 2 different regions.

Replicating 2 subsets of objects to different destination buckets

{
  "Role": "arn:aws:iam::<your_project_id>:role/s3-replication",
  "Rules": [
    {
      "ID": "rule1",
      "Status": "Enabled",
      "Priority": 1,
      "Filter" : {
        "Prefix": "dev"
      },
      "Destination": {
        "Bucket": "arn:aws:s3:::destination-bucket1"
      },
      "DeleteMarkerReplication": { "Status": "Enabled" }
    },
    {
      "ID": "rule2",
      "Status": "Enabled",
      "Priority": 2,
      "Filter" : {
        "Prefix": "prod"
      },
      "Destination": {
        "Bucket": "arn:aws:s3:::destination-bucket2"
      },
      "DeleteMarkerReplication": { "Status": "Disabled" }
    }
  ]
}

This configuration contains 2 replication rules:

• rule1 will replicate all objects with prefix "dev" to bucket destination-bucket1 and additionally, will replicate also deletion operations.
• rule2 will replicate all objects with prefix "prod" to bucket destination-bucket2 without replicating deletion operations.

Step-by-step instructions

Create source and destination buckets

The source bucket is the bucket whose objects are automatically replicated and the destination bucket is the bucket which will contain your object replicas.

aws s3 mb s3://<bucket_name>

Example: Creation of a source bucket and a destination bucket

aws s3 mb s3://my-source-bucket
aws s3 mb s3://my-destination-bucket

Activate versioning in source and destination bucket

aws s3api put-bucket-versioning --bucket <bucket_name> --versioning-configuration Status=Enabled

Example: Activation of versioning in previously created source and destination buckets

aws s3api put-bucket-versioning --bucket my-source-bucket --versioning-configuration Status=Enabled
aws s3api put-bucket-versioning --bucket my-destination-bucket --versioning-configuration Status=Enabled

Apply replication configuration

Via the AWS S3api

Via the OVHcloud Control Panel

Using the AWS CLI, replication configuration is applied on the source bucket.

aws s3api put-bucket-replication --bucket <source> --replication-configuration file://<conf.json>

Example: Replicate all objects with prefix "docs" to my-destination-bucket and replicate delete markers (i.e., objects marked as deleted in the source will be marked as deleted in the destination).

{
   "Role": "arn:aws:iam::<your_project_id>:role/s3-replication",
   "Rules": [
    {
      "ID": "replication-rule-456",
      "Status": "Enabled",
      "Priority": 1,
      "Filter": {
        "And": {
          "Prefix": "docs",
          "Tags": [
            {
              "Key": "importance",
              "Value": "high"
            }
          ]
        }
      },
      "Destination": {
        "Bucket": "arn:aws:s3:::my-destination-bucket"
      },
      "DeleteMarkerReplication": {
        "Status": "Enabled"
      }
    }
  ]
}

Info

You will need the following prerequisites:

• A source bucket and a destination bucket.
• Versioning must be enabled on both the source bucket and the destination bucket.

Go directly to the information panel for the bucket concerned, then click on Manage replication:

Select Add a replication rule.

Specify a name for your rule to help you identify it later. This name is required and must be unique within your bucket.

You can specify a prefix and/or tags to limit the scope of the objects to be replicated.

Warning

As a reminder, you cannot replicate delete markers if you are using tags to filter your objects.

Under Destination, select a destination bucket. The selected bucket must have versioning enabled and if object lock has been enabled on the source bucket then it also must be enabled on the destination bucket.

• By default, the objects will be replicated with the same storage class. However, you can choose to replicate the objects to another storage class.
• If there are two or more rules with the same destination bucket, objects will be replicated according to the rule with the highest priority. The higher the number, the higher the priority.

Under Status, Enabled is selected by default. An enabled rule starts to work as soon as you save it. If you want to disable the rule at creation and activate it later, choose Disabled.

To finish, click on Create the rule.

Delete a replication rule

In the Replication rules management view, you can delete a rule from the menu.

Offsite Replication option in 3-AZ regions

With Object Storage in a 3-AZ region, we have introduced a new option called Offsite Replication, which simplifies the replication process and automatically replicates your data to a remote site for greater resiliency with a one-click action in the OVHcloud Control Panel. This feature is only available for primary Object Storage in a 3-AZ region (to know more about existing 1-AZ and 3-AZ regions, see Endpoints and Object Storage geoavailability) and is based on an OVHcloud auto-generated and managed replication rule configuration:

• Data is replicated on a remote 1-AZ region. The system will automatically determine the most suitable location among Strasbourg, Gravelines, and Roubaix, ensuring efficient and reliable offsite data protection.
• Objects stored in the replica bucket are stored in an Infrequent Access class and are billed differently. View pricing on this page. This class is designed for less frequently accessed data and allow you to reduce your overall Object Storage bill together with Standard-class performance level. This said, if the destination bucket storage class doesn’t suit your needs, you can choose another approach and use the asynchronous replication feature and manage the replication rule configuration by yourself.
• The replica bucket and the replication rule configuration are then available for modification if needed.

Offsite Replication Q&A

How can I access the option in the OVHcloud Control Panel?

When creating a new bucket/container in a 3-AZ region, you will be asked if you want to activate the Offsite Replication option. If enabled, and because it relies on the asynchronous replication feature, versioning will be automatically enabled too.

What are the differences between the asynchronous replication feature and the Offsite Replication feature available in 3-AZ regions?

The Offsite Replication option offered in 3-AZ regions is based on the asynchronous replication feature. With this Offsite Replication option, OVHcloud automatically generates a replication rule configuration with pre-filled parameters, whereas the S3¹-compatible asynchronous replication functionality allows the user to take control of the entire function (configuration and deployment).

Where will the replicated data be stored, since the replication rule configuration is managed by OVHcloud?

Replicated data is stored like all other data, in a bucket automatically created by OVHcloud. Depending on what is available in the Control Panel, you may be offered a destination choice; otherwise OVHcloud selects automatically among Strasbourg, Gravelines, and Roubaix.

What if a replica bucket is deleted?

The source objects that are to be replicated will have a replication status FAILED when Object Storage tries to replicate them but you will not get any errors.

What about the initial replication rule?

The initial rule will remain unchanged and active until you modify or disable it.

What if a source bucket with offsite replication enabled is deleted? And what about the initial replication rule?

Your replica bucket still exists but no more objects are replicated to it from your original source bucket. The initial replication rule is set up on the source bucket and thus will also be deleted.

Where will be stored the replicated data as the replication rule configuration is managed by OVHcloud?

Your replicated data is stored as any other data and will be stored in a bucket automatically generated by OVHcloud in a region of our choosing. The system will automatically determine the most suitable location between Strasbourg, Gravelines, and Roubaix, ensuring efficient and reliable offsite data protection.

Can I modify the replication rule configuration?

We recommend that you do not modify the auto-generated replication rule configuration to ensure optimal operation. However, as it is a standard asynchronous replication rule configuration, you can decide to enrich/upgrade it for advanced protection by following the steps highlighted in the sections above.

What is the name of the replica bucket?

The destination bucket name follows the pattern backup-{src-region}-{dst-region}-{timestamp_in_ms}-{src-bucket}.

How can I access my backed up data and which actions are possible with the replica bucket?

You can list/head/delete objects on this replica bucket. Data stored on the replica bucket use the Infrequent Access storage class to help you optimize your storage costs. As it is used for data protection reasons and is supposed to be rarely accessed, this storage class is adapted and designed for such use cases. The replica bucket is exclusively dedicated to the Offsite Replication option. You can also read those objects, however a retrieval fee is applied for the Infrequent Access target storage class. View pricing on this page.

Which users/credentials can be used to access the destination bucket?

When you activate the Offsite Replication during the source bucket creation, the same user that you associated with your source bucket will be associated with the replica bucket.

How will my backup data be billed?

You can access the details of the Offsite Replication pricing on the global Object Storage pricing page. It is billed according to the storage space used, with a granularity of 1 GiB. To ensure readability, the price is displayed per GiB/month, but the billing granularity is per GiB/hour.

Go further

If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for a custom analysis of your project.

Join our community of users.

¹: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.