Use cross-bucket replication

This page provides an overview of cross-bucket replication and instructions on how to use the feature.

Overview of cross-bucket replication

In some cases, you might want to copy data from one Cloud Storage bucket to another. Cross-bucket replication copies new and updated objects asynchronously from a source bucket to a destination bucket.

Cross-bucket replication differs from default replication and turbo replication in that your data exists in two buckets, each with their own configurations such as storage location, encryption, access, and storage class. As a result, it offers data recovery and availability, but is also suitable for:

• Data sovereignty: Maintain data across geographically distant regions.
• Maintain separate development and production versions: Create distinct buckets and namespaces, so that development doesn't affect your production workload.
• Share data: Replicate data to a bucket owned by a vendor or partner.
• Data Aggregation: Combine data from different buckets into a single bucket to run analytics workloads.
• Manage cost, security, and compliance: Maintain your data under different ownerships, storage classes, and retention periods.

Cross-bucket replication uses Storage Transfer Service to replicate objects and Pub/Sub to get alerted of changes to the source and destination buckets. Cross-bucket replication can be enabled on new buckets you create and on existing buckets. Most objects can be replicated in the order of minutes, while objects larger than one GiB can take several hours.

Limitations

• Custom names are not supported for cross-bucket replication jobs. Create requests that contain a value for the name field return an error.

• Cross-bucket replication is not supported for hierarchical namespace buckets or zonal buckets.

• Object deletions in the source bucket are not replicated to the destination bucket.

• Object lifecycle configurations aren't replicated.

• When objects are replicated, timestamp metadata (for example, timeCreated and timeUpdated) is not preserved. See Transfers between Cloud Storage buckets for details on metadata preservation.

• Because cross-bucket replication can be used to replicate data between buckets located in any Google Cloud location, cross-bucket replication performance varies based on the locations selected. Consequently, cross-bucket replication does not offer a Recovery Point Objective (RPO).

• Objects that are already in the bucket when a replication job is created are not automatically replicated. Only new and updated objects are replicated. To replicate existing objects, create a one-time Storage Transfer Service transfer job from your existing bucket to the new bucket. See Create transfers for instructions.

Before you begin

Before you begin, complete the following steps.

Enable the Storage Transfer Service API

If you haven't already, enable the Storage Transfer Service API.

Get required roles

To get the permissions that you need to use cross-bucket replication, ask your administrator to grant you the Storage Transfer User (roles/storagetransfer.user) IAM role on the bucket or the source project.

This predefined role contains the permissions required to use cross-bucket replication. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to use cross-bucket replication:

• storagetransfer.jobs.create
• storagetransfer.jobs.delete
• storagetransfer.jobs.get
• storagetransfer.jobs.list
• storagetransfer.jobs.run
• storagetransfer.jobs.update

For instructions on granting roles on buckets, see Use IAM with buckets. For instructions on granting roles on projects, see Manage access to projects.

Grant required roles

Cross-bucket replication uses Storage Transfer Service to replicate objects from your source bucket to your destination bucket, and Pub/Sub to receive notifications of changes to your source bucket.

So that cross-bucket replication can use these services, you must grant the required permissions to the service account that's used by Storage Transfer Service to replicate data, as well as the service agent that's used by Pub/Sub to write notifications.

Grant required roles for Storage Transfer Service

The required permissions to replicate data using Storage Transfer Service can be granted to one of two service accounts. Either:

• The Storage Transfer Service service agent, which is automatically created and managed by Google. The email address of this service agent follows the naming format project-PROJECT_NUMBER@storage-transfer-service.iam.gserviceaccount.com. You can get the email address of the Storage Transfer Service service agent by using the Storage Transfer Service googleServiceAccounts.get API.

Or:

• A user-managed service account. The email address of this service account follows the naming format SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com. You can create your own service account and grant necessary permissions to replicate the data.

  For instructions on using a user-managed service account with Storage Transfer Service, see Delegate service agent permissions to a user-managed service account. There are additional permissions required beyond those in the following lists. In particular, you must grant the service agent access to the user-managed service account by assigning the Service Account Token Creator role (roles/iam.serviceAccountTokenCreator). See Grant access to the service agent.

The Storage Transfer Service service agent or user service account needs the following permissions to replicate your objects and set up Pub/Sub notifications for your source bucket:

These permissions can be granted through custom roles or by granting all of the following predefined roles:

• Pub/Sub Editor (roles/pubsub.editor) role on the source project
• Storage Legacy Bucket Owner (roles/storage.legacyBucketOwner) and Storage Object Viewer (roles/storage.objectViewer) roles on the source bucket
• Storage Legacy Bucket Writer (roles/storage.legacyBucketWriter) and Storage Object Viewer (roles/storage.objectViewer) roles on the destination bucket

Grant required roles for Pub/Sub

Cloud Storage uses a service agent to manage Pub/Sub notifications. The email address of this service agent follows the naming format service-PROJECT_NUMBER@gs-project-accounts.iam.gserviceaccount.com.

The Cloud Storage service agent needs the following permissions to set up Pub/Sub and publish messages to a topic: