Event-driven transfers from Cloud Storage

Storage Transfer Service can listen to event notifications in Google Cloud to automatically transfer data that has been added or updated in a Cloud Storage bucket. Learn more about the benefits of event-driven transfers.

Event-driven transfers from Cloud Storage use Pub/Sub notifications to know when objects in the source bucket have been modified or added. Object deletions are not detected; deleting an object at the source does not delete the associated object in the destination bucket.

Event-driven transfers always use a Cloud Storage bucket as the destination.

Configure permissions

In addition to the permissions required for all transfer jobs, event-driven transfers require the Pub/Sub Subscriber role.

1. Find the name of the Storage Transfer Service service agent for your project:

   1. Go to the googleServiceAccounts.get reference page.

      An interactive panel opens, titled Try this method.

   2. In the panel, under Request parameters, enter your project ID. The project you specify here must be the project you're using to manage Storage Transfer Service, which might be different from the source bucket's project.

   3. Click Execute.

   Your service agent's email is returned as the value of accountEmail. Copy this value.

   The service agent's email uses the format project-PROJECT_NUMBER@storage-transfer-service.iam.gserviceaccount.com.

2. Grant the Pub/Sub Subscriber role to the Storage Transfer Service service agent.

   Cloud console

   Follow the instructions in Controlling access through the Google Cloud console to grant the Pub/Sub Subscriber role to the Storage Transfer Service service. The role can be granted at the topic, subscription, or project level.

   gcloud CLI

   Follow the instructions in Set a policy to add the following binding:

   {
     "role": "roles/pubsub.subscriber",
     "members": [
       "serviceAccount:project-PROJECT_NUMBER@storage-transfer-service.iam.gserviceaccount.com"
   }

Configure Pub/Sub

1. Make sure that you've satisfied the Prerequisites for using Pub/Sub with Cloud Storage.

2. Create a Pub/Sub notification for the source Cloud Storage bucket.

   You cannot manage Pub/Sub notifications with the Google Cloud console. Use the gcloud CLI or one of the available client libraries instead.

   gcloud storage buckets notifications create gs://SOURCE_BUCKET_NAME --topic=TOPIC_NAME

3. Create a pull subscription for the topic. You must create a separate subscription for each transfer job.

   The following example shows the Google Cloud CLI command to create a pull subscription. For console instructions and client library code, see Create a pull subscription.

   gcloud pubsub subscriptions create SUBSCRIPTION_ID --topic=TOPIC_NAME --ack-deadline=300

Create a transfer job

You can use the Google Cloud console, Google Cloud CLI, or REST API to create an event-based transfer job.

Don't include sensitive information such as personally identifiable information (PII) or security data in your transfer job name. Resource names may be propagated to the names of other Google Cloud resources and may be exposed to Google-internal systems outside of your project.

gcloud transfer jobs create \
  gs://SOURCE_BUCKET \
  gs://DESTINATION_BUCKET \
  --event-stream-name=projects/PROJECT_NAME/subscriptions/SUBSCRIPTION_ID \
  --event-stream-starts=EVENT_STREAM_STARTS \
  --event-stream-expires=EVENT_STREAM_EXPIRES

transfer_job {
  "description": "YOUR DESCRIPTION",
  "status": "ENABLED",
  "projectId": "PROJECT_ID",
  "transferSpec" {
    "gcsDataSource" {
      "bucketName": "GCS_SOURCE_NAME"
    },
    "gcsDataSink": {
        "bucketName": "GCS_SINK_NAME"
    }
  }
  "eventStream" {
    "name": "projects/PROJECT_NAME/subscriptions/SUBSCRIPTION_ID",
    "eventStreamStartTime": "2022-12-02T01:00:00+00:00",
    "eventStreamExpirationTime": "2023-01-31T01:00:00+00:00"
  }
}

func createEventDrivenGCSTransfer(w io.Writer, projectID string, gcsSourceBucket string, gcsSinkBucket string, pubSubId string) (*storagetransferpb.TransferJob, error) {
	// Your Google Cloud Project ID.
	// projectID := "my-project-id"

	// The name of the source GCS bucket.
	// gcsSourceBucket := "my-source-bucket"

	// The name of the GCS bucket to transfer objects to.
	// gcsSinkBucket := "my-sink-bucket"

	// The Pub/Sub topic to subscribe the event driven transfer to.
	// pubSubID := "projects/PROJECT_NAME/subscriptions/SUBSCRIPTION_ID"

	ctx := context.Background()
	client, err := storagetransfer.NewClient(ctx)
	if err != nil {
		return nil, fmt.Errorf("storagetransfer.NewClient: %w", err)
	}
	defer client.Close()

	req := &storagetransferpb.CreateTransferJobRequest{
		TransferJob: &storagetransferpb.TransferJob{
			ProjectId: projectID,
			TransferSpec: &storagetransferpb.TransferSpec{
				DataSource: &storagetransferpb.TransferSpec_GcsDataSource{
					GcsDataSource: &storagetransferpb.GcsData{BucketName: gcsSourceBucket}},
				DataSink: &storagetransferpb.TransferSpec_GcsDataSink{
					GcsDataSink: &storagetransferpb.GcsData{BucketName: gcsSinkBucket}},
			},
			EventStream: &storagetransferpb.EventStream{Name: pubSubId},
			Status:      storagetransferpb.TransferJob_ENABLED,
		},
	}
	resp,