Configure access to a source: Microsoft Azure Storage

Before transferring data from an Azure Storage bucket, you must configure access to that bucket so that Storage Transfer Service can retrieve its objects.

Storage Transfer Service supports the following Azure authentication methods:

  • Shared access signature (SAS) tokens. SAS tokens can be specified directly when creating a transfer job, or can be stored in Secret Manager.

  • Azure Shared Keys can be stored in Secret Manager and the secret passed when creating a transfer job.

  • Federated credentials are passed in a federatedIdentityConfig object during transfer job creation.

This document also includes information on adding Storage Transfer Service worker IP addresses to your Azure Storage firewall to allow access. See IP restrictions for details.

Supported regions

Storage Transfer Service is able to transfer data from the following Microsoft Azure Storage regions:
  • Americas: East US, East US 2, West US, West US 2, West US 3, Central US, North Central US, South Central US, West Central US, Canada Central, Canada East, Brazil South
  • Asia-Pacific: Australia Central, Australia East, Australia Southeast, Central India, South India, West India, Southeast Asia, East Asia, Japan East, Japan West, Korea South, Korea Central
  • Europe, Middle East, Africa (EMEA): France Central, Germany West Central, Norway East, Sweden Central, Switzerland North, North Europe, West Europe, UK South, UK West, Qatar Central, UAE North, South Africa North

Option 1: Authenticate using an SAS token

Follow these steps to configure access to a Microsoft Azure Storage container using an SAS token. You can alternatively save your SAS token in Secret Manager; to do so, follow the instructions in Authenticate using an Azure Shared Key or SAS token in Secret Manager.

  1. Create or use an existing Microsoft Azure Storage user to access the storage account for your Microsoft Azure Storage Blob container.

  2. Create an SAS token at the container level. See Grant limited access to Azure Storage resources using shared access signatures for instructions.

    1. The Allowed services must include Blob.

    2. For Allowed resource types select both Container and Object.

    3. The Allowed permissions must include Read and List. If the transfer is configured to delete objects from source, you must also include Delete permission.

    4. The default expiration time for SAS tokens is 8 hours. Set a reasonable expiration time that enables you to successfully complete your transfer.

    5. Do not specify any IP addresses in the Allowed IP addresses field. Storage Transfer Service uses various IP addresses and doesn't support IP address restriction.

    6. The Allowed protocols should be HTTPS only.

  3. Once the token is created, note the SAS token value that is returned. You need this value when configuring your transfer with Storage Transfer Service.

Option 2: Authenticate using an Azure Shared Key or SAS token in Secret Manager

Secret Manager is a secure service that stores and manages sensitive data such as passwords. It uses strong encryption, role-based access control, and audit logging to protect your secrets.

Storage Transfer Service supports Secret Manager resource names that reference your securely stored Azure credentials.

To use an Azure Shared Key, you must save the key in Secret Manager. SAS tokens can be saved in Secret Manager or passed directly.

When you specify a Shared Key, Storage Transfer Service uses that key to generate a service SAS that is restricted in scope to the Azure container specified in the transfer job.

Enable the API

Enable the Secret Manager API.

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

Enable the API

Configure additional permissions

User permissions

The user creating the secret requires the following role:

  • Secret Manager Admin (roles/secretmanager.admin)

Learn how to grant a role.

Service agent permissions

The Storage Transfer Service service agent requires the following IAM role:

  • Secret Manager Secret Accessor (roles/secretmanager.secretAccessor)

To grant the role to your service agent:

Cloud console

  1. Follow the instructions to retrieve your service agent email.

  2. Go to the IAM page in the Google Cloud console.

    Go to IAM

  3. Click Grant access.

  4. In the New principals text box, enter the service agent email.

  5. In the Select a role drop-down, search for and select Secret Manager Secret Accessor.

  6. Click Save.

gcloud

Use the gcloud projects add-iam-policy-binding command to add the IAM role to your service agent.

  1. Follow the instructions to retrieve your service agent email.

  2. From the command line, enter the following command:

    gcloud projects add-iam-policy-binding PROJECT_ID \
  --member='serviceAccount:SERVICE_AGENT_EMAIL' \
  --role='roles/secretmanager.secretAccessor'

Create a secret

Create a secret with Secret Manager:

Cloud console

  1. Go to the Secret Manager page in the Google Cloud console.

    Go to Secret Manager

  2. Click Create secret.

  3. Enter a name.

  4. In the Secret value text box, enter your credentials in one of the following formats.

    {
  "sas_token" : "SAS_TOKEN_VALUE"
}

    Or:

    {
  "access_key" : "ACCESS_KEY"
}

  5. Click Create secret.

  6. Once the secret has been created, note the secret's full resource name:

    1. Select the Overview tab.

    2. Copy the value of Resource name. It uses the following format:

      projects/1234567890/secrets/SECRET_NAME

gcloud

To create a new secret using the gcloud command-line tool, pass the JSON-formatted credentials to the gcloud secrets create command:

printf '{
  "sas_token" : "SAS_TOKEN_VALUE"
}' | gcloud secrets create SECRET_NAME --data-file=-

Or:

printf '{
  "access_key" : "ACCESS_KEY"
}' | gcloud secrets create SECRET_NAME --data-file=-

Retrieve the secret's full resource name:

gcloud secrets