Create transfers

This page shows you how to create and start transfer jobs.

To see if your source and destination (also known as a sink) are supported by Storage Transfer Service, refer to Supported sources and sinks.

Agents and agent pools

Depending on your source and destination, you may need to create and configure an agent pool and install agents on a machine with access to your source or destination.

Transfers from Amazon S3, Microsoft Azure, URL lists, or Cloud Storage to Cloud Storage do not require agents and agent pools.
Transfers whose source and/or destination is a file system, or from S3-compatible storage, do require agents and agent pools. See Manage agent pools for instructions.

Before you begin

Before configuring your transfers, make sure you have configured access:

For users and service accounts:
- Agentless transfers
- Agent-based transfers
To your source data and to your destination.

If you're using gcloud commands, install the gcloud CLI.

Create a transfer

Don't include sensitive information such as personally identifiable information (PII) or security data in your transfer job name. Resource names may be propagated to the names of other Google Cloud resources and may be exposed to Google-internal systems outside of your project.

Google Cloud console

Go to the Storage Transfer Service page in the Google Cloud console.

Go to Storage Transfer Service
Click Create transfer job. The Create a transfer job page is displayed.
Choose a source:
Cloud Storage
Your user account must have storage.buckets.get permission to select source and destination buckets. Alternatively, you can type the name of the bucket directly. For more information, see Troubleshooting access.
1. Under Source type, select Cloud Storage.
2. Select your Destination type.
3. If your destination is Cloud Storage, select your Scheduling mode. Batch transfers execute on a one-time or scheduled basis. Event-driven transfers continuously monitor the source and transfer data when it's added or modified.
  
  To configure an event-driven transfer, follow the instructions at Event-driven transfers.
4. Click Next step.
5. Select a bucket and (optionally) a folder in that bucket by doing one of the following:
  
  Enter an existing Cloud Storage bucket name and path in the Bucket or folder field without the prefix gs://. For example, my-test-bucket/path/to/files. To specify a Cloud Storage bucket from another project, type the name exactly into the Bucket name field.
  
  Select a list of existing buckets in your projects by clicking Browse, then selecting a bucket.
  
  When you click Browse, you can select buckets in other projects by clicking the Project ID, then selecting the new Project ID and bucket.
  
  To create a new bucket, click Create new bucket.
  
  If you specify a path within the folder, see Specifying source and destination paths for important details.
6. If this is an event-driven transfer, enter the Pub/Sub subscription name, which takes the following format:
  
  projects/PROJECT_NAME/subscriptions/SUBSCRIPTION_ID
7. Optionally, choose to filter objects by prefix or by last modified date. If you specified a folder as your source location, prefix filters are relative to that folder. For example, if your source is my-test-bucket/path/, an include filter of file includes all files starting with my-test-bucket/path/file.
8. Click Next step.
Amazon S3
See Transfer from Amazon S3 to Cloud Storage.

S3-compatible storage
See Transfer from S3-compatible storage to Cloud Storage.
Microsoft Azure Blob Storage
Storage Transfer Service is able to transfer data from the following Microsoft Azure Storage regions:
Americas: East US, East US 2, West US, West US 2, West US 3, Central US, North Central US, South Central US, West Central US, Canada Central, Canada East, Brazil South

Asia-Pacific: Australia Central, Australia East, Australia Southeast, Central India, South India, West India, Southeast Asia, East Asia, Japan East, Japan West, Korea South, Korea Central

Europe, Middle East, Africa (EMEA): France Central, Germany West Central, Norway East, Sweden Central, Switzerland North, North Europe, West Europe, UK South, UK West, Qatar Central, UAE North, South Africa North
1. Under Source type, select Azure Blob Storage or Data Lake Storage Gen2.
2. Click Next step.
3. Specify the following:
  
  Storage account name — the source Microsoft Azure Storage account name.
  
  The storage account name is displayed in the Microsoft Azure Storage portal under All services > Storage > Storage accounts.
  
  Container name — the Microsoft Azure Storage container name.
  
  The container name is displayed in the Microsoft Azure Storage portal under Storage explorer > Blob containers.
  
  Shared access signature (SAS) — the Microsoft Azure Storage SAS token created from a stored access policy. For more information, see Grant limited access to Azure Storage resources using shared access signatures (SAS).
  The default expiration time for SAS tokens is 8 hours. When you create your SAS token, be sure to set a reasonable expiration time that enables you to successfully complete your transfer.
  Caution: Basic SAS tokens can't be revoked, and the only way to invalidate a basic SAS token is to remove the storage access key of your account. We strongly recommend that you create SAS tokens from stored access policies, so that you can revoke a policy to invalidate an SAS token. For more information, see Best practices when using SAS.
  
  Note: When creating an SAS key, avoid including an IP restriction. Storage Transfer Service uses various IP addresses and doesn't support IP address restriction.
4. Optionally, choose to filter objects by prefix or by last modified date. If you specified a folder as your source location, prefix filters are relative to that folder. For example, if your source is my-test-bucket/path/, an include filter of file includes all files starting with my-test-bucket/path/file.
5. Click Next step.
File system
1. Under Source type, select POSIX file system.
2. Select your Destination type and click Next step.
3. Select an existing agent pool, or select Create agent pool and follow the instructions to create a new pool.
4. Specify the fully qualified path of the file system directory. If you specify a path other than root, see Specifying source and destination paths for important details.
5. Click Next step.
HDFS
See Transfer from HDFS to Cloud Storage.
URL list
1. Under Source type, select URL list and click Next step.
2. Under URL of TSV file, provide the URL to a tab-separated values (TSV) file. See Creating a URL List for details about how to create the TSV file.
3. Optionally, choose to filter objects by prefix or by last modified date. If you specified a folder as your source location, prefix filters are relative to that folder. For example, if your source is my-test-bucket/path/, an include filter of file includes all files starting with my-test-bucket/path/file.
4. Click Next step.
Choose a destination:
Cloud Storage
1. In the Bucket or folder field, enter the destination bucket and (optionally) folder name, or click Browse to select a bucket from a list of existing buckets in your current project. To create a new bucket, click Create new bucket.
  
  If you specify a path within the bucket, see Specifying source and destination paths for important details.
2. Click Next step.
3. Choose your scheduling options:
  
  Note:

Create transfers Stay organized with collections Save and categorize content based on your preferences.

Agents and agent pools

Before you begin

Create a transfer

Google Cloud console

Cloud Storage

Amazon S3

S3-compatible storage

Microsoft Azure Blob Storage

File system

HDFS

URL list

Cloud Storage​

Create transfers

Cloud Storage