Manage agent pools

An agent pool is a collection of agents that use the same configuration, with uniform access and visibility to your source and destination. Agent pools also provide control over transfer bandwidth limits.

All agents must belong to an agent pool.

This guide describes how to use agent pools.

Before you begin

You must have the correct permissions on your project to work with agent pools. Refer to Access control with IAM for details.

You can view your current permissions.

To use gcloud commands, install the gcloud command-line tool.

Create an agent pool

Don't include sensitive information such as personally identifiable information (PII) or security data in your agent pool name or agent ID prefix. Resource names may be propagated to the names of other Google Cloud resources and may be exposed to Google-internal systems outside of your project.

To create an agent pool:

Google Cloud console

  1. In the Google Cloud console, go to the Agent pools page.

    Go to Agent pools

    The Agent pools page is displayed, listing your existing agent pools.

  2. Click Create another pool.

  3. Name your pool, and optionally describe it.

  4. You may choose to set a bandwidth limit that will apply to the pool as a whole. The specified bandwidth in MB/s will be split amongst all of the agents in the pool. See Manage network bandwidth for more information.

  5. Click Create.

REST API

Use projects.agentPools.create:

POST https://storagetransfer.googleapis.com/v1/projects/PROJECT_ID/agentPools?agent_pool_id=AGENT_POOL_ID

Replace the following:

  • PROJECT_ID: The project ID that you're creating the agent pool in.
  • AGENT_POOL_ID: The agent pool ID that you are creating.

If an agent pool is stuck in the Creating state for more than 30 minutes, we recommend deleting the agent pool and creating it again.

Revoking required Storage Transfer Service permissions from a project while an agent pool is in the Creating state leads to incorrect service behavior.

gcloud CLI

To create an agent pool with the gcloud command line tool, run gcloud transfer agent-pools create.

gcloud transfer agent-pools create AGENT_POOL

Where the following options are available:

  • AGENT_POOL is a unique, permanent identifier for this pool.

  • --no-async blocks other tasks in your terminal until the pool has been created. If not included, pool creation runs asynchronously.

  • --bandwidth-limit defines how much of your bandwidth in MB/s to make available to this pool's agents. A bandwidth limit applies to all agents in a pool and can help prevent the pool's transfer workload from disrupting other operations that share your bandwidth. For example, enter '50' to set a bandwidth limit of 50 MB/s. By leaving this flag unspecified, this pool's agents will use all bandwidth available to them.

  • --display-name is a modifiable name to help you identify this pool. You can include details that might not fit in the pool's unique full resource name.

Assign agents to a pool

New agent pools are empty and must have agents assigned to them. An agent can only be assigned to an agent pool at the time of the agent's creation.

If an agent pool is not specified, the agent is assigned to the transfer_service_default pool.

To create an agent and assign it to a pool:

Google Cloud console

  1. From the Agent pools page, select the pool to assign the agent(s) to.

  2. Click Install agent. The Agent installation guide appears.

  3. Follow the instructions to create the agent in this agent pool.

gcloud CLI

To use the gcloud command-line tool to create an agent and assign it to a pool, run gcloud transfer agents install. The agent must be installed on a machine with root access to the file system.

gcloud transfer agents install --pool=AGENT_POOL

To install agents to use with an S3-compatible source, provide access credentials either as environment variables as the values of AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, or stored as default credentials in your system's configuration files.

export AWS_ACCESS_KEY_ID=ID
export AWS_SECRET_ACCESS_KEY=SECRET
gcloud transfer agents install --pool=AGENT_POOL

See Manage transfer agents for more details.

For additional options, see Advanced agent options.

To manage agents, see Manage transfer agents.

Assign an agent pool to a job

Once your agent pool has been created, it can be assigned to a job during job creation or