Manage TPU Spot VMs

Spot VMs make unused capacity available at highly-discounted rates. Spot VMs are preemptible at any time, but unlike preemptible TPUs, there is no limit on runtime duration. You can't restart TPU Spot VMs, and you must recreate them after preemption. You also can't use 120-second preemption notice duration (Preview) with TPU Spot VMs. For more information about Spot VMs in general, see the Compute Engine documentation about Spot VMs.

Create TPU Spot VMs

You can create TPU Spot VMs using queued resources, which adds your create request to a queue and lets you receive capacity once it is available. Creating TPUs as queued resources is a best practice. For more information, see Queued resources user guide.

The following example shows how to create TPU Spot VMs using queued resources:

gcloud

Add the --spot flag to the gcloud compute tpus queued-resources command:

gcloud compute tpus queued-resources create QUEUED_RESOURCE_ID \
    --node-id=NODE_ID \
    --zone=europe-west4-a \
    --accelerator-type=v3-8 \
    --runtime-version=tpu-vm-tf-2.17.1-pod-pjrt \
    --spot

curl

Add spot: {} to the request body:

curl -X POST https://tpu.googleapis.com/v2alpha1/projects/PROJECT_ID/locations/europe-west4-a/queuedResources?queued_resource_id=QUEUED_RESOURCE_ID \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H "Content-Type: application/json" \
    -d '{
        tpu: {
        node_spec: {
            parent: "projects/PROJECT_ID/locations/europe-west4-a",
            node_id: "NODE_ID",
            node: {
            accelerator_type: "v3-8",
            runtime_version: "tpu-vm-tf-2.17.1-pod-pjrt"
            }
        }
        },
        spot: {}
    }'

Console

  1. In the Google Cloud console, go to the TPUs page: