Automate task retries

This page describes how to automatically retry tasks after all or some failures.

A Batch job fails when at least one of its tasks fails, which can happen for various reasons. By default, each task in a job only runs once; if a task fails, it is not retried. However, some issues that cause a task to fail can be easily resolved just by retrying the task. In these cases, configuring the job to automatically retry tasks can substantially help reduce troubleshooting friction and the overall run time of your jobs.

Automatic retries are well-suited to loosely coupled (independent) tasks and can help with a variety of issues. For example, automatic task retries can resolve time-sensitive issues like following:

You can configure automatic task retries for each task when you create a job. Specifically, for each task, you can use one of the following configuration options:

  • By default, each task is not retried when it fails.
  • Retry tasks for all failures: You can configure the maximum times to automatically retry failed tasks. You can specify between 0 (default) and 10 retries.
  • Retry tasks for some failures: You can configure different task actions—either automatic retry or fail without retry—for specific failures. The opposite action is taken for all unspecified failures. Specific failures can each be identified by an exit code that is defined by your application or Batch.

Before you begin

  1. If you haven't used Batch before, review Get started with Batch and enable Batch by completing the prerequisites for projects and users.

  2. To get the permissions that you need to create a job, ask your administrator to grant you the following IAM roles:

    For more information about granting roles, see Manage access to projects, folders, and organizations.

    You might also be able to get the required permissions through custom roles or other predefined roles.

Retry tasks for all failures

You can define the maximum number of automatic retries (maxRetryCount field) for a job's failed tasks using the gcloud CLI or Batch API.

gcloud

  1. Create a JSON file that specifies the job's configuration details and the maxRetryCount field.

    For example, to create a basic script job that specifies the maximum retries for failed tasks, create a JSON file with the following contents:

    {
  "taskGroups": [
    {
      "taskSpec": {
        "runnables": [
          {
            "script": {
              "text": "echo Hello world from task ${BATCH_TASK_INDEX}"
            }
          }
        ],
        
        "maxRetryCount": MAX_RETRY_COUNT
        
      },
      "taskCount": 3
    }
  ],
  "logsPolicy": {
    "destination": "CLOUD_LOGGING"
  }
}

    Replace MAX_RETRY_COUNT with the maximum number of retries for each task. For a job to be able to retry failed tasks, this value must be set to an integer between