Analyze a job using logs

This document describes how to enable, generate, and view logs from Cloud Logging for a Batch job.

You can use logs to get information that is useful for analyzing your jobs. For example, logs can help you debug failed jobs.

Notably, logs are only generated after a job starts running and only if logging was enabled for the job. If you need to analyze a job without logs, view status events instead.

Before you begin

If you haven't used Batch before, review Get started with Batch and enable Batch by completing the prerequisites for projects and users.
To get the permissions that you need to analyze a job using logs, ask your administrator to grant you the following IAM roles:
- To create a job:
  - Batch Job Editor (roles/batch.jobsEditor) on the project
  - Service Account User (roles/iam.serviceAccountUser) on the job's service account, which by default is the default Compute Engine service account
- To view logs: Logs Viewer (roles/logging.viewer) on the project
For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Enable logging for a job

To allow logs to be generated for a job, enable logs from Cloud Logging when you create the job:

If you create a job using the Google Cloud console, logs from Cloud Logging are always enabled.
If you create a job using the gcloud CLI or the Batch API, logs from Cloud Logging are disabled by default. To enable logs from Cloud Logging, include the following configuration for the logsPolicy field while creating the job:
```
{
    ...
    "logsPolicy": {
        "destination": "CLOUD_LOGGING"
    }
    ...
}
```

Write and generate logs for a job

When logs from Cloud Logging are enabled for a job, Cloud Logging automatically generates any of the logs that are written for the job. Specifically, Batch jobs can have the following log types:

agent logs (batch_agent_logs): logs for activities from the Batch service agent.

Batch automatically writes agent logs for every job that has enabled logging.
task logs (batch_task_logs): logs for any data that you've configured a job's runnables to write to the standard output (stdout) stream or standard error (stderr) stream.

You can optionally write task logs for each job that has enabled logging.

View logs for a job

You can view a job's logs using the Google Cloud console, the gcloud CLI, Logging API, Go, Java, Python, or C++.

Console

To view a job's logs using the Google Cloud console, do the following:

In the Google Cloud console, go to the Job list page.

Go to Job list
In the Job name column, click the name of a job. The Job details page opens.
Click the Logs tab. Batch displays all logs associated with the job.
Optional: To filter the logs do any of the following:
- Enter a filter in the Filter field.
- In the Severity list, select a severity.
- Click View in Logs Explorer to build a query in the Logs Explorer using the Batch filter parameters.

gcloud

To view logs using the gcloud CLI, use the gcloud logging read command:

gcloud logging read "QUERY"

where QUERY is a query for Batch logs that contains Batch filter parameters.

API

To view logs using the Logging API, use the entries.list method:

POST https://logging.googleapis.com/v2/entries:list
{
    "resourceNames": [
        "projects/PROJECT_ID"
    ],
    "filter": "QUERY"
    "orderBy": "timestamp desc"
}

Replace the following:

PROJECT_ID: the project ID of your project.
QUERY: a query for Batch logs that contains Batch filter parameters.

Go

For more information, see the Batch Go API reference documentation.

To authenticate to Batch, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

import (
	"context"
	"fmt"
	"io"

	batch "cloud.google.com/go/batch/apiv1"
	"cloud.google.com/go/batch/apiv1/batchpb"
	"cloud.google.com/go/logging"
	"cloud.google.com/go/logging/logadmin"
	"google.golang.org/api/iterator"
)

// Retrieve the logs written by the given job to Cloud Logging
func printJobLogs(w io.Writer, projectID string, job *batchpb.Job) error {
	// projectID := "your_project_id"

	ctx :=