This document explains how Cloud Logging routes log entries received from Google Cloud services, your applications, and other cloud providers. By default, Cloud Logging routes all logs to log buckets in the project, folder, or organization where they originate. However, you can configure sinks to route data to custom buckets, Cloud Storage, BigQuery, or Pub/Sub for storage or analysis. Services such as Cloud Storage and Pub/Sub support exporting your log data to third-party tools.
At a high level, this is how Cloud Logging routes and stores log entries:
The previous figure applies to log buckets that are in a project. You can't create user-defined log buckets in folders or organizations. Also, you can't extend the retention period of any log bucket that is in a folder or organization.
Routing versus exporting log entries
Routing is the process of moving log entries received by a Google Cloud project, folder, or organization to a destination. A destination is a service that processes received log entries and performs an action. For example, you can route log entries to a log-bucket destination. This destination scans the log entries for error information and then writes them to storage.
Exporting is the process of moving log entries from Google Cloud to an external location. For example, you can route log entries to Pub/Sub and then export them to third-party tools.
Cloud Logging uses log sinks to route log entries to destinations. By default, your log entries are routed to one of two log buckets in your Google Cloud project, folder, or organization. You can create log sinks to route your log data to other destinations. You can also modify one of the system-created log sinks. This document describes these options.
The Logs Explorer lets you download log entries that are stored in log buckets to local storage. However, the download option is limited to 10,000 log entries. To export your log entries from Google Cloud, consider these options:
Configure a log sink to route incoming log entries to Pub/Sub, and then export that data from Google Cloud.
Copy log entries from a log bucket to Cloud Storage, and then export the data from Google Cloud. For information about how to copy logs, see Batch and route logs retroactively.
About Log Routers
Each Google Cloud project, billing account, folder, and organization has a Log Router, which manages the flow of log entries through resource-level sinks. A Log Router also manages the flow of a log entry through sinks that are in the entry's resource hierarchy. Sinks control how log entries are routed to destinations.
A Log Router stores a log entry temporarily. This behavior buffers against temporary disruptions and outages that might occur when a log entry flows through sinks. Temporary storage doesn't protect against configuration errors.
A Log Router's temporary storage is distinct from the longer term storage provided by Logging buckets.
Incoming log entries with timestamps that are more than the logs retention period in the past or that are more than 24 hours in the future are discarded.
About log sinks
When a log sink receives a log entry, it determines whether to ignore or route the log entry to the sink's destination. The log sink compares the log entry to its filters to make this decision. A sink's destination can be a project, a storage location, or a service such as Pub/Sub. For example, a sink can route log entries to a log bucket.
Log sinks belong to a given Google Cloud resource: Google Cloud projects, billing accounts, folders, and organizations. These resources also contain multiple log sinks. When a resource receives a log entry, every log sink in that resource independently evaluates the log entry. As a result, multiple log sinks can route the same log entry.
By default, log data is stored in the project where the data originates. However, there are several reasons why you might want to change this configuration:
- To centralize the storage of your log data.
- To join your log data with other business data.
- To organize your log data in a way that is useful to you.
- To stream your logs to other applications, other repositories, or third parties. For example, you might want to export your logs from Google Cloud so that you can view them on a third-party platform. To export your log entries, create a log sink that routes your log entries to Pub/Sub.
A misconfigured log sink doesn't route log entries. When a sink is misconfigured, log entries that report the detail of the error are written. Also, an email is sent to the Essential Contacts for the resource. For more information, see Troubleshoot: View errors.
Log sinks can't retroactively route log entries. That is, a log sink can't route a log entry that was received before the sink was created. Similarly, if a sink is misconfigured, then the sink only routes log entries that arrive after the configuration error is resolved. However, you can retroactively copy log data from a log bucket to Cloud Storage. For more information, see Copy logs.
Support for organizations and folders
To help you manage the log data in an organization or folder, you can do the following:
You can create aggregated sinks, which route log entries for an organization or folder and their children, to the destination specified by the sink. There are two types of aggregated sinks:
- Non-intercepting aggregated sinks