Managed Service for Apache Kafka collects metrics that you can use to monitor your Kafka clusters. This page describes how to view these metrics in the Google Cloud console.
Overview
Managed Service for Apache Kafka exports several metrics available in the open-source Kafka distribution, as well as service-specific metrics like consumer group offset lag.
The metrics are organized into four resource categories:
Cluster: These metrics are useful for maintaining the overall health of a cluster.
Topic: These metrics include publisher and consumer rates and errors. They monitor the overall health of Kafka applications, and issues specific to a broker.
Topic Partition: These metrics are intended for monitoring and debugging performance problems specific to individual partitions, such as uneven key distribution.
Topic Partition Consumer Group: These metrics monitor the health of consumer applications, primarily consumer lag. Open-source Kafka error metrics for consumer groups are not available by partition, but only at the topic level.
Some metrics can be grouped by broker index. Based on the broker index, you can look up the zone where that broker is provisioned. For more information, see View brokers.
View metrics for a Kafka cluster
You can view the metrics in the following ways:
The Cluster details page includes monitoring dashboards for clusters, topics, and consumer groups. These dashboards contain predefined charts that let you see the overall health and performance of your cluster.
You can use Metrics Explorer to view all available metrics, create custom charts, or aggregate metrics across multiple clusters.
Required roles and permissions
To get the permissions that
you need to view monitoring charts,
ask your administrator to grant you the
Managed Kafka Viewer (roles/managedkafka.Viewer) IAM role on your project.
For more information about granting roles, see Manage access to projects, folders, and organizations.
You might also be able to get the required permissions through custom roles or other predefined roles.
Use the monitoring dashboards
To view the monitoring dashboards for a Managed Service for Apache Kafka cluster, perform the following steps:
In the Google Cloud console, go to the Clusters page.
Click the name of the cluster.
To view metrics for the cluster, select the Monitoring tab.
To view metrics for a topic in the cluster:
Select the Resources tab.
In the Topics list, click the name of the topic.
In the Topic details page, select the Monitoring tab.
To view metrics for a consumer group in the cluster:
Select the Resources tab.
In the Consumer groups list, click the name of the consumer group.
In the Consumer group details page, select the Monitoring tab.
For more information, see View a Kafka cluster.
Use Metrics Explorer
To view Managed Service for Apache Kafka metrics by using Metrics Explorer, perform the following steps:
In the Google Cloud console, go to the Metrics explorer page.
In the Configuration section, click Select a metric.
In the filter, enter
Apache Kafka.In Active resources, select one of the following:
Apache Kafka Cluster
Apache Kafka Topic
Apache Kafka Topic Partition
Apache Kafka Topic Partition Consumer Group
Select a metric and click Apply.
For more information about Metrics Explorer, see Create charts with Metrics Explorer.