About metadata change feeds

This document provides an overview of Knowledge Catalog (formerly Dataplex Universal Catalog) metadata change feeds. These metadata change feeds let you track metadata changes in your Knowledge Catalog instance in near real-time and build event-driven workflows based on those changes.

Automated monitoring of metadata changes

In Knowledge Catalog, an entry represents a data asset, such as a BigQuery table, and an aspect is a set of related metadata fields attached to an entry that describes it. When an entry or aspect is created, updated, or deleted, Knowledge Catalog publishes a notification message to a Pub/Sub topic that you specify. These notifications, also called a metadata change feed, contain information about the change. This includes when the change happened, what resource changed, and the type of change. For more information about entries and aspects, see About metadata management in Knowledge Catalog.

The following architecture diagram shows how Knowledge Catalog captures metadata changes (create, update, delete) and pushes them to Pub/Sub for downstream event-driven workflows.

To control which changes generate notifications, you can configure a metadata change feed to monitor specific resources. You do this by specifying a scope, such as your entire organization, specific projects, or specific entry groups. While scope lets you define which resources to monitor, you can use filters to further refine when Knowledge Catalog sends notifications. For example, you might want to receive notifications only when tables of type bigquery-table are updated, but not when they're created or deleted. To do this, you can apply one or more filters to your metadata change feed based on entry type, aspect type, or change type (CREATE, UPDATE, or DELETE).

For example, an online retail company uses BigQuery to manage product inventory in a dedicated project. To monitor only schema changes in their inventory tables, they create a metadata change feed with the project as scope, and apply filters for entry_type=bigquery-table and change_type=UPDATE. If the schema of a critical table, such as product_stock, updates, this change generates an UPDATE notification that matches the metadata change feed's filter. The metadata change feed then sends a notification to a Pub/Sub topic. An automated workflow subscribed to this Pub/Sub topic can immediately pause downstream reporting pipelines and alert the inventory management team to prevent decisions based on inconsistent data.

Use cases

You can use metadata change feeds for various purposes, including:

• Metadata sync: continuously sync Knowledge Catalog metadata changes to an external or third-party data catalog or search index.
• Policy enforcement: automatically apply or update security policies when a data classification aspect changes on an entry.
• Data quality automation: trigger a data quality scan or alert the data owner when the schema of a table changes.
• ETL/ELT triggering: start a data transformation job when a new table entry is created or updated.
• Auditing: log all metadata changes to an audit table for compliance purposes.

Terminology

A metadata change feed is a Knowledge Catalog resource that monitors metadata changes (create, update, delete) for entries and aspects, and sends notifications to a Pub/Sub topic. In the API, this resource is called metadataFeeds (projects/PROJECT_ID/locations/LOCATION/metadataFeeds/FEED_ID).

You configure a metadata change feed by defining its scope, filters, and destination. When a metadata change occurs that matches the metadata change feed's scope and filters, Knowledge Catalog publishes a notification message to the destination Pub/Sub topic.

Metadata change feed configuration

You can configure a metadata change feed by defining the following:

• Scope: the set of resources to monitor for changes, such as your entire organization, specific projects, or specific entry groups. In the API, you specify the resource name. The following example shows the resource name format for an entry group: projects/PROJECT_ID/locations/LOCATION/entryGroups/ENTRY_GROUP_ID.

• Filters: criteria to filter which changes generate notifications based on entry type, aspect type, or change type (CREATE, UPDATE, or DELETE). In the API, you specify the resource name. The following example shows the resource name format for an entry type: projects/PROJECT_ID/locations/global/entryTypes/ENTRY_TYPE. If you don't specify any filters, then all change types (CREATE, UPDATE, and DELETE) within the feed's scope generate notifications.

• Destination: the Pub/Sub topic where Knowledge Catalog publishes notification messages. In the API, you specify the topic name. The following example shows the resource name format for a Pub/Sub topic: projects/PROJECT_ID/topics/TOPIC_ID.

The following example shows a metadata change feed configured to monitor projects PROJECT_ID_1 and PROJECT_ID_2 for CREATE events, sending notifications to TOPIC_ID:

{
  "scope": {
    "projects": [
      "projects/PROJECT_ID_1",
      "projects/PROJECT_ID_2"
    ]
  },
  "filter": {
    "changeTypes": [
      "CREATE"
    ]
  },
  "pubsubTopic": "projects/PROJECT_ID_PUBSUB/topics/TOPIC_ID"
}

For instructions on how to create and manage metadata change feeds, see Receive notifications with metadata change feeds.

Notification message format

When a metadata change triggers a notification, Knowledge Catalog publishes a message to the specified Pub/Sub topic. The change event detail is captured in a Pub/Sub message. The message consists of attributes for filtering, and a data payload with details of the change.

For more information about consuming these messages, see Consume notification messages.

Attributes

The attributes enable filtering messages within the topic. You can filter messages on subscription using Pub/Sub subscription filters.

The following fields are provided in the attributes:

• timestamp: The timestamp of when the change happened.
• entry_name: The resource name of the entry, in the format projects/PROJECT_ID/locations/LOCATION/entryGroups/