Receive notifications with metadata change feeds

This document describes how to configure Knowledge Catalog (formerly Dataplex Universal Catalog) metadata change feeds to receive notifications through Pub/Sub when metadata within Knowledge Catalog is created, updated, or deleted.

For more information about metadata change feeds, see Metadata change feeds overview.

Before you begin

Familiarize yourself with Pub/Sub and the Dataplex API.

  1. Enable the Dataplex API and Pub/Sub APIs.

    Roles required to enable APIs

    To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

    Enable the APIs

  2. Create a Pub/Sub topic to receive notifications. For more information, see Create a topic.

  3. Install gcloud. Make sure the short alias of gcloud is set.

  4. Set an alias of gcurl. This creates a shortcut that includes your authentication token and sets the JSON content-type for API requests:

    alias gcurl='curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -H "Content-Type: application/json"'
    
  5. Set the DATAPLEX_API variable:

    DATAPLEX_API="dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION"
    

    Replace the following:

    • PROJECT_ID: your project ID where the Dataplex API is enabled
    • LOCATION: the location where the job runs—for example, us-central1, europe-west3, or asia-south1

Required roles and permissions

To configure metadata change feeds, ensure that you and the Knowledge Catalog service account have the required IAM roles and permissions.

User roles and permissions

To get the permissions that you need to export metadata and access metadata change feeds messages, ask your administrator to grant you the following IAM roles on the project or organization:

For more information about granting roles, see Manage access to projects, folders, and organizations.

These predefined roles contain the permissions required to export metadata and access metadata change feeds messages. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to export metadata and access metadata change feeds messages:

  • Export metadata:
    • dataplex.metadataFeeds.create on the project
    • dataplex.entryGroups.export on the resource defining the scope of the feed (Organization, Project, or Entry Group)
    • resourcemanager.projects.get on the project
    • resourcemanager.projects.list on the project
  • Access metadata change feeds messages:
    • pubsub.subscriptions.consume on the subscription
    • pubsub.topics.attachSubscription on the topic

You might also be able to get these permissions with custom roles or other predefined roles.

Knowledge Catalog service account roles and permissions

To ensure that the Knowledge Catalog service account has the necessary permission to publish metadata change feeds messages, ask your administrator to grant the Pub/Sub Publisher (roles/pubsub.publisher) and Pub/Sub Viewer (roles/pubsub.viewer) IAM role to the Knowledge Catalog service account on the Pub/Sub topic.

For more information about granting roles, see Manage access to projects, folders, and organizations.

This predefined role contains the pubsub.topics.publish and pubsub.topics.get permission, which is required to publish metadata change feeds messages.

Your administrator might also be able to give the Knowledge Catalog service account this permission with custom roles or other predefined roles.

Grant permissions to the Knowledge Catalog service account

The service agent in Knowledge Catalog is created when you enable the Dataplex API. You can identify the service agent by its email ID:

service-PROJECT_NUMBER@gcp-sa-dataplex.iam.gserviceaccount.com

Here, PROJECT_NUMBER is the project number of the project where you enabled the Dataplex API.

The Knowledge Catalog service account requires the ability to publish messages to your Pub/Sub topic and retrieve topic metadata. To grant these permissions, give the service account the Pub/Sub Publisher role (roles/pubsub.publisher) and the Pub/Sub Viewer role (roles/pubsub.viewer) on the Pub/Sub topic:

gcloud

Run the gcloud pubsub topics add-iam-policy-binding command:

gcloud pubsub topics add-iam-policy-binding TOPIC_ID \
    --member="serviceAccount:service-PROJECT_NUMBER@gcp-sa-dataplex.iam.gserviceaccount.com" \
    --role="roles/pubsub.publisher"

gcloud pubsub topics add-iam-policy-binding TOPIC_ID \
    --member="serviceAccount:service-PROJECT_NUMBER@gcp-sa-dataplex.iam.gserviceaccount.com" \
    --role="roles/pubsub.viewer"

Replace the following:

  • TOPIC_ID: the ID of your Pub/Sub topic to which the metadata change feed messages are published.
  • PROJECT_NUMBER: the project number of your Knowledge Catalog project where the service account is located.

Console

  1. In the Google Cloud console, go to the Pub/Sub Topics page.

    Go to topics

  2. Select the topic you're using for metadata feed notifications and if required click Show info panel.

  3. In the Permissions tab, click Add principal.

  4. In the New principals field, enter the Knowledge Catalog service account: service-PROJECT_NUMBER@gcp-sa-dataplex.iam.gserviceaccount.com.

  5. In the Assign roles field, select Pub/Sub Publisher.

  6. Click Add another role and select Pub/Sub Viewer.

  7. Click Save.

Create a metadata change feed

To control which changes generate notifications, you can configure a metadata change feed to monitor specific resources. You do this by specifying a scope, such as your entire organization, specific projects, or specific entry groups. While scope lets you define which resources to monitor, you can use filters to further refine when Knowledge Catalog sends notifications.

For more information, see Metadata change feed.

C#

C#

Before trying this sample, follow the C# setup instructions in the Knowledge Catalog quickstart using client libraries. For more information, see the Knowledge Catalog C# API reference documentation.

To authenticate to Knowledge Catalog, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

using Google.Api.Gax.ResourceNames;
using Google.Cloud.Dataplex.V1;
using Google.LongRunning;

public sealed partial class GeneratedCatalogServiceClientSnippets
{
    /// <summary>Snippet for CreateMetadataFeed</summary>
    /// <remarks>
    /// This snippet has been automatically generated and should be regarded as a code template only.
    /// It will require modifications to work:
    /// - It may require correct/in-range values for request initialization.
    /// - It may require specifying regional endpoints when creating the service client as shown in
    ///   https://cloud.google.com/dotnet/docs/reference/help/client-configuration#endpoint.
    /// </remarks>
    public void CreateMetadataFeedRequestObject()
    {
        // Create client
        CatalogServiceClient catalogServiceClient = CatalogServiceClient.Create();
        // Initialize request argument(s)
        CreateMetadataFeedRequest request = new CreateMetadataFeedRequest
        {
            ParentAsLocationName = LocationName.FromProjectLocation("[PROJECT]", "[LOCATION]"),
            MetadataFeed = new MetadataFeed(),
            MetadataFeedId = "",
            ValidateOnly = false,
        };
        // Make the request
        Operation<MetadataFeed, OperationMetadata> response = catalogServiceClient.CreateMetadataFeed(request);

        // Poll until the returned long-running operation is complete
        Operation<MetadataFeed, OperationMetadata> completedResponse = response.PollUntilCompleted();
        // Retrieve the operation result
        MetadataFeed result = completedResponse.Result;

        // Or get the name of the operation
        string operationName = response.Name;
        // This name can be stored, then the long-running operation retrieved later by name
        Operation<MetadataFeed, OperationMetadata> retrievedResponse = catalogServiceClient.PollOnceCreateMetadataFeed(operationName);
        // Check if the retrieved long-running operation has completed
        if (retrievedResponse.IsCompleted)
        {
            // If it has completed, then access the result
            MetadataFeed retrievedResult = retrievedResponse.Result;
        }
    }
}

Go

Go

Before trying this sample, follow the Go setup instructions in the Knowledge Catalog quickstart using client libraries. For more information, see the Knowledge Catalog Go API reference documentation.

To authenticate to Knowledge Catalog, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.


//go:build examples

package main

import (
	"context"

	dataplex "cloud.google.com/go/dataplex/apiv1"
	dataplexpb "cloud.google.com/go/dataplex/apiv1/dataplexpb"
)

func main() {
	ctx := context.Background()
	// This snippet has been automatically generated and should be regarded as a code template only.
	// It will require modifications to work:
	// - It may require correct/in-range values for request initialization.
	// - It may require specifying regional endpoints when creating the service client as shown in:
	//   https://pkg.go.dev/cloud.google.com/go#hdr-Client_Options
	c, err := dataplex.NewCatalogClient(ctx)
	if err != nil {
		// TODO: Handle error.
	}
	defer c.Close()

	req := &dataplexpb.CreateMetadataFeedRequest{
		// TODO: Fill request struct fields.
		// See https://pkg.go.dev/cloud.google.com/go/dataplex/apiv1/dataplexpb#CreateMetadataFeedRequest.
	}
	op, err := c.CreateMetadataFeed(ctx, req)
	if err != nil {
		// TODO: Handle error.
	}

	resp, err := op.Wait(ctx)
	if err != nil {
		// TODO: Handle error.
	}
	// TODO: Use resp.
	_ = resp
}

Java

Java

Before trying this sample, follow the Java setup instructions in the Knowledge Catalog quickstart using client libraries. For more information, see the Knowledge Catalog Java API reference documentation.

To authenticate to Knowledge Catalog, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

import com.google.cloud.dataplex.v1.CatalogServiceClient;
import com.google.cloud.dataplex.v1.CreateMetadataFeedRequest;
import com.google.cloud.dataplex.v1.LocationName;
import com.google.cloud.dataplex.v1.MetadataFeed;

public class SyncCreateMetadataFeed {

  public static void main(String[] args) throws Exception {
    syncCreateMetadataFeed();
  }

  public static void syncCreateMetadataFeed() throws Exception {
    // This snippet has been automatically generated and should be regarded as a code template only.
    // It will require modifications to work:
    // - It may require correct/in-range values for request initialization.
    // - It may require specifying regional endpoints when creating the service client as shown in
    // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
    try (CatalogServiceClient catalogServiceClient = CatalogServiceClient.create()) {
      CreateMetadataFeedRequest request =
          CreateMetadataFeedRequest.newBuilder()
              .setParent(LocationName.of("[PROJECT]", "[LOCATION]").toString())
              .setMetadataFeed(MetadataFeed.newBuilder().build())
              .setMetadataFeedId("metadataFeedId1633421160")
              .setValidateOnly(true)
              .build();
      MetadataFeed response = catalogServiceClient.createMetadataFeedAsync(request).get();
    }
  }
}

Node.js

Node.js

Before trying this sample, follow the Node.js setup instructions in the Knowledge Catalog quickstart using client libraries. For more information, see the Knowledge Catalog Node.js API reference documentation.

To authenticate to Knowledge Catalog, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

/**
 * This snippet has been automatically generated and should be regarded as a code template only.
 * It will require modifications to work.
 * It may require correct/in-range values for request initialization.
 * TODO(developer): Uncomment these variables before running the sample.
 */
/**
 *  Required. The resource name of the parent location, in the format
 *  `projects/{project_id_or_number}/locations/{location_id}`
 */
// const parent = 'abc123'
/**
 *  Required. The metadata job resource.
 */
// const metadataFeed = {}
/**
 *  Optional. The metadata job ID. If not provided, a unique ID is generated
 *  with the prefix `metadata-job-`.
 */
// const metadataFeedId = 'abc123'
/**
 *  Optional. The service validates the request without performing any
 *  mutations. The default is false.
 */
// const validateOnly = true

// Imports the Dataplex library
const {CatalogServiceClient} = require('@google-cloud/dataplex').v1;

// Instantiates a client
const dataplexClient = new CatalogServiceClient();

async function callCreateMetadataFeed() {
  // Construct request
  const request = {
    parent,
    metadataFeed,
  };

  // Run request
  const [operation] = await dataplexClient.createMetadataFeed(request);
  const [response] = await operation.promise();
  console.log(response);
}

callCreateMetadataFeed();

Python

Python

Before trying this sample, follow the Python setup instructions in the Knowledge Catalog quickstart using client libraries. For more information, see the Knowledge Catalog Python API reference documentation.

To authenticate to Knowledge Catalog, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

# This snippet has been automatically generated and should be regarded as a
# code template only.
# It will require modifications to work:
# - It may require correct/in-range values for request initialization.
# - It may require specifying regional endpoints when creating the service
#   client as shown in:
#   https://googleapis.dev/python/google-api-core/latest/client_options.html
from google.cloud import dataplex_v1


def sample_create_metadata_feed():
    # Create a client
    client = dataplex_v1.CatalogServiceClient()

    # Initialize request argument(s)
    metadata_feed = dataplex_v1.MetadataFeed()
    metadata_feed.pubsub_topic = "pubsub_topic_value"

    request = dataplex_v1.CreateMetadataFeedRequest(
        parent="parent_value",
        metadata_feed=metadata_feed,
    )

    # Make the request
    operation = client.create_metadata_feed(request=request)

    print("Waiting for operation to complete...")

    response = operation.result()

    # Handle the response
    print(response)

REST

To create a metadata change feed, use the projects.locations.metadataFeeds.create method.

Organization scope

Run the following command to create a metadata change feed that monitors the entire organization:

gcurl -X POST -d "$(cat <<EOF
{
  "scope": {
    "organizationLevel": true
  },
  "pubsubTopic": "projects/PROJECT_ID_PUBSUB/topics/TOPIC_ID"
}
EOF
)" "https://${DATAPLEX_API}/metadataFeeds?metadataFeedId=FEED_ID"

Replace the following:

  • PROJECT_ID_PUBSUB: the project ID where the Pub/Sub topic is located—for example, example-project
  • TOPIC_ID: the Pub/Sub topic ID to which the metadata change feed messages are published—for example, example-topic
  • FEED_ID: the metadata change feed ID that you want to create—for example, example-feed

Project scope

Run the following command to create a metadata change feed that monitors specific projects:

gcurl -X POST -d "$(cat <<EOF
{
  "scope": {
    "projects": [
      "projects/PROJECT_ID_1",
      "projects/PROJECT_ID_2"
    ]
  },
  "pubsubTopic": "projects/PROJECT_ID_PUBSUB/topics/TOPIC_ID"
}
EOF
)" "https://${DATAPLEX_API}/metadataFeeds?metadataFeedId=FEED_ID"

Replace the following:

  • PROJECT_ID_PUBSUB: the project ID where the Pub/Sub topic is located—for example, example-project
  • TOPIC_ID: the Pub/Sub topic ID to which the metadata change feed messages are published—for example, example-topic
  • FEED_ID: the metadata change feed ID that you want to create—for example, example-feed

Entry group scope

Run the following command to create a metadata change feed that monitors specific entry groups:

gcurl -X POST -d "$(cat <<EOF
{
  "scope": {
    "entryGroups": [
      "projects/PROJECT_ID/locations/LOCATION/entryGroups/ENTRY_GROUP_ID_1",
      "projects/PROJECT_ID/locations/LOCATION/entryGroups/ENTRY_GROUP_ID_2"
    ]
  },
  "pubsubTopic": "projects/PROJECT_ID_PUBSUB/topics/TOPIC_ID"
}
EOF
)" "https://${DATAPLEX_API}/metadataFeeds?metadataFeedId=FEED_ID"

Replace the following:

  • PROJECT_ID_PUBSUB: the project ID where the Pub/Sub topic is located—for example, example-project
  • LOCATION: the location where you want to create the feed—for example, us-central1
  • ENTRY_GROUP_ID: the entry group ID to monitor—for example, example-entry-group
  • TOPIC_ID: the Pub/Sub topic ID to which the metadata change feed messages are published—for example, example-topic
  • FEED_ID: the metadata change feed ID that you want to create—for example, example-feed

With filters

Run the following command to create a metadata change feed with filters (filtering by entry type):

gcurl -X POST -d "$(cat <<EOF
{
  "scope": {
    "projects": [
      "projects/PROJECT_ID_1",
      "projects/PROJECT_ID_2"
    ]
  },
  "filter": {
    "entryTypes": [
      "projects/PROJECT_ID/locations/global/entryTypes/bigquery-table"
    ]
  },
  "pubsubTopic": "projects/PROJECT_ID_PUBSUB/topics/TOPIC_ID"
}
EOF
)" "https://${DATAPLEX_API}/metadataFeeds?metadataFeedId=FEED_ID"

Replace the following:

  • PROJECT_ID_PUBSUB: the project ID where the Pub/Sub topic is located—for example, example-project
  • TOPIC_ID: the Pub/Sub topic ID to which the metadata change feed messages are published—for example, example-topic
  • FEED_ID: the metadata change feed ID that you want to create—for example, example-feed

Response:

{
  "name": "projects/PROJECT_ID/locations/LOCATION/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.dataplex.v1.OperationMetadata",
    "createTime": "2023-10-02T15:01:23Z",
    "target": "projects/PROJECT_ID/locations/LOCATION/metadataFeeds/FEED_ID",
    "verb": "create",
    "apiVersion": "v1"
  },
  "done": false
}

View metadata change feeds

You can view the details of a metadata change feed.

C#

C#

Before trying this sample, follow the C# setup instructions in the Knowledge Catalog quickstart using client libraries. For more information, see the Knowledge Catalog C# API reference documentation.

To authenticate to Knowledge Catalog, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

using Google.Cloud.Dataplex.V1;

public sealed partial class GeneratedCatalogServiceClientSnippets
{
    /// <summary>Snippet for GetMetadataFeed</summary>
    /// <remarks>
    /// This snippet has been automatically generated and should be regarded as a code template only.
    /// It will require modifications to work:
    /// - It may require correct/in-range values for request initialization.
    /// - It may require specifying regional endpoints when creating the service client as shown in
    ///   https://cloud.google.com/dotnet/docs/reference/help/client-configuration#endpoint.
    /// </remarks>
    public void GetMetadataFeedRequestObject()
    {
        // Create client
        CatalogServiceClient catalogServiceClient = CatalogServiceClient.Create();
        // Initialize request argument(s)
        GetMetadataFeedRequest request = new GetMetadataFeedRequest
        {
            MetadataFeedName = MetadataFeedName.FromProjectLocationMetadataFeed("[PROJECT]", "[LOCATION]", "[METADATA_FEED]"),
        };
        // Make the request
        MetadataFeed response = catalogServiceClient.GetMetadataFeed(request);
    }
}

Go

Go

Before trying this sample, follow the Go setup instructions in the Knowledge Catalog quickstart using client libraries. For more information, see the Knowledge Catalog Go API reference documentation.

To authenticate to Knowledge Catalog, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.


//go:build examples

package main

import (
	"context"

	dataplex "cloud.google.com/go/dataplex/apiv1"
	dataplexpb "cloud.google.com/go/dataplex/apiv1/dataplexpb"
)

func main() {
	ctx := context.Background()
	// This snippet has been automatically generated and should be regarded as a code template only.
	// It will require modifications to work:
	// - It may require correct/in-range values for request initialization.
	// - It may require specifying regional endpoints when creating the service client as shown in:
	//   https://pkg.go.dev/cloud.google.com/go#hdr-Client_Options
	c, err := dataplex.NewCatalogClient(ctx)
	if err != nil {
		// TODO: Handle error.
	}
	defer c.Close()

	req := &dataplexpb.GetMetadataFeedRequest{
		// TODO: Fill request struct fields.
		// See https://pkg.go.dev/cloud.google.com/go/dataplex/apiv1/dataplexpb#GetMetadataFeedRequest.
	}
	resp, err := c.GetMetadataFeed(ctx, req)
	if err != nil {