Export metadata

This document explains how to export metadata from Knowledge Catalog (formerly Dataplex Universal Catalog) using a metadata export job for use in external systems. You can export metadata to analyze it with tools such as BigQuery, programmatically process large volumes, or integrate it with custom applications.

This guide assumes you are familiar with Knowledge Catalog metadata concepts, including entry groups, entry types, and aspect types.

Job scope

The job scope defines the metadata to export. You must provide one of the following job scopes for each metadata export job:

  • Organization: exports the metadata that belongs to your organization.
  • Projects: exports the metadata that belongs to the specified projects.
  • Entry groups: exports the metadata that belongs to the specified entry groups.

You can further restrict the scope by specifying the entry types or aspect types to include in the job. The job exports only the entries and aspects that belong to these entry types and aspect types.

VPC Service Controls

Knowledge Catalog uses VPC Service Controls to provide additional security for metadata export jobs. The project that the job belongs to determines the VPC Service Controls perimeter, as follows:

  • If you set the job scope to the organization level, the following things happen:
    • The export scope is the organization that the job belongs to.
    • Only the entries that are within the VPC Service Controls perimeter are exported.
    • Any projects that are within the job's organization but outside the VPC Service Controls perimeter are excluded.
  • If you set the job scope to projects or entry groups, the projects or entry groups must be in the same VPC Service Controls perimeter as the job. If any of the projects or entry groups violate VPC Service Controls rules, the job fails.

Before you begin

Before you export metadata, complete the tasks in this section.

Required roles for end users

To get the permissions that you need to manage metadata export jobs, ask your administrator to grant you the following IAM roles: