Use auto data quality

This document describes how to use Knowledge Catalog (formerly Dataplex Universal Catalog) data quality scans to measure, monitor, and manage the quality of your data. Data quality scans help you automate the process of validating your data for completeness, validity, and consistency.

With data quality scans, you can define rules to check for missing values, ensure values match a regular expression or belong to a set, verify uniqueness, or use custom SQL for more complex validations such as anomaly detection. This document explains how to create and manage data quality scans.

Learn more about data quality scans. Learn more about reusing data quality rules across multiple scans.

Before you begin

  1. Enable the Dataplex API.

    Roles required to enable APIs

    To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

    Enable the API

  2. Optional: To generate recommendations for data quality rules based on the results of a data profile scan, create and run the data profile scan.

Required roles and permissions

This section describes the IAM roles and permissions needed to use Knowledge Catalog data quality scans.

User roles and permissions

To get the permissions that you need to run and manage data quality scans, ask your administrator to grant you the following IAM roles:

For more information about granting roles, see Manage access to projects, folders, and organizations.

These predefined roles contain the permissions required to run and manage data quality scans. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to run and manage data quality scans:

  • Run a data quality scan on a BigQuery table:
    • bigquery.jobs.create on the project to run scan jobs
    • bigquery.tables.get on the BigQuery table to be scanned
    • bigquery.tables.getData on the BigQuery table to be scanned
  • Publish data quality scan results to Knowledge Catalog:
    • bigquery.tables.update on the scanned table
    • dataplex.entryGroups.useDataQualityScorecardAspect on the @bigquery entry group in the same location as the table
  • Create a DataScan: dataplex.datascans.create on the project
  • Delete a DataScan: dataplex.datascans.delete on the project
  • View DataScan metadata: dataplex.datascans.get on the project
  • View DataScan details including rules and results: dataplex.datascans.getData on the project
  • List DataScans: dataplex.datascans.list on the project
  • Run a DataScan: dataplex.datascans.run on the project
  • Update a DataScan: dataplex.datascans.update on the project
  • Get or set IAM policy on a DataScan:
    • dataplex.datascans.getIamPolicy on the project
    • dataplex.datascans.setIamPolicy on the project

You might also be able to get these permissions with custom roles or other predefined roles.

If you need to access columns protected by BigQuery column-level access policies, then you also need permissions for those columns.

Knowledge Catalog service account roles and permissions

If you haven't created any data quality or data profile scans or you don't have a Knowledge Catalog lake in this project, create a service identifier by running: gcloud beta services identity create --service=dataplex.googleapis.com. This command returns a Knowledge Catalog service identifier if it exists.

To ensure that the Knowledge Catalog service account of the project containing the data quality scan has the necessary permissions to read data from various sources and export results, ask your administrator to grant the following IAM roles to the Knowledge Catalog service account of the project containing the data quality scan:

  • Read BigQuery table data: BigQuery Data Viewer (roles/bigquery.dataViewer) on BigQuery tables to be scanned and any other tables referenced in rules
  • Read Iceberg REST Catalog table data: BigLake Viewer (roles/biglake.viewer) on Iceberg REST Catalog tables to be scanned and any other tables referenced in rules
  • Export scan results to a BigQuery table: BigQuery Data Editor (roles/bigquery.dataEditor) on the results dataset and table
  • Scan BigQuery data organized in a Knowledge Catalog lake:
  • Scan a BigQuery external table from Cloud Storage: Storage Object Viewer (roles/storage.objectViewer) on the Cloud Storage bucket

For more information about granting roles, see Manage access to projects, folders, and organizations.

These predefined roles contain the permissions required to read data from various sources and export results. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to read data from various sources and export results:

  • Read BigQuery table data:
    • bigquery.tables.get on BigQuery tables
    • bigquery.tables.getData on BigQuery tables
  • Export scan results to a BigQuery table:
    • bigquery.datasets.get on results dataset and table
    • bigquery.tables.create on results dataset and table
    • bigquery.tables.get on results dataset and table
    • bigquery.tables.getData on results dataset and table
    • bigquery.tables.update on results dataset and table