Use data quality tasks

This document shows you how to create Knowledge Catalog data quality tasks that let you schedule and run data quality checks for your built-in and external BigQuery tables.

For more information, see Data quality tasks overview.

Before you begin

This document assumes that you have an existing Knowledge Catalog lake to create the data quality task in.

Enable Google APIs and services

1. Enable the Dataproc API.

   Enable the API

2. Enable Private Google Access for your network and subnetwork. Enable Private Google Access on the network that you plan to use with Knowledge Catalog data quality tasks. If you don't specify a network or subnetwork when you create the Knowledge Catalog data quality task, Knowledge Catalog uses the default subnet. In that case, you need to enable Private Google Access on the default subnet.

Create a specification file

Knowledge Catalog uses open source CloudDQ as the driver program. Knowledge Catalog data quality check requirements are defined within CloudDQ YAML specification files.

As input to the data quality task, you can have a single YAML file or a single zip archive containing one or more YAML files. It's recommended that you capture the data quality check requirements in separate YAML specification files, with one file for each section.

To prepare a specification file, do the following:

1. Create one or more CloudDQ YAML specification files that define your data quality check requirements. For more information about the required syntax, see the About the specification file section of this document.

   Save the YAML specification file in .yml or .yaml format. If you create multiple YAML specification files, save all of the files in a single zip archive.

2. Create a Cloud Storage bucket.
3. Upload the specification file to the Cloud Storage bucket.

About the specification file

Your CloudDQ YAML specification file needs to have the following sections:

• Rules (defined in the top-level rules YAML node): A list of rules to run. You can create these rules from predefined rule types, such as NOT_NULL and REGEX, or you can extend them with custom SQL statements such as CUSTOM_SQL_EXPR and CUSTOM_SQL_STATEMENT. The CUSTOM_SQL_EXPR statement flags any row that custom_sql_expr evaluated to False as a failure. The CUSTOM_SQL_STATEMENT statement flags any value returned by the whole statement as a failure.

• Row filters (defined in the top-level row_filters YAML node): SQL expressions returning a boolean value that define filters to fetch a subset of data from the underlying entity subject for validation.

• Rule bindings (defined in the top-level rule_bindings YAML node): Defines rules and rule filters to apply to the tables.

• Rule dimensions (defined in the rule_dimensions YAML node): Defines the allowed list of data quality rule dimensions that a rule can define in the corresponding dimension field.

  For example:

  rule_dimensions:
    - consistency
    - correctness
    - duplication
    - completeness
    - conformance

  The dimension field is optional for a rule. The rule dimensions section is mandatory if dimension is listed on any rule.

For more information, see the CloudDQ reference guide and the sample specification files.

Create a dataset to store the results

• To store the results, create a BigQuery dataset.

  The dataset must be in the same region as the tables that you run the data quality task on.

  Knowledge Catalog uses this dataset, and it creates or reuses a table of your choice to store the results.

Create a service account

Create a service account that has the following Identity and Access Management (IAM) roles and permissions:

• Read access to the Cloud Storage path containing the YAML specifications. You can use the Storage Object Viewer role (roles/storage.objectViewer) on the Cloud Storage bucket.
• Read access to BigQuery datasets with data to be validated. You can use the BigQuery Data Viewer role (roles/bigquery.dataViewer).
• Write access to the BigQuery dataset to create a table (if necessary) and write the results to that table. You can use the BigQuery Data Editor role (roles/bigquery.dataEditor) at the dataset level.
• BigQuery Job User role (roles/bigquery.jobUser) at the project level in order to create BigQuery jobs in a project.
• The Dataplex Metadata Reader role (roles/dataplex.metadataReader) at the project or lake level.
• The Service Usage Consumer role (roles/serviceusage.serviceUsageConsumer) at the project level.
• The Dataproc Worker role (roles/dataproc.worker).
• The iam.serviceAccounts.actAs permission given to the user submitting the job.
• The Service account user role granted to the Knowledge Catalog lake service account. You can view the Knowledge Catalog lake service account in the Google Cloud console.

Use advanced settings

These steps are optional:

• BigQuery runs data quality checks in the current project by default. You can choose a different project to run the BigQuery jobs. Use the --gcp_project_id TASK_ARGS argument for the --execution-args property of the task.

• If the project ID specified to run BigQuery queries is different from the project in which the service account (specified by --execution-service-account) is created, ensure that the organization policy that disables cross-project service account usage (iam.disableServiceAccountCreation) is turned off. Also, ensure that the service account can access the BigQuery job schedule in the project where BigQuery queries are being run.

Limitations

All the tables specified for a given data quality task must belong to the same Google Cloud region.

Schedule a data quality task

export USER_CLOUDDQ_YAML_CONFIGS_GCS_PATH="USER_CLOUDDQ_YAML_CONFIGS_GCS_PATH"

# Google Cloud project where the Knowledge Catalog task is created.
export GOOGLE_CLOUD_PROJECT="GOOGLE_CLOUD_PROJECT"

# Google Cloud region for the Knowledge Catalog lake.
export DATAPLEX_REGION_ID="DATAPLEX_REGION_ID"

# Public Cloud Storage bucket containing the prebuilt data quality executable artifact. There is one bucket for each Google Cloud region.
export DATAPLEX_PUBLIC_GCS_BUCKET_NAME="dataplex-clouddq-artifacts-${DATAPLEX_REGION_ID}"

# The Knowledge Catalog lake where your task is created.
export DATAPLEX_LAKE_ID="DATAPLEX_LAKE_ID"

# The service account used for running the task. Ensure that this service account
has sufficient IAM permissions on your project, including
BigQuery Data Editor, BigQuery Job User,
Dataplex Editor, Dataproc Worker, and Service
Usage Consumer.
export DATAPLEX_TASK_SERVICE_ACCOUNT="DATAPLEX_TASK_SERVICE_ACCOUNT"

# If you want to use a different dataset for storing the intermediate data quality summary results
and the BigQuery views associated with each rule binding, use the following:
export CLOUDDQ_BIGQUERY_DATASET="CLOUDDQ_BIGQUERY_DATASET"

# The BigQuery dataset where the final results of the data quality checks are stored.
This could be the same as CLOUDDQ_BIGQUERY_DATASET.
export TARGET_BQ_DATASET="TARGET_BQ_DATASET"

# The BigQuery table where the final results of the data quality checks are stored.
export TARGET_BQ_TABLE="TARGET_BQ_TABLE"

# The unique identifier for the task.
export TASK_ID="TASK_ID"

gcloud dataplex tasks create \
    --location="${DATAPLEX_REGION_ID}" \
    --lake="${DATAPLEX_LAKE_ID}" \
    --trigger-type=ON_DEMAND \
    --execution-service-account="$DATAPLEX_TASK_SERVICE_ACCOUNT" \
    --spark-python-script-file="gs://${DATAPLEX_PUBLIC_GCS_BUCKET_NAME}/clouddq_pyspark_driver.py" \
    --spark-file-uris="gs://${DATAPLEX_PUBLIC_GCS_BUCKET_NAME}/clouddq-executable.zip","gs://${DATAPLEX_PUBLIC_GCS_BUCKET_NAME}/clouddq-executable.zip.hashsum","${USER_CLOUDDQ_YAML_CONFIGS_GCS_PATH}" \
    --execution-args=^::^TASK_ARGS="clouddq-executable.zip, ALL, ${USER_CLOUDDQ_YAML_CONFIGS_GCS_PATH}, --gcp_project_id='GOOGLE_CLOUD_PROJECT', --gcp_region_id='${DATAPLEX_REGION_ID}', --gcp_bq_dataset_id='${TARGET_BQ_DATASET}', --target_bigquery_summary_table='${GOOGLE_CLOUD_PROJECT}.${TARGET_BQ_DATASET}.${TARGET_BQ_TABLE}'," \
    "$TASK_ID"