This document is intended for data product owners who want to create and configure data products in Knowledge Catalog (formerly Dataplex Universal Catalog). When you define these resources, ensure their IDs adhere to Knowledge Catalog resource naming guidelines.
For more information about the architecture and key concepts of data products, see About data products.
Before you begin
Before you create data products, complete the following prerequisites.
Enable Gemini
Configuring Gemini in your data asset is an optional but highly recommended step before you create your first data product.
By default, creating a data product requires you to manually enter business descriptions, technical definitions, and onboarding documentation for your assets. When you enable Gemini integration, Knowledge Catalog leverages AI assistance to automatically analyze your schemas and data scan results to generate the following:
- Business documentation: Generates documentation templates and clear descriptions for your data product and its individual data assets.
- Insights and sample queries: Constructs ready-to-use sample queries based on the asset's schema layout, enabling data consumers to immediately start querying the product upon approval.
If you choose not to enable Gemini, you can skip this section. However, you must manually provide all asset metadata and query templates during creation.
For more information about enabling Gemini in BigQuery, see Set up Gemini in BigQuery.
Enable APIs
Enable the Dataplex, BigQuery APIs.
Roles required to enable APIs
To enable APIs, you need the serviceusage.services.enable permission. If you
created the project, then you likely already have this permission through the
Owner role (roles/owner). Otherwise, you can get this permission through the
Service Usage Admin role (roles/serviceusage.serviceUsageAdmin).
Learn how to grant roles.
Create data assets
Ensure that your data assets (for example, BigQuery datasets, tables, and views) are created and populated.
For more information about creating data assets, see the following documents:
- Create BigQuery datasets
- Create and use BigQuery tables
- Create logical views in BigQuery
- Create materialized views in BigQuery
Configure identities
Identify or create the Google Groups or service accounts that you want to configure in your data product.
Required roles
This section outlines the minimum IAM roles required for the following primary sections:
Data product owners: users who create, configure, and manage data products and their associated assets
Data product consumers: users who search for, view, and request access to published data products
Required roles for data product owners
To get the permissions that you need to create and manage data products, ask your administrator to grant you the following IAM roles on the project:
-
Full permissions to create, update, delete, manage permissions, and approve or reject access requests for data products:
Dataplex Data Products Admin (
roles/dataplex.dataProductsAdmin) -
Update and manage permissions, and approve or reject access requests for data products:
Dataplex Data Products Editor (
roles/dataplex.dataProductsEditor) -
Add metadata aspects (such as
schema,overview,contacts, andqueries): Dataplex Entry and EntryLink Owner (roles/dataplex.entryOwner) -
Search for and add assets:
Dataplex Catalog Viewer (
roles/dataplex.catalogViewer) -
Edit system aspect types (such as
overview,contact,contract, andqueries): Dataplex Catalog Editor (roles/dataplex.catalogEditor) -
Create or retrieve insights data scans for automated documentation and insights generation:
Dataplex DataScan Admin (
roles/dataplex.dataScanAdmin)
For more information about granting roles, see Manage access to projects, folders, and organizations.
These predefined roles contain the permissions required to create and manage data products. To see the exact permissions that are required, expand the Required permissions section:
Required permissions
The following permissions are required to create and manage data products:
-
Create a data product:
dataplex.dataProducts.create -
List data products in a project:
dataplex.dataProducts.list -
Get or view data product:
dataplex.dataProducts.get -
Edit an existing data product:
dataplex.dataProducts.update -
Delete data product:
dataplex.dataProducts.delete -
Approve data product access request:
dataplex.dataProducts.approve -
Search for a data product using Knowledge Catalog:
-
dataplex.dataProducts.get -
dataplex.projects.search
-
-
Create data product access request:
dataplex.dataProducts.get -
Create a data asset:
dataplex.dataAssets.create -
List data assets within a data product:
dataplex.dataAssets.list -
Get data asset:
dataplex.dataAssets.get -
Edit an existing data asset:
dataplex.dataAssets.update -
Delete data asset:
dataplex.dataAssets.delete -
Create a data scan:
dataplex.datascans.create -
List all data scans:
dataplex.datascans.list -
Get a data scan:
dataplex.datascans.get -
Run a data scan:
dataplex.datascans.run -
Edit the
overviewsystem aspect type:dataplex.entryGroups.useOverviewAspect -
Edit the
refresh cadencesystem aspect type:dataplex.entryGroups.useRefreshCadenceAspect -
Edit the
queriessystem aspect type:dataplex.entryGroups.useQueriesAspect
You might also be able to get these permissions with custom roles or other predefined roles.
Required roles for data product consumers
For the data product consumers to search for, view, and request access to data products, as a data product owner, you must ensure the data product is discoverable. To do this, grant the data product consumers the following IAM roles on the data product:
- Search for data products and request access to them:
Dataplex Data Product Consumer (
dataplex.dataProductsConsumer) and Dataplex Catalog Viewer (roles/dataplex.catalogViewer) - Read-only access to view data product definitions and metadata:
Dataplex Data Product Viewer (
dataplex.dataProductsViewer)
Create and configure a data product
Creating a data product involves the following high-level tasks:
Create a data product
This mandatory initial step requires defining core details such as a unique data product name, description, region where the data product is created, and contact details.
Optional: Add assets
In this phase, you select assets to include in the data product. A key constraint is that assets must reside in the same region as the data product itself. You can add up to 10 assets at a time, with a total maximum of 50 assets allowed per data product.
For the list of supported assets, see Assets supported.
Optional: Configure access groups and asset permissions
In this optional phase, you simplify access control by creating access groups. These access groups act as user-friendly aliases (for example,
AnalystorReader) for underlying Google Groups and service accounts. You then assign permissions by selecting a specific IAM role and mapping it to an access group for a specific asset.Optional: Add contract and aspect details
In this phase, you enhance governance and data discoverability by attaching metadata frameworks. You can add a contract to formally communicate your data refresh cadence, specifying parameters such as refresh frequency, timing, and variance thresholds. You can also attach custom aspects to provide additional business or technical metadata for your data product.
Optional: Add additional details
In this final phase, you add rich text documentation, such as user onboarding guides, business definitions, and sample queries, to help consumers interact with the data product immediately upon approval.
To create and configure a data product, complete the steps in the following sections:
Create a data product
Console
In the Google Cloud console, go to the Knowledge Catalog Data products page.
Click Create.
In the Create data products pane, enter the following details:
- Data product name: Enter a unique name for your data product.
- Data product ID: This is an auto-generated unique identifier. You can edit this field.
- Project ID: This is a unique identifier of the project where the data product is created. Browse and select the project.
- Region: Select the region or multi-region where the data product is created.
- Data product icon: Browse and select an icon to visually identify the data product. This is optional.
- Description: Enter a brief description of the data product.
Contacts: Provide the point of contact information for governance and approval workflows:
- Data product owner(s) email address: Enter the email address of the data product owners.
- Data product approver(s) email address: Enter the email address of the designated approvers responsible for signing off on access requests or modifications.
Labels: Add key-value labels to organize your resources. This is optional.
Click Create data product.
Terraform
To create a data product, use the
google_dataplex_data_product
and IAM
resources.
resource "google_dataplex_data_product" "default" {
project = "PROJECT_ID"
location = "LOCATION"
data_product_id = "DATA_PRODUCT_ID"
display_name = "DISPLAY_NAME"
description = "DESCRIPTION"
owner_emails = ["EMAIL_IDs"]
# Optional: Configures designated approvers for access requests
access_approval_config {
approver_emails = ["APPROVER_EMAIL_IDs"]
}
# Optional: Base64-encoded icon (Max 3.0MiB)
icon = "ICON_BASE64_STRING"
}
resource "google_dataplex_data_product_iam_member" "consumer_binding" {
project = "PROJECT_ID"
location = "LOCATION"
data_product_id = "DATA_PRODUCT_ID"
role = "roles/dataplex.dataProductsConsumer"
member = "MEMBER"
}
Replace the following:
- PROJECT_ID: the ID of your Google Cloud project.
- LOCATION: the region in which you want to create the data product.
- DATA_PRODUCT_ID: a unique ID for your data product.
- DISPLAY_NAME: a user-friendly name for your data product.
- DESCRIPTION: a brief description of the data product.
- EMAIL_IDs: comma-separated email addresses of
the data product owners, for
example—
["user1@example.com", "user2@example.com"]. - APPROVER_EMAIL_IDs: comma-separated email addresses of the designated approvers responsible for signing off on access requests or modifications for this data product.
- ICON_BASE64_STRING: the raw base64-encoded image data representing the data product. The maximum size is 3.0 MiB. Don't include the data URL scheme prefix.
- MEMBER: the user, service account, or group
receiving the consumer role (for example,
group:all-analysts@example.com).
C#
Before trying this sample, follow the C# setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog C# API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
C#
Go
Before trying this sample, follow the Go setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Go API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Go
Java
Before trying this sample, follow the Java setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Java API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Java
Node.js
Before trying this sample, follow the Node.js setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Node.js API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Node.js
Python
Before trying this sample, follow the Python setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Python API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Python
REST
To create a data product, use the
dataProducts.create
method.
For example, send the following POST request:
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
-d '{"display_name": "DISPLAY_NAME", "owner_emails": ["EMAIL_IDs"], "access_approval_config": { "approver_emails": ["APPROVER_EMAIL_IDs"]} }' \
https://dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/dataProducts?data_product_id=DATA_PRODUCT_ID
Replace the following:
- DISPLAY_NAME: a user-friendly name for your data product
- EMAIL_IDs: comma-separated email addresses of the data product owners
- APPROVER_EMAIL_IDs: comma-separated email addresses of the designated approvers responsible for signing off on access requests or modifications.
- PROJECT_ID: the ID of your Google Cloud project
- LOCATION: the region in which you want to create the data product
- DATA_PRODUCT_ID: a unique ID for your data product
Optional: Add assets
You can add various data assets, such as BigQuery tables, views, datasets, and models to your data product. For the list of supported assets, see Assets supported.
Console
In the Add assets pane, click +Add.
Search for and select the assets that you want to add to your data product. The assets you select must reside in the same region as the data product.
If you have necessary permissions, you can view the metadata of assets by clicking the asset.
To refine the search results, use Filters.
After you select the assets, click Add.
Click Continue.
Terraform
To add a data asset to your data product, use the
google_dataplex_data_product_data_asset
resource.
resource "google_dataplex_data_product_data_asset" "example_asset" {
project = "PROJECT_ID"
location = "LOCATION"
data_product_id = "DATA_PRODUCT_ID"
data_asset_id = "DATA_ASSET_ID"
resource = "RESOURCE_NAME"
}
Replace the following:
- PROJECT_ID: the ID of your Google Cloud project
- LOCATION: the region where the data product exists
- DATA_PRODUCT_ID: the ID of the data product
- DATA_ASSET_ID: a unique ID for this data asset within the data product
- RESOURCE_NAME: the
full resource name of the data asset (for
example,
//bigquery.googleapis.com/projects/PROJECT_ID/datasets/DATASET_ID/tables/TABLE_ID)
C#
Before trying this sample, follow the C# setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog C# API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
C#
Go
Before trying this sample, follow the Go setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Go API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Go
Java
Before trying this sample, follow the Java setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Java API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Java
Node.js
Before trying this sample, follow the Node.js setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Node.js API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Node.js
Python
Before trying this sample, follow the Python setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Python API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Python
REST
To add a data asset to your data product, use the
dataAssets.create
method.
For example, send the following POST request:
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
-d '{"resource": "RESOURCE_NAME"}' \
https://dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/dataProducts/DATA_PRODUCT_ID/dataAssets?data_asset_id=DATA_ASSET_ID
Replace the following:
- RESOURCE_NAME: the
full resource name of the data asset (for
example,
//bigquery.googleapis.com/projects/PROJECT_ID/datasets/DATASET_ID/tables/TABLE_ID) - PROJECT_ID: the ID of your Google Cloud project
- LOCATION: the region where the data product exists
- DATA_PRODUCT_ID: the ID of the data product
- DATA_ASSET_ID: a unique ID for this data asset within the data product
Optional: Configure access groups and asset permissions
In the Configure access groups and asset permissions pane, you can create access groups and assign permissions to assets.
Configure access groups
Console
Click Add access group.
In the Access group name field, enter a name for the access group. For example,
Analyst.In the Access group description field, enter a description for the access group.
In the Access group identifier field, enter the email address of a Google Group that you want to assign to this access group
Data product consumers who request access for themselves are added as members to the mapped Google Group.
For more information about creating Google Groups, see Create and manage Google Groups in the Google Cloud console.
In the Access group service account field, enter the email address of a service account that you want to assign to this access group.
Data product consumers who request access for their service accounts are granted the Service Account Token Creator (
roles/iam.serviceAccountTokenCreator) IAM role to impersonate the data producer service account mapped to the access group.For more information about creating service accounts, see Create service accounts.
Click Done.
To add another access group, click Add access group and repeat the steps.
You can add a maximum of three access groups per data product.
Click Save.
Terraform
To define access groups for your data product, use the access_groups nested
block within the
google_dataplex_data_product
resource.
For example, use the following configuration:
resource "google_dataplex_data_product" "example_data_product" {
project = "PROJECT_ID"
location = "LOCATION"
data_product_id = "DATA_PRODUCT_ID"
display_name = "DISPLAY_NAME"
owner_emails = ["EMAIL_IDs"]
access_groups {
id = "analyst" # Internal identifier for configuration
group_id = "analyst" # Unique identifier of the access group, should be same as the 'id'
display_name = "Business Analyst"
description = "Access group for regional analysts"
principal {
google_group = "analyst-team@example.com"
}
# Service account principal
access_groups {
id = "scientist"
group_id = "scientist"
display_name = "Data Scientist"
principal {
service_account = "SERVICE_ACCOUNT_EMAIL_ADDRESS"
}
}
}
Replace the following:
- PROJECT_ID: the ID of your Google Cloud project
- LOCATION: the region where the data product exists
- DATA_PRODUCT_ID: a unique ID for the data product
- DISPLAY_NAME: a user-friendly name for your data product
- EMAIL_IDs: comma-separated email addresses of
the data product owners, for
example—
["user1@example.com", "user2@example.com"] - SERVICE_ACCOUNT_EMAIL_ADDRESS: the email address of the service account
C#
Before trying this sample, follow the C# setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog C# API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
C#