Create and manage resources in the Lakehouse runtime catalog

Lakehouse for Apache Iceberg supports resource management through the Lakehouse runtime catalog.

This shared catalog eliminates the requirement to maintain multiple catalogs for open source workloads across different data processing engines. Supported operations include creating, viewing, modifying, and deleting resources such as namespaces and tables.

Before you begin

  1. Verify that billing is enabled for your Google Cloud project.

  2. Enable the BigQuery, BigQuery Storage, and Managed Service for Apache Spark APIs.

    Roles required to enable APIs

    To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

    Enable the APIs

Required roles

To get the permissions that you need to manage Apache Iceberg resources in the Lakehouse runtime catalog, ask your administrator to grant you the following IAM roles on your project:

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Create catalog resources

The following sections describe how to create resources in the Lakehouse runtime catalog.

Create namespaces

Select one of the following options:

API

Use the datasets.insert method, and specify the ExternalCatalogDatasetOptions field in the dataset resource that you pass in.

{
  "datasetReference": {
    "projectId": "PROJECT_ID",
    "datasetId": "DATASET_ID"
  },
  "externalCatalogDatasetOptions": {
    "defaultStorageLocationUri": "URI",
    "parameters": {
      ...
    }
  },
  "location": "LOCATION"
}

Replace the following:

  • PROJECT_ID: the ID of the project that contains your target dataset.
  • DATASET_ID: the ID of your target dataset.
  • URI: the Cloud Storage URI for all tables in the dataset.
  • LOCATION: the BigQuery location that you want to create the dataset in.

Apache Spark SQL 

CREATE NAMESPACE SPARK_CATALOG.NAMESPACE;

Replace the following:

  • SPARK_CATALOG: the name of your Apache Spark catalog.
  • NAMESPACE: the name of your new namespace.

Terraform 

provider "google" {
  project = "PROJECT_ID"
}

resource "google_bigquery_dataset" "default" {
  dataset_id = "DATASET_ID"
  location   = "LOCATION"

  external_catalog_dataset_options {
    default_storage_location_uri = "URI"
    parameters = {
      ...
    }
  }
}

Replace the following:

  • PROJECT_ID: the ID of the project that contains your target dataset.
  • DATASET_ID: the ID of your target dataset.
  • LOCATION: the BigQuery location that you want to create the dataset in.
  • URI: the Cloud Storage URI for all tables in the dataset.

Create Apache Iceberg tables

Select one of the following options:

API

Use the tables.insert method, and specify the ExternalCatalogTableOptions field in the table resource that you pass in.

{
  "tableReference": {
    "projectId": "PROJECT_ID",
    "datasetId": "DATASET_ID",
    "tableId": "TABLE_ID"
  },
  "externalCatalogTableOptions": {
    "parameters": {
      "table_type": "iceberg",
      "metadata_location": "METADATA_URI"
    },
    "connection_id": "CONNECTION_ID"
  }
}

Replace the following:

  • PROJECT_ID: the ID of the project that contains your target table.
  • DATASET_ID: the ID of the dataset that contains your target table.
  • TABLE_ID: the ID of your target table.
  • METADATA_URI: the Cloud Storage URI for the latest Apache Iceberg metadata file. For example, gs://mybucket/mytable/metadata/1234.metadata.json.
  • CONNECTION_ID: the ID of your connection to Cloud Storage.

Apache Spark SQL 

CREATE TABLE SPARK_CATALOG.