About data lineage

Data lineage is a visual map that tracks the entire lifecycle of your data. It shows you where your data comes from (the origin), where it travels (the destinations), and all the changes or transformations that happen along the way.

You can view this complete map of your data's journey directly in the Google Cloud console for assets created in products such as Knowledge Catalog (formerly Dataplex Universal Catalog), BigQuery (including external tables created for Iceberg REST Catalog), and Vertex AI. Because workflows often span multiple regions, Knowledge Catalog supports multi-region lineage, which provides a unified view of your data's journey across the global Google Cloud ecosystem. Advanced users can also retrieve this information by using the Data Lineage API.

Why you need data lineage

Modern companies move and change large amounts of data constantly. For example, transforming raw customer purchases into reports, dashboards, and machine learning models. This complexity creates critical challenges for your team:

• Trust and verification. Data users often struggle to confirm that the reports and numbers they see are accurate and come from a trusted source.

• Troubleshooting. When an error appears in a final report, data teams might find it difficult and time-consuming to trace the issue back through every step to its root cause.

• Change management. Before changing or deleting a piece of data (such as a column in a table), teams need to know every single downstream report or model that relies on it to avoid breaking critical systems.

• Compliance. Leaders need visibility into how sensitive data (such as customer or financial information) is used across the organization to meet regulatory requirements.

Data lineage solves these problems by providing a clear, visual, and documented journey of your data. This lets you understand data sources, trace errors, assess the impact of changes, and maintain compliance.

How data lineage works

The data lineage workflow includes the following steps:

1. Data sources and ingestion: lineage information from your data sources initiates the entire process.

   • Google Cloud services: when the Data Lineage API is enabled, supported services such as BigQuery and Dataflow automatically report lineage events whenever data is moved or transformed.

   • Custom sources: for any systems not automatically supported by Google Cloud integrations, you can use the Data Lineage API to manually record lineage information. We recommend importing events formatted according to the OpenLineage standard.

2. Lineage platform: this central platform ingests, models, and stores all lineage data.

   • Data Lineage API: this API acts as the single entry point for all incoming lineage information. It uses a hierarchical data model consisting of three core concepts: process, run, and event.

   • Processing and storage: the platform processes incoming data and stores it in reliable, query-optimized databases.

3. User experience: you can interact with the stored lineage information in two primary ways:

   • Visual exploration: in the Google Cloud console, a frontend service fetches and renders the lineage data as an interactive graph or list. This is supported for Knowledge Catalog, BigQuery, Lakehouse for Apache Iceberg (for Iceberg REST Catalog tables), physical layer (Cloud Storage), and Vertex AI (for models, datasets, through pipelines; and feature store views, and feature groups). This is ideal for visually exploring your data's journey.

   • Programmatic access: using an API client, you can directly communicate with the Data Lineage API to automate lineage management. This lets you write lineage information from custom sources. It also lets you read and query the stored lineage data for use in other applications or for building custom reports.

Which method should I use for data lineage?

To perform immediate, single-level lookups, use the SearchLinks method. To build a complete lineage graph or perform deep impact analysis (up to 100 levels), use the SearchLineageStreaming method.

Depending on your use case, select the most appropriate method:

Identify direction

• Upstream (Origins):
  • In SearchLinks, set the target field to your asset's FQN.
  • In SearchLineageStreaming, set direction to UPSTREAM.
• Downstream (Destinations):
  • In SearchLinks, set the source field to your asset's FQN.
  • In SearchLineageStreaming, set direction to DOWNSTREAM.

Data lineage information model

Lineage is a record of data being transformed from sources to targets. The Data Lineage API collects this information and organizes it into a hierarchical data model that uses the concepts of processes, runs, and events.

What is a lineage process?

A process is the definition of a data transformation operation for a specific system. For BigQuery lineage, a process is a job of a supported job type. All executions of the same SQL query are linked to a single process, which lets you track every instance where a specific transformation logic is used.

For example, the following SQL query is a process. This query creates a table by counting the total number of trips for each vendor from two source tables.

  CREATE TABLE `dataplex-docs.data_lineage_demo.total_green_trips_22_21`
  AS
  SELECT
      vendor_id,
      COUNT(*) AS number_of_trips
  FROM
      (
          SELECT vendor_id
          FROM `dataplex-docs.data_lineage_demo.nyc_green_trips_2022`
          UNION ALL
          SELECT vendor_id
          FROM `dataplex-docs.data_lineage_demo.nyc_green_trips_2021`
      )
  GROUP BY
      vendor_id;

The REST resource name format for a process is projects/PROJECT_NUMBER/locations/LOCATION/processes/PROCESS_ID.

For example: