Connect to Azure Data Lake Storage in Microsoft Purview

This article outlines the process to register and govern an Azure Data Lake Storage (ADLS Gen2) data source in Microsoft Purview, including instructions to authenticate and interact with the ADLS Gen2 source.

Supported capabilities

Scanning capabilities

Metadata Extraction Full Scan Incremental Scan Scoped Scan
Yes Yes Yes Yes

When scanning Azure Data Lake Storage Gen2 source, Microsoft Purview supports extracting technical metadata including:

  • Storage account
  • Data Lake Storage Gen2 Service
  • File system (container)
  • Folders
  • Files
  • Resource sets

When setting up scan, you can choose to scan the entire ADLS Gen2 or selective folders. Learn about the supported file format here.

Other capabilities

For classifications, sensitivity labels, policies, data lineage, and live view, see the list of supported capabilities.

Prerequisites

Register

This section will enable you to register the ADLS Gen2 data source for scan and data share in Purview.

Prerequisites for register

  • You'll need to be a Data Source Admin and one of the other Purview roles (for example, Data Reader or Data Share Contributor) to register a source and manage it in the Microsoft Purview governance portal. See our Microsoft Purview Permissions page for details.

Steps to register

It's important to register the data source in Microsoft Purview prior to setting up a scan for the data source.

  1. Go to the Microsoft Purview governance portal by:

  2. Navigate to the Data Map --> Sources

    Screenshot that shows the link to open Microsoft Purview governance portal

    Screenshot that navigates to the Sources link in the Data Map

  3. Create the Collection hierarchy using the Collections menu and assign permissions to individual subcollections, as required

    Screenshot that shows the collection menu to create collection hierarchy

  4. Navigate to the appropriate collection under the Sources menu and select the Register icon to register a new ADLS Gen2 data source

    Screenshot that shows the collection used to register the data source

  5. Select the Azure Data Lake Storage Gen2 data source and select Continue

    Screenshot that allows selection of the data source

  6. Provide a suitable Name for the data source, select the relevant Azure subscription, existing Data Lake Store account name and the collection and select Apply. Leave the Data Policy Enforcement toggle on the disabled position until you have a chance to carefully go over this document.

    Screenshot that shows the details to be entered in order to register the data source

  7. The ADLS Gen2 storage account will be shown under the selected Collection

    Screenshot that shows the data source mapped to the collection to initiate scanning

Scan

Tip

To troubleshoot any issues with scanning:

  1. Confirm you have properly set up authentication for scanning
  2. Review our scan troubleshooting documentation.

Authentication for a scan

Your Azure network may allow for communications between your Azure resources, but if you've set up firewalls, private endpoints, or virtual networks within Azure, you'll need to follow one of these configurations below.

Networking constraints Integration runtime type Available credential types
No private endpoints or firewalls Azure IR Managed identity (Recommended), service principal, or account key
Firewall enabled but no private endpoints Azure IR Managed identity
Private endpoints enabled *Self-Hosted IR Service principal, account key

Important

  1. *To use a self-hosted integration runtime, you'll first need to choose the right one for your scenario, create one, and confirm your network settings for Microsoft Purview.
  2. If you use the Self-Hosted Integration runtime to scan parquet files, you need to install the 64-bit JRE 8 (Java Runtime Environment) or OpenJDK on your IR machine. You can check our Java Runtime Environment installation guide.

Using a system or user assigned managed identity for scanning

There are two types of managed identity you can use:

  • System-assigned managed identity (Recommended) - As soon as the Microsoft Purview Account is created, a system-assigned managed identity (SAMI) is created automatically in Microsoft Entra tenant. Depending on the type of resource, specific RBAC role assignments are required for the Microsoft Purview system-assigned managed identity (SAMI) to perform the scans.

  • User-assigned managed identity (preview) - Similar to a system managed identity, a user-assigned managed identity (UAMI) is a credential resource that can be used to allow Microsoft Purview to authenticate against Microsoft Entra ID. For more information, you can see our User-assigned managed identity guide.

It's important to give your Microsoft Purview account or user-assigned managed identity (UAMI) the permission to scan the ADLS Gen2 data source. You can add your Microsoft Purview account's system-assigned managed identity (which has the same name as your Microsoft Purview account) or UAMI at the Subscription, Resource Group, or Resource level, depending on what level scan permissions are needed.

Note

You need to be an owner of the subscription to be able to add a managed identity on an Azure resource.

  1. From the Azure portal, find either the subscription, resource group, or resource (for example, an Azure Data Lake Storage Gen2 storage account) that you would like to allow the catalog to scan.

    Screenshot that shows the storage account

  2. Select Access Control (IAM) in the left navigation and then select + Add --> Add role assignment

    Screenshot that shows the access control for the storage account

  3. Set the Role to Storage Blob Data Reader and enter your Microsoft Purview account name or user-assigned managed identity under the Select input box. Then, select Save to give this role assignment to your Microsoft Purview account.

    Screenshot that shows the details to assign permissions for the Microsoft Purview account

    Note

    For more details, please see steps in Authorize access to blobs and queues using Microsoft Entra ID

    Note

    If you have firewall enabled for the storage account, you must use managed identity authentication method when setting up a scan.

  4. Go into your ADLS Gen2 storage account in Azure portal

  5. Navigate to Security + networking > Networking

    Screenshot that shows the details to provide firewall access

  6. Choose Selected Networks under Allow access from

    Screenshot that shows the details to allow access to selected networks

  7. In the Exceptions section, select Allow trusted Microsoft services to access this storage account and hit Save

    Screenshot that shows the exceptions to allow trusted Microsoft services to access the storage account

Create the scan

  1. Open your Microsoft Purview account and select the Open Microsoft Purview governance portal

  2. Navigate to the Data map --> Sources to view the collection hierarchy

  3. Select the New Scan icon under the ADLS Gen2 data source registered earlier

    Screenshot that shows the screen to create a new scan

  4. Choose either the Azure integration runtime if your source is publicly accessible, a managed virtual network integration runtime if using a managed virtual network, or a self-hosted integration runtime if your source is in a private virtual network. For more information about which integration runtime to use, see the choose the right integration runtime configuration article.

If using a system or user assigned managed identity

  1. Provide a Name for the scan, select the system-assigned or user-assigned managed identity under Credential, choose the appropriate collection for the scan, and select Test connection. On a successful connection, select Continue.

    Screenshot that shows the managed identity option to run the scan

Scope and run the scan

  1. You can scope your scan to specific folders and subfolders by choosing the appropriate items in the list.

    Scope your scan

  2. Then select a scan rule set. You can choose between the system default, existing custom rule sets, or create a new rule set inline.

    Scan rule set

  3. If creating a new scan rule set, select the file types to be included in the scan rule.

    Scan rule set file types

  4. You can select the classification rules to be included in the scan rule

    Scan rule set classification rules

    Scan rule set selection

  5. Choose your scan trigger. You can set up a schedule or run the scan once.

    scan trigger

  6. Review your scan and select Save and run.

    review scan

View your scans and scan runs

To view existing scans:

  1. Go to the Microsoft Purview portal. On the left pane, select Data map.
  2. Select the data source. You can view a list of existing scans on that data source under Recent scans, or you can view all scans on the Scans tab.
  3. Select the scan that has results you want to view. The pane shows you all the previous scan runs, along with the status and metrics for each scan run.
  4. Select the run ID to check the scan run details.

Manage your scans

To edit, cancel, or delete a scan:

  1. Go to the Microsoft Purview portal. On the left pane, select Data Map.

  2. Select the data source. You can view a list of existing scans on that data source under Recent scans, or you can view all scans on the Scans tab.

  3. Select the scan that you want to manage. You can then:

    • Edit the scan by selecting Edit scan.
    • Cancel an in-progress scan by selecting Cancel scan run.