RDMA network profiles

This page provides an overview of Remote Direct Memory Access (RDMA) network profiles in Google Cloud.

Overview

RDMA network profiles let you create Virtual Private Cloud (VPC) networks that provide low-latency, high-bandwidth RDMA communication between the memory or GPUs of Compute Engine instances that are created in the network.

RDMA network profiles are useful for running AI workloads. For more information about running AI workloads in Google Cloud, see AI Hypercomputer overview.

You can create the following types of VPC networks by using RDMA network profiles:

VPC network type Protocol Network profile resource name Instance types NIC type
Falcon VPC network RDMA over Falcon transport ZONE-vpc-falcon H4D IRDMA
RoCE VPC network RDMA over converged ethernet v2 (RoCE v2)

For VM instances:
ZONE-vpc-roce

For bare metal instances:
ZONE-vpc-roce-metal

VM:
A3 Ultra, A4, A4X

Bare metal:
A4X Max

MRDMA

Supported zones

RDMA network profiles are available in a limited set of zones. You can only create a Falcon VPC network or RoCE VPC network in a zone where the corresponding network profile is available.

To view the supported zones, see List network profiles.

Alternatively, you can view the supported zones for the machine type that you intend to create in the network. RDMA network profiles are available in the same zones as their supported machine types. For more information, see the following:

Specifications

VPC networks created with an RDMA network profile have the following specifications:

  • Zonal constraint. Resources using a VPC network with an RDMA network profile are limited to the same zone as the RDMA network profile associated with the VPC network during the network creation. This zonal limit has the following effects:

    • All instances that have network interfaces in the VPC network must be created in the zone that matches the zone of the RDMA network profile used by the VPC network.

    • All subnets created in the VPC network must be located in the region that contains the zone of the RDMA network profile used by the VPC network.

  • RDMA network interfaces only. A VPC network with an RDMA network profile supports attachments only from specific network interfaces:

    • Falcon VPC networks only support IRDMA network interfaces (NICs).
    • RoCE VPC networks only support MRDMA NICs.

    All non-RDMA NICs of an instance must be attached to a regular VPC network.

  • 8896 byte MTU. For best performance, we recommend a maximum transmission unit (MTU) of 8896 bytes for VPC networks with an RDMA network profile. This allows the RDMA driver in the instance's guest operating system to use smaller MTUs if needed.

    If you create a VPC network with an RDMA network profile, then 8896 bytes is the default MTU.

  • Firewall differences. See the following information about firewall differences in VPC networks with an RDMA network profile:

    • VPC networks with an RDMA network profile use the following implied firewall rules, which are different from the implied firewall rules used by regular VPC networks:

      • Implied allow egress
      • Implied allow ingress

    • Cloud NGFW support depends on the type of VPC network:

      • For RoCE VPC networks, see the following:

        • RoCE VPC networks for VM instances only support regional network firewall policies that have an RoCE firewall policy type. The set of parameters for rules within a supported regional network firewall policy are limited. For more information, see Cloud NGFW for RoCE VPC networks.
        • RoCE VPC networks for bare metal instances don't support configuring Cloud NGFW rules or policies.

      • Falcon VPC networks don't support configuring Cloud NGFW rules or policies.

  • No Connectivity Tests support. Connectivity Tests doesn't support VPC networks with an RDMA network profile.

  • Other VPC features. VPC networks with an RDMA network profile support a limited set of other VPC features. For more information, see the following Supported and unsupported features section.

Supported and unsupported features

The following sections describe which VPC features are supported and unsupported for each type of VPC network that you can create by using an RDMA network profile:

Falcon VPC networks (falcon)

The following table lists the features that are supported by Falcon VPC networks, which are created by using the falcon network profile.

Feature Supported Network profile property Network profile property value Details
RDMA NICs interfaceTypes IRDMA

Falcon VPC networks support only IRDMA NICs

Other NIC types, such as MRDMA, GVNIC, and VIRTIO_NET, aren't supported.
Multi-NIC in the same network allowMultiNicInSameNetwork MULTI_NIC_IN_SAME_NETWORK_ALLOWED

Falcon VPC networks support multi-NIC instances, allowing two or more RDMA NICs of the same instance to be in the same VPC network. Each NIC must attach to a unique subnet in the VPC network.