EVPN Hybrid IRB Mode

EVPN Hybrid IRB Mode

Information About EVPN Hybrid IRB Mode

Cisco NX-OS Release 10.2(1)F introduces support for EVPN Hybrid IRB mode. This feature allows NX-OS VTEP devices operating in symmetric IRB mode to seamlessly integrate with asymmetric IRB VTEPs within the same fabric.

EVPN IRB Models

EVPN VXLAN supports Integrated Routing and Bridging (IRB) functionality which allows VTEPs in a VXLAN network to both bridge intra-subnet traffic and route inter-subnet traffic. Inter-subnet routing in an EVPN-IRB overlay network is implemented across fabric VTEPs in two ways:

  • Asymmetric IRB

  • Symmetric IRB

Asymmetric IRB

Asymmetric IRB uses EVPN purely as a Layer-2 VPN overlay, with inter-subnet traffic routed only at the ingress VTEP. As a result, ingress VTEP performs both routing and bridging, while the egress VTEP performs only bridging. On the ingress VTEP, packet is bridged towards the Default Gateway in the source subnet, then routed into the destination subnet local on the ingress VTEP. From that ingress routing operation, traffic is bridge via the Layer-2 VPN (VNI) tunnel. Post receiving and de-encapsulation on the egress VTEP, the packet is simply bridged to the destination end point. In essence, all packet processing associated with inter-subnet forwarding semantics is confined to the ingress VTEP. This model requires all Layer-2 VPNs to exist on all IRB VTEPs that are involved in the inter-subnet procedure for an IP VRF with consistent ARP/ND population across the fabric.

Symmetric IRB

Symmetric IRB uses EVPN as a Layer-2 and Layer-3 VPN overlay, with distributed inter-subnet traffic routed at any VTEP, ingress and egress. As a result, ingress and egress VTEP performs both routing and bridging. On the ingress VTEP, packet is bridged towards the Default Gateway in the source subnet, then routed into the destination VRF local on the ingress VTEP. From that ingress routing operation, traffic is routed via the Layer-3 VPN (VNI) tunnel. Post receiving and de-encapsulation on the egress VTEP, the packet is first routed and then bridged to the destination end point. In essence, all packet processing associated with inter-subnet forwarding semantics is truly distributed across all VTEPs. This model allows only locally attached Layer-2 VPNs to exist on IRB VTEPs that are involved in the inter-subnet procedure for an IP VRF; the ARP/ND consumption is local to where the end point is attached.

Asymmetric and Symmetric Interop

NX-OS supports EVPN-IRB using symmetric IRB mode. While control plane and data plane is needed to enable intra subnet bridging, the procedure is identical across symmetric and asymmetric IRB modes. While the intra subnet approach is the same, the inter subnet procedure between the two IRB modes are incompatible. As a result, inter subnet routing between a symmetric IRB VTEP and a asymmetric IRB VTEP within the same fabric is not possible.

With Cisco's Hybrid IRB mode, the symmetric IRB VTEPs will support an incremental enhancement that allows to seamlessly inter-operate with VTEPs running in asymmetric IRB mode in the same fabric. NX-OS VTEPs enabled with this hybrid mode will continue to operate in the more scalable symmetric IRB mode, whenever communicating with hybrid or symmetric IRB VTEPs. In addition, the hybrid IRB will at the same time inter-operate with the asymmetric IRB VTEPs, if any exist in the same fabric.

EVPN hybrid feature is supported on the Cisco Nexus 9300 - EX, FX, FX2, FX3, GX, N9K-9364C, N9K-9332C, N9K-C9236C, N9K-C9504 TOR and Modular platforms.

Beginning with Cisco NX-OS Release 10.5(2)F, VXLAN QoS is supported on Cisco Nexus 9500 Series switches with N9K-X9736C-FX3 line card.

Inter-op Control Plane

Main difference between asymmetric and symmetric IRB control plane is with respect to how host MAC+IP routes (EVPN route type 2) are formatted. In asymmetric IRB, MAC+IP host routes are advertised with only layer-2 VNI encapsulation and MAC VRF route targets (RT). In symmetric IRB, MAC+IP host routes are advertised with “additional” layer-3 VNI and with “additional” IP VRF RTs to enable inter-subnet routing.

  • NX-OS VTEPs provisioned in hybrid mode continue to advertise local MAC+IP routes using symmetric IRB route type 2 format with additional L3 VNI information and IP VRF RTs, such that hybrid mode NX-OS VTEPs can continue to use symmetric routing between them.

  • VTEPs operating in asymmetric mode simply ignore these additional L3 VNI and IP VRF RT fields and handle these routes using asymmetric route procedure by installing layer-3 adjacencies, and host routes via these adjacencies in IP VRF. Layer-3 adjacency is a ARP/ND entry.

  • NX-OS VTEPs provisioned in hybrid mode handle MAC+IP routes received from an asymmetric VTEP using asymmetric route handling. As a result, they install layer-3 adjacencies, and host routes via these adjacencies for remote hosts advertised from an asymmetric VTEP.

  • Note that as a result, on an NX-OS hybrid VTEP, layer-3 adjacencies are still only installed towards hosts behind asymmetric VTEPs, and not towards hosts behind other NX-OS hybrid VTEPs.

Inter-op Provisioning Requirements

  • NX-OS symmetric IRB VTEPs must be provisioned with all subnets in an IP VRF that are stretched to asymmetric VTEPs in the fabric.

  • NX-OS symmetric IRB VTEPs must be provisioned with subnets in an IP VRF that are stretched to asymmetric VTEPs in “hybrid” mode using “fabric forwarding mode anycast-gateway hybrid” CLI under the subnet SVI interface.

  • All symmetric IRB VTEPs must have the hybrid mode enabled when interoperating with asymmetric VTEPs in each fabric.

Inter-op Data Plane

As a result of the above requirements:

  • NX-OS VTEP continues to follow symmetric routing data path with other NX-OS hybrid VTEPs in both directions. Traffic is bridged in source subnet and routed in IP VRF on ingress VTEP with L3 VNI encapsulation and then routed in IP VRF and bridged in destination subnet on the egress VTEP.

  • NX-OS VTEP follows asymmetric routing data path and encapsulation towards hosts behind asymmetric VTEPs. Traffic is bridged in source subnet, routed in IP VRF with host MAC rewrite, and then bridged in destination subnet on source VTEP, while it is simply bridged in destination subnet on the egress VTEP.

Supported Features

  • Hybrid mode can be enabled per L3 interfaces.

  • IPv4 and IPv6 overlay end points

  • Host mobility is supported with hybrid mode

  • Both Ingress replication as well as multicast underlay is supported.

  • Co-existance of multicast and IR underlay is supported across different VLANs

  • Distributed Anycast Gateway

  • vPC

Guidelines and Limitations

  • Hybrid mode is not supported with DCI Border gateway.

  • In Distributed Anycast Gateway mode, asymmetric IRB also needs to be provisioned with same anycast gateway MAC and IP.

Configuration Example: EVPN Hybrid IRB Mode

The following example provides the configuration of EVPN Hybrid IRB Mode:


vlan 201
vn-segment 20001
interface vlan201
no shutdown
vrf member vrf_30001
ip address 10.1.1.1/16
fabric forwarding mode anycast-gateway hybrid

The following example display the VNIs and the Hybrid IRB Mode:


switch# show nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
SU - Suppress Unknown Unicast
Xconn - Crossconnect
MS-IR - Multisite Ingress Replication
HYB - Hybrid IRB Mode
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 5001 234.1.1.1 Up CP L2 [1001]
nve1 5002 234.1.1.1 Up CP L2 [1002]
nve1 5010 225.1.1.1 Up CP L2 [3003] HYB
nve1 6010 n/a Up CP L3 [vni_6010]
nve1 10001 n/a Up CP L3 [vni_10001]
nve1 30001 234.1.1.1 Up CP L2 [3001] HYB
nve1 30002 234.1.1.1 Up CP L2 [3002] HYB