-
The following guidelines and limitations apply to VXLAN/VTEP using BGP EVPN:
For more information, see the Cisco Nexus 9000 Series NX-OS System Management Configuration Guide, Release 9.3(x).
-
When SVI is enabled on a VTEP (flood and learn, or EVPN) regardless of ARP suppression, make sure that ARP-ETHER TCAM is carved
using the hardware access-list tcam region arp-ether 256 double-wide command. This requirement does not apply to Cisco Nexus 9200, 9300-EX, and 9300-FX/FX2/FX3 and 9300-GX platform switches and Cisco Nexus 9500 platform switches with 9700-EX/FX line cards.
-
For the Cisco Nexus 9504 and 9508 with R-series line cards, VXLAN EVPN (Layer 2 and Layer 3) is only supported with the 9636C-RX
and 96136YC-R line cards.
-
VXLAN is not supported on N9K-C92348GC-X switches.
-
You can configure EVPN over segment routing or MPLS. See the Cisco Nexus 9000 Series NX-OS Label Switching Configuration Guide, Release 9.3(x) for more information.
-
You can use MPLS tunnel encapsulation using the new CLI encapsulation mpls command. You can configure the label allocation
mode for the EVPN address family. See the Cisco Nexus 9000 Series NX-OS Label Switching Configuration Guide, Release 9.3(x) for more information.
-
In a VXLAN EVPN setup that has 2K VNI scale configuration, the control plane down time may take more than 200 seconds. To
avoid potential BGP flap, extend the graceful restart time to 300 seconds.
-
The command "clear ip arp <interface> vrf <vrf-name> force-delete" on specific interface normally deletes entries from ARP
belonging to that interface and will relearn on traffic. However, when ARP for same IP is resolved on all ECMP paths, force-deleting
ARP entry belonging to one of the ECMP interface will result in automatic relearning of that entry unless that link is down.
-
IP unnumbered in EVPN underlay supports ECMP. Multiple IP unnumbered links are connected back to back between same switches.
ARP will be resolved on all connected interfaces, thus providing ECMP.
-
Beginning with Cisco NX-OS Release 10.2(2)F, the following scale limits are enhanced — Layer 2 VNIs, Extended Layer 2 VNIs,
Layer 3 VNIs, SVI with Distributed Anycast Gateway, IPv4 and IPv6 host routes in internet-peering mode and the ECMP paths.
For the VXLAN scale limit information, see the Cisco Nexus 9000 Series NX-OS Verified Scalability Guide, Release 10.2(2)F.
-
Beginning with Cisco NX-OS Release 10.2(1q)F, VXLAN EVPN is supported on Cisco Nexus N9KC9332D-GX2B platform switches.
-
Beginning with Cisco NX-OS Release 10.2(3)F, VXLAN EVPN is supported on Cisco Nexus 9364D-GX2A, and 9348D-GX2A platform switches.
-
Starting from Cisco NX-OS Release 9.3(5), new VXLAN uplink capabilities are introduced:
-
A physical interface in default VRF is supported as VXLAN uplink.
-
A parent interface in default VRF, carrying subinterfaces with VRF and dot1q tags, is supported as VXLAN uplink.
-
A subinterface in any VRF and/or with dot1q tag remains not supported as VXLAN uplink.
-
An SVI in any VRF remains not supported as VXLAN uplink.
-
In vPC with physical peer-link, a SVI can be leveraged as backup underlay, default VRF only between the vPC members (infra-VLAN,
system nve infra-vlans).
-
On a vPC pair, shutting down NVE or NVE loopback on one of the vPC nodes is not a supported configuration. This means that
traffic failover on one-side NVE shut or one-side loopback shut is not supported.
-
FEX host interfaces remain not supported as VXLAN uplink and cannot have VTEPs connected (BUD node).
-
During the vPC Border Gateway boot up process the NVE source loopback interface undergoes the hold down timer twice instead
of just once. This is a day-1 and expected behavior.
-
The value of the delay timer on NVE interface must be configured to a value that is less than the multi-site delay-restore
timer.
-
You need to configure the VXLAN uplink with ip unreachables in order to enable Path maximum transmission unit (MTU) discovery (PMTUD) in a VXLAN set up. PMTUD prevents fragmentation
in the path between two endpoints by dynamically determining the lowest MTU along the path from the packet's source to its
destination.
-
In a VXLAN EVPN setup, border nodes must be configured with unique route distinguishers, preferably using the auto rd command. Not using unique route distinguishers across all border nodes is not supported. The use of unique route distinguishers
is strongly recommended for all VTEPs of a fabric.
-
ARP suppression is only supported for a VNI if the VTEP hosts the First-Hop Gateway (Distributed Anycast Gateway) for this
VNI. The VTEP and the SVI for this VLAN have to be properly configured for the distributed Anycast Gateway operation, for
example, global Anycast Gateway MAC address configured and Anycast Gateway feature with the virtual IP address on the SVI.
-
The ARP suppression setting must match across the entire fabric. For a specific VNID, all VTEPs must be either configured
or not configured.
-
Mobility Sequence number of a locally originated type-2 route (MAC/MAC-IP) can be mismatched between vPC peers, with one
vTEP having a sequence number K while other vTEP in the same complex can have the same route with sequence number 0. This
does not cause any functional impact and the traffic is not impacted even after the host moves.
-
DHCP snooping (Dynamic Host Configuration Protocol snooping) is not supported on VXLAN VLANs.
-
RACLs are not supported on VXLAN uplink interfaces. VACLs are not supported on VXLAN de-capsulated traffic in egress direction;
this applies for the inner traffic coming from network (VXLAN) towards the access (Ethernet).
As a best practice, always use PACLs/VACLs for the access (Ethernet) to the network (VXLAN) direction. See the Cisco Nexus 9000 Series NX-OS Security Configuration Guide, Release 9.3(x) for other guidelines and limitations for the VXLAN ACL feature.
-
The Cisco Nexus 9000 QoS buffer-boost feature is not applicable for VXLAN traffic.
-
For SVI-related triggers (such as shut/unshut or PIM enable/disable), a 30-second delay was added, allowing the Multicast
FIB (MFIB) Distribution module (MFDM) to clear the hardware table before toggling between L2 and L3 modes or vice versa.
-
For VXLAN BGP EVPN fabrics with EBGP, the following recommendations are applicable:
-
Bind the NVE source-interface to a dedicated loopback interface and do not share this loopback with any function or peerings
of Layer-3 protocols. A best practice is to use a dedicated loopback address for the VXLAN VTEP function.
-
You must bind NVE to a loopback address that is separate from other loopback addresses that are required by Layer 3 protocols.
NVE and other Layer 3 protocols using the same loopback is not supported.
-
The NVE source-interface loopback is required to be present in the default VRF.
-
Only EBGP peering between a VTEP and external nodes (Edge Router, Core Router or VNF) is supported.
-
EBGP peering from the VTEP to the external node using a physical interface or subinterfaces is recommended and it is a best
practice (external connectivity).
-
The EBGP peering from the VTEP to the external node can be in the default VRF or in a tenant VRF (external connectivity).
-
The EBGP peering from the VTEP to a external node over VXLAN must be in a tenant VRF and must use the update-source of a loopback
interface (peering over VXLAN).
-
Using an SVI for EBGP peering on a from the VTEP to the External Node requires the VLAN to be local (not VXLAN extended).
-
When configuring VXLAN BGP EVPN, only the "System Routing Mode: Default" is applicable for the following hardware platforms:
-
Cisco Nexus 9300 platform switches
-
Cisco Nexus 9300-EX platform switches
-
Cisco Nexus 9300-FX/FX2/FX3 platform switches
-
Cisco Nexus 9300-GX/GX2 platform switches
-
Cisco Nexus 9500 platform switches with X9500 line cards
-
Cisco Nexus 9500 platform switches with X9700-EX and X9700-FX line cards
-
Changing the “System Routing Mode” requires a reload of the switch.
-
Cisco Nexus 9516 platform is not supported for VXLAN EVPN.
-
VXLAN is supported on Cisco Nexus 9500 platform switches with the following line cards:
-
9500-R
-
9564PX
-
9564TX
-
9536PQ
-
9700-EX
-
9700-FX
-
Cisco Nexus 9500 platform switches with 9700-EX or -FX line cards support 1G, 10G, 25G, 40G, 100G and 400G for VXLAN uplinks.
-
Cisco Nexus 9200 and 9300-EX/FX/FX2/FX3 and -GX support 1G, 10G, 25G, 40G, 100G and 400G for VXLAN uplinks.
-
Beginning with Cisco NX-OS Release 10.2(3)F, Cisco Nexus 9300-GX2 platform switches support 10G, 25G, 40G, 100G and 400G for
VXLAN uplinks.
-
The Cisco Nexus 9000 platform switches use standards conforming UDP port number 4789 for VXLAN encapsulation. This value is
not configurable.
-
The Cisco Nexus 9200 platform switches with Application Spine Engine (ASE2) have throughput constrains for packet sizes of
99-122 bytes; packet drops might be experienced.
-
The VXLAN network identifier (VNID) 16777215 is reserved and should explicitly not be configured.
-
Non-Disruptive In Service Software Upgrade (ND-ISSU) is supported on Nexus 9300 with VXLAN enabled. Exception is ND-ISSU support
for Cisco Nexus 9300-FX3 and 9300-GX platform switch.
-
Gateway functionality for VXLAN to MPLS (LDP), VXLAN to MPLS-SR (Segment Routing) and VXLAN to SRv6 can be operated on the
same Cisco Nexus 9000 Series platform.
-
VXLAN to MPLS (LDP) Gateway is supported on the Cisco Nexus 3600-R and the Cisco Nexus 9500 with R-Series line cards.
-
VXLAN to MPLS-SR Gateway is supported on the Cisco Nexus 9300-FX2/FX3/GX and Cisco Nexus 9500 with R-Series line cards.
-
Beginning with Cisco NX-OS Release 10.2(3)F, VXLAN to MPLS-SR Gateway is supported on the Cisco Nexus 9300-GX2 platform switches.
-
VXLAN to SRv6 is supported on the Cisco Nexus 9300-GX platform.
-
Beginning with Cisco NX-OS Release 10.2(3)F, VXLAN to SRv6 is supported on the Cisco Nexus 9300-GX2 platform switches.
-
Beginning with Cisco NX-OS Release 10.2(3)F, VXLAN and GRE co-existence is supported on Cisco Nexus 9300-EX/FX/FX2/FX3/GX/GX2
switches, and N9K-C93108TC-FX3P, N9K-C93180YC-FX3, N9K-X9716D-GX switches. Only GRE RX path (decapsulation) is supported.
GRE TX path (encapsulation) is not supported.
-
Multiple Tunnel Encapsulations (VXLAN, GRE and/or MPLS, static label or segment routing) can not co-exist on the same Cisco
Nexus 9000 Series switch with Network Forwarding Engine (NFE).
-
Resilient hashing is supported on the following switch platform with a VXLAN VTEP configured:
Note
|
Resilient hashing is disabled by default.
|
-
Beginning with Cisco NX-OS Release 10.2(3)F, the ECMP resilient hashing is supported on the Cisco Nexus 9300-GX2 platform
switches.
-
It is recommended to use the vpc orphan-ports suspend command for single attached and/or routed devices on a Cisco Nexus 9000 platform switch acting as vPC VTEP.
-
Cisco Nexus supports Type-6 EVPN routes (for IPv4) based on earlier version of draft-ietf-bess-evpn-igmp-mld-proxy draft, where SMET flag field is set as optional.
-
Routing protocol adjacencies using Anycast Gateway SVIs is not supported.
-
When running VXLAN EVPN, any SVI for a VLAN extended over VXLAN must be configured with Anycast Gateway. Any other mode of
operation is not supported.