Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1)SV2(2.2)

Overview

A Virtual Extensibel LAN creates LAN segments by using an overlay approach with MAC-in-UDP encapsulation and a 24-bit segment identifier in the form of a VXLAN ID. The encapsulation carries the original Layer 2 (L2) frame from the Virtual Machine (VM) that is encapsulated from within the Virtual Ethernet Module (VEM). Each VEM is assigned a IP address that are used as the source IP address when encapsulating MAC frames to be sent on the network. You can have multiple vmknics per VEM that are used as sources for this encapsulated traffic. The encapsulation carries the VXLAN identifier used to scope the MAC address of the payload frame. The VXLAN ID to which a VM belongs is indicated within the port profile configuration of the vNIC and is applied when the VM connects to the network. A VXLAN supports three different modes for broadcast, multicast, and MAC distribution mode transport:

• Multicast Mode— A VXLAN uses an IP multicast network to send broadcast, multicast, and unknown unicast flood frames. When a VM joins a VXLAN segment, the server joins a multicast group. Broadcast traffic from the VM is encapsulated and is sent using the multicast outer destination IP address to all the servers in the same multicast group. Subsequent unicast packets are encapsulated and unicast directly to the destination server without multicast IP address.
• Unicast-only Mode— A VXLAN uses each VEM's single unicast IP address as the destination IP address to send broadcast, multicast, and unknown unicast flood frames. Broadcast traffic from the VM is replicated to each VEM by encapsulating it with a VXLAN header and the designated IP address as the outer destination IP address.

– MAC Distribution Mode(supported only in unicast mode)—In this mode, the unknown unicast flooding is reduced because the Virtual Supervisor Module (VSM) learns all the MAC addresses from the VEMs in all VXLANs and distributes those MAC addresses with VXLAN Tunnel Endpoint (VTEP) IP mappings to other VEMs.

The VXLAN creates LAN segments by using an overlay approach with MAC in IP encapsulation. The encapsulation carries the original Layer 2 (L2) frame from the Virtual Machine (VM) which is encapsulated from within the Virtual Ethernet Module (VEM). Each VEM is assigned an IP address which is used as the source IP address when encapsulating MAC frames to be sent on the network. You can have multiple vmknics per VEM that are used as sources for this encapsulated traffic. The encapsulation carries the VXLAN identifier which is used to scope the MAC address of the payload frame.

VXLAN Tunnel EndPoint

Each VEM requires at least one IP/MAC pair to terminate VXLAN packets. This IP/MAC address pair is known as the VXLAN Tunnel End Point (VTEP) IP/MAC addresses. The VEM supports IPv4 addressing for this purpose. The IP/MAC address that the VTEP uses is configured when you enter the capability vxlan command. You can have a maximum of four VTEPs in a single VEM.

One VTEP per VXLAN segment is designated to receive all broadcast, multicast, and unknown unicast flood traffic for the VEM.

When encapsulated traffic is destined to a VEM that is connected to different subnet, the VEM does not use the VMware host routing table. Instead, the VTEPs initiate the Address Resolution Protocol (ARP) for remote VEM IP addresses. If the VTEPs in the different VEMs are in different subnets, you must configure the upstream router to respond by using the Proxy ARP.

VXLAN Gateway

VXLAN termination (encapsulation and decapsulation) is supported on virtual switches. As a result, the only endpoints that can connect into VXLANs are VMs that are connected to a virtual switch. Physical servers cannot be in VXLANs and routers or services that have traditional VLAN interfaces cannot be used by VXLAN networks. The only way that VXLANs can currently interconnect with traditional VLANs is through VM-based software routers.

VXLAN Trunks

A VXLAN trunk allows you to trunk multiple VXLANs on a single virtual Ethernet interface. In order to achieve this configuration, you must encapsulate a VXLAN-VLAN mapping on the virtual Ethernet interface.

VXLAN-VLAN mappings are configured through the VSM and must always be a 1:1 mapping for each Layer 2 domain. VXLAN-VLAN mappings are applied on a virtual Ethernet interface using a port-profile. A single port profile can support multiple VLAN-VXLAN mappings.

Multi-MAC Capability

You can use multi-MAC addresses to mark a virtual Ethernet interface as capable of sourcing packets from multiple MAC addresses. For example, you can use this feature if you have a virtual Ethernet port and you have enabled VXLAN trunking on it and the VM that is connected to the port bridges packets that are sourced from multiple MAC addresses.

By using this feature, you can easily identify such multi-MAC capable ports and handle live migration scenarios correctly for those ports.

Fragmentation

The VXLAN encapsulation overhead is 50 bytes. In order to prevent performance degradation due to fragmentation, the entire interconnection infrastructure between all VEMs exchange VXLAN packets must be configured to carry 50 bytes more than what the VM VNICs are configured to send. For example, if the default VNIC configuration is 1500 bytes, the VEM uplink port profile, upstream physical switch port, and interswitch links, and any routers if present, must be configured to carry a maximum transmission unit (MTU) of at least 1550 bytes. If that is not possible, we recommend that the MTU within the guest VMs you configure to be smaller by 50 bytes.

If you do not configure a smaller MTU, the VEM attempts to notify the VM if it performs Path MTU (PMTU) Discovery. If the VM does not send packets with a smaller MTU, the VM fragments the IP packets. Fragmentation occurs only at the IP layer. If the VM sends a frame that is too large, the frame will be dropped after VXLAN encapsulation and if the frame does not contain an IP packet.

Scalability

Maximum Number of VXLANs

The Cisco Nexus 1000V supports a total of 4096 VLANs or VXLANs (or a maximum of 2048 VLANs or 2048 VXLANs in any combination that totals 4096).

Supported Features

This section contains the following topics:

• Jumbo Frames
• Disabling the VXLAN Feature Globally

Jumbo Frames

Jumbo frames are supported by the Cisco Nexus 1000V if there is space on the frame to accommodate the VXLAN encapsulation overhead of at least 50 bytes, and the physical switch/router infrastructure has the capability to transport these jumbo-sized IP packets.

Disabling the VXLAN Feature Globally

As a safety precaution, do not use the no feature segmentation command if there are any ports associated with a VXLAN port profile. You must remove all associations before you can disable this feature. You can use the no feature segmentation command to remove all the VXLAN bridge domain configurations on the Cisco Nexus 1000V.

VSM Commands

To display ports belonging to a specific segment:

To display the vEthernet bridge domain configuration:

To display the vEthernet bridge configuration with ifindex as an argument:

To display the total number of bridge domain ports:

To display the bridge domain internal configuration:

To display VXLAN vEthernet information:

Additional show commands:

VXLAN Gateway Commands

To display VXLAN Gateway information attached to VSM:

To display VXLAN Gateway information that is not attached to the VSM:

To display VXLAN Gateway statistics:

To display the VXLAN Gateway packet path:

To display the bridge-domain configuration on VSM:

To display the vlan-vxlan mappings programmed on the VSM:

To display the interfaces on the VSM:

To display the the bridge domain-to-VTEP mappings that are maintained by the VSM and are pushed to all VEMs:

To displays the MACs learnt on VSM through VEM distribution:

To verify the port configuration on VSM:

To verify the VTEP distribution on VSM:

Note: You can compare the VTEP table version with the echo show vxlan version-table on VEM.

Additional show commands:

VEM Commands

To verify VXLAN vEthernet programming:

To verify VXLAN vmknic programming:

To verify bridge domain creation on the VEM:

To verify remote IP learning:

To display statistics:

To display detailed per-port statistics for a VXLAN vEthernet/vmknic:

To display detailed per-port-per-bridge domain statistics for a VXLAN vmknic for all bridge domains:

To display detailed per-port-per-bridge domain statistics for a VXLAN vmknic for a specified bridge domain:

To verify the bridge-domain configuration on VEM:

To display the MAC address table that shows the MACs pushed by the VSM:

To verify the port configuration on VEM:

To verify the VTEP distribution on VEM:

To verify if the MAC address table displays the remote IP learning in the segment-cisco bridge domain:

To display the vlan-vxlan mappings programmed on a VEM:

To display the multi-MAC capable interfaces on a VEM:

Vemlog Debugging

To debug the bridge domain setup or configuration, use the following command:

To debug port configuration/CBL/vEthernet LTL pinning, use the following command:

(for encap/decap setup and decisions)

To debug for actual packet editing, VXLAN interface handling, and multicast handling, use the following command:

To debug multicast joins or leaves on the DPA socket, use the following command:

To debug the bridge domain configuration, use the following command:

To debug port configuration, use the following command:

To debug hitless reconnect (HR) for capability l2-lisp, use the following command:

To debug CBL programming.

To debug VXLAN agent interacting with the VSM, use the following command:

Tocheck the VTEP and MAC version, use the following command:

To check the MACs to be distributed on the VSM, use the following command:

Vempkt

Vempkt has been enhanced to display VLAN/SegmentID. Use vempkt to trace the packet path through VEM.

• Encap: Capture ingress on Seg-VEth LTL – Egress on uplink
• Decap: Capture ingress on uplink – Egress on Seg-VEth LTL

Statistics

To display a summary of per-port statistics, use the following command:

To display detailed per-port statistics for VXLAN vmknic, use the following command:

To display detailed per-port statistics for vEthernet in a VXLAN, use the following command:

To display detailed per-port-per-bridge domain statistics for a VXLAN vmknic for all bridge domains, use the following command:

To display detailed per-port-per-bridge domain statistics for a VXLAN vmknic for the specified bridge domain, use the following command:

To display which VXLAN vmknic used for encap and subsequent pinning to uplink PC for static MAC learned on port, use the following command:

To display which VXLAN vmknic used for encapsulation and subsequent pinning to uplink PC, use the following command:

Show Commands

Table 23-1 lists available vemcmd show commands.