Cisco Nexus 5000 Troubleshooting Guide

Data traffic is flooding

Data is not getting forwarded, instead it is being flooded to all ports of a VLAN.

Possible Cause

MAC address learning is disabled because of loop detection. A severity 2 syslog message, STM_LOOP_DETECT, should have been received.

Solution

After waiting for 180 seconds, learning is enabled automatically. A severity 2 syslog message, STM_LEARNING_RE_ENABLE, should be received.

Possible Cause

The MAC address table is full. A severity 2 syslog message, STM_LIMIT_REACHED, should have been received.

Solution

After waiting for 180 seconds, the MAC table is flushed and learning is enabled automatically.
Alternatively, wait until some MAC entries are aged out, so that the total learned entries fall below 1500, or enter the clear mac address-table dynamic [address <mac> ] command to clear out entries. This creates free space for new MAC entries to be learned. A severity 2 syslog message, STM_LEARNING_RE_ENABLE, should be received.

Possible Cause

MAC address learning is disabled due to a learning overload (that is, too many new addresses in a short time). A severity 4 syslog message, STM_LEARNING_OVERLOAD, should have been received.

Solution

After waiting for 120 seconds, learning is enabled automatically.

MAC address not learned

MAC address is not learned by the switch. This causes the MAC address not to be listed in the MAC table.

Possible Cause

MAC address learning is disabled because of loop detection. A severity 2 syslog message, STM_LOOP_DETECT, should have been received.

Solution

After waiting for 180 seconds, learning is enabled automatically. A severity 2 syslog message, STM_LEARNING_RE_ENABLE, should be received.

Possible Cause

The MAC address table is full. A severity 2 syslog message, STM_LIMIT_REACHED, should have been received.

Solution

After waiting for 180 seconds, the MAC table is flushed and learning is enabled automatically.
Alternatively, wait until some MAC entries are aged out, so that the total learned entries fall below 1500, or enter the clear mac address-table dynamic [address <mac> ] command to clear out entries. This creates free space for new MAC entries to be learned. A severity 2 syslog message, STM_LEARNING_RE_ENABLE, should be received.

Possible Cause

MAC address learning is disabled due to a learning overload (that is, too many new addresses in a short time). A severity 4 syslog message, STM_LEARNING_OVERLOAD, should have been received.

Solution

After waiting for 120 seconds, learning is enabled automatically.

Possible Cause

No egress path was set for the incoming data traffic. The MAC address from a data stream is not learned if there is no path for that data going out of the switch.

Solution

Configure an outgoing path for the data.

For example, the VLAN may not have been enabled on any of the interfaces other than the one on which data is coming in on. Alternatively, the outgoing interfaces may be down. If this is the case, you need to bring up those interfaces.

Traffic flooding in a VPC setup

Data is not getting forwarded, instead it is being flooded in the presence of a VPC scenario.

Possible Cause

The MAC address is learned on one switch only. Typically, this situation would be a bug regarding the synchronization of the MAC address with the VPC peer.

Solution

Clear the MAC address from the switch where it was learned. This triggers new learning and synchronization of the MAC addresses across the VPC switches.

HIFs go down with the BPDUGuard errDisable message

HIFs go down accompanied with the message, BPDUGuard errDisable.

Possible Cause

By default, the HIFs are in STP edge mode with the BPDU guard enabled. This means that the HIFs are supposed to be connected to hosts or non-switching devices. If they are connected to a non-host device/switch that is sending BPDUs, the HIFs become error-disabled upon receiving a BPDU.

Solution

Enable the BPDU filter on the HIF and on the peer connecting device. With the filter enabled, the HIFs do not send or receive any BPDUs. Use the following commands to confirm the details of the STP port state for the port:

• show spanning-tree interface <id> detail
• show spanning-tree interface <id>

FWM-2-STM_LOOP_DETECT detected on switch, dynamic learning disabled

When FWM-2-STM_LOOP_DETECT is detected on the switch, dynamic learning is disabled.

Possible Cause

• MAC addresses are moving because of incorrect STP-port state convergence.
• MAC addresses are moving because the source of the data being physically moved across all switches while STP states are converged and in correct states.

Use the following commands to verify the STP port state across VLANS on the switches:

• show spanning-tree
• show spanning-tree vlan <id>

Solution

• Check for a correct STP convergence and for STP port states across all switches in the topography. Also confirm that there are no disputes or incorrect port states.
• If the source of the data frames, which are physically moving, is identified, then control the source to halt rapid and continuous moves.
• By default, dynamic learning is opened after 180 seconds. At that point, any STP disputes or inconsistencies should be resolved.

Port stuck in STP blocking state with BLK*(Type_Inc)

A port is stuck in STP blocking state with BLK*(Type_Inc).

Possible Cause

A type inconsistency might exist on an access port when it is connected to a trunk port on the other end. The port becomes BLK*(Type_Inc) to indicate that there is an incorrect configuration on the link. Use the following commands to confirm the details of the STP port state for the port:

• show spanning-tree interface <id> detail
• show spanning-tree interface <id>

Solution

Check the switch port modes configured at both ends (ports) of the link. Ensure that they are in the same mode. Both should be in access or trunk mode. Once the modes are synchronized, the port moves out of the inconsistency state.

Port stuck in STP blocking state with BLK*(PVID_Inc)

A port is stuck in STP blocking state with BLK*(PVID_Inc).

Possible Cause

A PVID inconsistency may exist when there is a native VLAN mismatch across a trunk link. When this occurs, the port state becomes BLK* (PVID_Inc). Use the following commands to confirm the details of the STP port state for the port:

• show spanning-tree interface <id> detail
• show spanning-tree interface <id>

Solution

Check the native VLAN configured at both ends (ports) of the link. Ensure that they have the same native VLAN. Once the native VLANs are synchronized, the port moves out of the inconsistency state.

Port stuck in STP blocking state with BLK*(Loop_Inc)

A port is stuck in STP blocking state with BLK*(Loop_Inc).

Possible Cause

This situation occurs when the loop guard is configured on the port and the port stops receiving BPDUs. This is supposed to prevent loops when unidirectional link failures occur. However, the port is put into a BLK* (Loop_Inc) state. Use the following commands to confirm the details of the STP port state for the port:

• show spanning-tree interface <id> detail
• show spanning-tree interface <id>

Solution

Check the native VLAN configured at both ends (ports) of the link. Ensure that they have the same native VLAN. Once the native VLANs are synchronized, the port moves out of the inconsistency state.

STP port limit exceeded error message

The "STP port limit exceeded" error message occurs when the number of vlan-port instances exceeds the limit on trunk ports.

Possible Cause

This error signals a scalability limit for the Cisco Nexus 5000. The software calculates the number of STP VLAN port instances by multiplying the number of ports by the number of VLANs. The "STP port limit exceeded" error message occurs when the limit on trunk ports is exceeded is logged on the Cisco Nexus 5000 switch.

Note The calculation of STP VLAN port instances is not affected when either MST or RPVST+ is used.

Solution

Decrease the number of VLANs carried over trunk ports or decrease the number of active VLANs.

Source MAC addresses of IGMP joins are learned

In this situation, the source MAC addresses of IGMP joins are learned. However, source MAC addresses of IGMP joins are usually not learned by the switch in order to conserve MAC address space.

Possible Cause

Receiving joins and performing an ISSU simultaneously might cause the situation.

Solution

The MAC addresses age out (expire) if the join stops. Alternatively, you can clear the MAC addresses specifically by using the clear mac address-table dynamic mac <mac> command.

Multicast data traffic not received by host

Host does not receive multicast data traffic.

Possible Cause

The join is not registered.

Solution

• Ensure that the host application is sending the joins.
• Check if the switch port is configured for the VLAN on which the joins are being sent using the show vlan id <vlan> command.
• Check if the relevant VLAN is active by using the show vlan id <vlan> command.
• Check if the switch port is in STP forwarding state by using the show spanning-tree vlan <vlan> command.

Multicast data traffic not received when host is registered for group

Multicast data traffic is not received when the host is registered for the group.

Possible Cause

A bug may exist in exist in the communication between the IGMP and FWM processes.

Review the output from the following commands:

• show ip igmp snooping groups vlan 1001
• show mac address-table multicast vlan 1001 igmp-snooping
• show platform fwm info vlan 1001 all_macgs verbose

Solution

Perform a shut/no-shut operation on the host interface and send the join again.

Multicast traffic is being flooded in a VPC setup

In a VPC setup multicast traffic is being flooded.

Possible Cause

IGMP snooping is disabled on one of the switches.

Solution

Enable IGMP snooping on both switches.

Note Groups for link local IP addresses (that is, 224.0.0.X) are not created.

Nexus 5000 does not have the same VLANs as switch running VTP server

VLANs for the Nexus 5000 are not the same as for the switch running the VTP server.

Possible Cause

The Nexus 5000 currently supports VTP only in transparent mode (4.2(1)N1(1) and later releases).

Solution

This situation indicates that VLANs must be configured locally. However a VTP client and server can both communicate through a Nexus 5000 by using the following commands:

VLAN cannot be created

A VLAN cannot be created.

Possible Cause

An internal VLAN range is used.

Solution

Use a VLAN number that is not being reserved for internal use.

Note The VLAN range of 3968 to 4047 is reserved for internal use.

Example:

Interface VLAN is down

The interface VLAN is down.

Possible Cause

The VLAN was not created.

Solution

Although VLAN <###> is not yet created, the NX-OS allows the configuration of the interface vlan <###>. As a result, the interface vlan <###> does not come up. Use the show vlan command to determine if VLAN <###> exists. If it does not exist, use the vlan <###> command to create the VLAN. After the VLAN is created, you must bounce the interface VLAN to have it come up.

Example:

Possible Cause

VLAN was suspended by the vPC configuration on the Nexus 5000 pair.

Solution

Show that the vPC consistency parameters are global and make sure that the VLAN was not suspended. Otherwise, fix the configuration mismatch on the Nexus 5000 pair:

Example:

Configuring interface to access port does not allow VLAN <###> to go through

After configuring an interface to access a port for allowing VLAN <###>, the VLAN <###> does not go through.

Possible Cause

The VLAN was not created.

Solution

In NX-OS, configuring with the switchport access vlan <###> command on an interface does not automatically create VLAN <###>. You must specifically create VLAN <###> using the vlan <###> command. Use the show vlan command to determine if VLAN <###> exists. If it does not exist, then use the vlan <###> command to create the VLAN.

Cannot create VLAN

The VLAN cannot be created.

Possible Cause

All VLAN resources are exhausted.

Solution

For the Nexus 5000, the maximum number of active VLANs and VSANs per switch is 512 (31 reserved for VSAN; remainder reserved for VLAN). Use the show resource vlan command to determine the number of available VLANs.

Example:

Cannot create SVI

The SVI cannot be created.

Possible Cause

The interface-vlan feature is not enabled.

Solution

The interface-vlan feature must be enabled before configuring the SVI. Use the show feature command to determine which features are enabled.

Example:

Cannot create private VLAN (PVLAN)

The private VLAN (PVLAN) cannot be created.

Possible Cause

The private-vlan feature is not enabled.

Solution

The private-vlan feature must be enabled prior to PVLAN configuration, which makes the PVLAN command available. Use the show feature command to determine which features are enabled.

Example:

SFP validation failed error message

The “SFP validation failed” error message occurs.

Possible Cause

This error message might occur when a 1-Gigabit SFP transceiver is inserted into a port without configuring the interface speed to 1000 mb.

By default, all ports of a Cisco Nexus 5000 switch have an interface speed of 10 Gigabits.

Solution

Set the interface speed to 1000 mb.

Possible Cause

This error message might occur when a Fabric Extender Transceiver (FET) SFP is inserted into a non-fex-fabric mode port.

Solution

FETs are only supported on fex-fabric port connections when connecting a Cisco Nexus 2000 FEX to it's parent Cisco Nexus 5000 switch. The switch port on the parent switch must be enabled for fex-fabric mode with the switchport mode fex-fabric command.

Identifying drops

There are logical and physical causes for the Nexus 5000 to drop a frame. There are also situations when a frame cannot be dropped because of the cut-through nature of the switch architecture. If a drop is necessary, but the frame is being switched in a cut-through path, then the only option is to stomp the Ethernet frame check sequence (FCS). Stomping a frame involves setting the FCS to a known value that does not pass a CRC check. This causes subsequent CRC checks to fail later in the path for this frame. A downstream store-and-forward device, or a host, will be able to drop this frame.

Note When a frame is received on a 10 Gb/s interface, it is considered to be in the cut-through path.

The following example output shows all discards and drops seen on a given interface, except for queuing drops. The queuing drops may be expected or resulting from errors. (Drops are more common than discards.)

Example:

For some commands, you need to know on which chip your port resides.

In the following example, the chip is called Gatos. The example shows which Gatos and which Gatos port is associated with ethernet 1/1.

Expected/Logical drops

During normal operation, the Nexus 5000 encounters frames that cannot be forwarded based on logical conclusions.

For example, if you learn a MAC address on a given interface and receive traffic on that interface with a destination MAC address on the source interface, then you cannot forward the frame. Because it is a known address and cannot be flooded, you can never send traffic out from where it came. This is a requirement to avoid looping Layer 2 topologies.

The error counter, shown in the following example, increments when the ingress port is the only port in the VLAN.

Example (same Gatos instance as in earlier example):

Note The show platform fwm info gatos-errors command increments 3 times for a given drop.

Other expected drops

Queue is full

When a queue is full, you need to increment discards in the respective queue on the ingress interface.

Example:

MTU violation

The Nexus 5000 is a cut-through switch at 10 Gb/s. This means that an MTU can be checked, but the frame will already be transmitting before the length is known. Therefore, the frame cannot be dropped. The frame is truncated after the MTU is reached and the CRC value is stomped. The ingress interface increments an Rx Jumbo and the egress interface will increment a Tx CRC and a Tx Jumbo.

• If jumbo frames are seen with the show interface or the show hardware internal gatos port ethernet 1/1 counters rx commands, this is not an indication that the frames are being dropped. A jumbo frame is just an Ethernet frame, greater than 1500 bytes, that was received or transmitted.
• The <b> show queuing interface <i> ex/y </i></b> command shows the current configured MTU (per class).
• A drop due to an MTU violation can be seen with the show hardware internal gatos counters interrupt match mtu* command.
• A counter that matches the Gatos number and fw_instance from the show hardware internal gatos port ethernet 1/1 | include instance|mac command is the indicator that an MTU violation has taken place and that the frame has been stomped.

Handling CRC errors

When a CRC error is seen in the FCS on a cut-through port, the Rx CRC counter of the show interface command is incremented. However, the frame cannot be dropped because the FCS is at the end of the Ethernet frame on the wire.

The egress interface increments a Tx CRC error and it propagates through to the next device in the path.

You can use the show hardware internal gatos counters interrupt match stomp command to determine if the Nexus 5000 is propagating CRCs or generating them.

• If stomp values exist, they should have matching CRC values on that interface.
• If Rx CRC values exist, then you know it entered the switchport with the error already. You can move on to the connected device to trace it back.

MAC Statistics

During normal operation, a Nexus 5000 encounters frames that cannot be forwarded.

Frames are characterized as good frames or bad frames.

• A good frame is a frame that does not have a CRC error or other kind of error
• A bad frame is a frame that has a CRC error or other kind of error

All counters include MAC Control frames where applicable.

MAC TX Statistics

MAC RX Statistics