Troubleshoot HSRP Problems in Catalyst Switch Networks

Introduction

This document describes common issues and ways to troubleshoot Hot Standby Router Protocol (HSRP) problems.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Understand HSRP

Background Information

This document covers these most-common issues that relate to HSRP:

• Router report of a duplicate HSRP standby IP address

• Constant HSRP state changes (active, standby, speak)

• HSRP peers not present

• Switch error messages that relate to HSRP

• Excessive network unicast flooding to the HSRP configuration

Note: This document details how to troubleshoot HSRP in Catalyst switch environments. The document contains many references to software versions and network topology design. Nevertheless, the sole purpose of this document is to facilitate and guide engineers on who to troubleshoot HSRP. This document is not intended to be a design guide, software-recommendation document, or a best practices document.

Businesses and consumers that rely on intranet and Internet services for their mission-critical communications require and expect their networks and applications to be continuously available to them. Customers can satisfy their demands for near-100 percent network uptime if they leverage the HSRP in Cisco IOS® Software. HSRP, which is unique to Cisco platforms, provides network redundancy for IP networks in a manner that ensures that user traffic immediately and transparently recovers from first-hop failures in network edge devices or access circuits.

Two or more routers can act as a single, virtual router if they share an IP address and a MAC (Layer 2 [L2]) address. The address is necessary for host workstation default gateway redundancy. Most host workstations do not contain routing tables and use only a single next hop IP and MAC address. This address is known as a default gateway. With HSRP, members of the virtual router group continually exchange status messages. One router can assume the routing responsibility of another if a router goes out of commission for either planned or unplanned reasons. Hosts are configured with a single default gateway and continue to forward IP packets to a consistent IP and MAC address. The changeover of devices that do the routing is transparent to the end workstations.

Note: You can configure host workstations that run Microsoft OS for multiple default gateways. But the multiple default gateways are not dynamic. The OS only uses one single default gateway at a time. The system only selects an additional configured default gateway at boot time if the first configured default gateway is determined unreachable by Internet Control Management Protocol (ICMP).

Basic Operation

A set of routers that run HSRP works in concert to present the illusion of a single default gateway router to the hosts on the LAN. This set of routers is known as an HSRP group or standby group. A single router that is elected from the group is responsible to forward the packets that hosts send to the virtual router. This router is known as the active router. Another router is elected as the standby router. If the active router fails, the standby assumes the packet forwarding duties. Although an arbitrary number of routers can run HSRP, only the active router forwards the packets that are sent to the virtual router IP address.

In order to minimize network traffic, only the active and the standby routers send periodic HSRP messages after the protocol has completed the election process. Additional routers in the HSRP group remain in the Listen state. If the active router fails, the standby router takes over as the active router. If the standby router fails or becomes the active router, another router is elected as the standby router.

Each standby group emulates a single virtual router (default gateway). For each group, a single well-known MAC and IP address is allocated to that group. Multiple standby groups can coexist and overlap on a LAN, and individual routers can participate in multiple groups. In this case, the router maintains a separate state and timers for each group.

HSRP Terms

HSRP Addressing

HSRP Router Communication

Routers that run HSRP communicate HSRP information between each other through HSRP hello packets. These packets are sent to the destination IP multicast address 224.0.0.2 on User Datagram Protocol (UDP) port 1985. IP multicast address 224.0.0.2 is a reserved multicast address that is used to communicate to all routers. The active router sources hello packets from its configured IP address and the HSRP virtual MAC address. The standby router sources hellos from its configured IP address and the burned-in MAC address (BIA). This use of source addressing is necessary so that HSRP routers can correctly identify each other.

In most cases, when you configure routers to be part of an HSRP group, the routers listen for the HSRP MAC address for that group as well as their own BIA. The only exception to this behavior is for Cisco 2500, 4000, and 4500 routers. These routers have Ethernet hardware that only recognizes a single MAC address. Therefore, these routers use the HSRP MAC address when they serve as the active router. The routers use their BIA when they serve as the standby router.

HSRP Standby IP Address Communication on All Media Except Token Ring

Because host workstations are configured with their default gateway as the HSRP standby IP address, hosts must communicate with the MAC address that is associated with the HSRP standby IP address. This MAC address is a virtual MAC address that is composed of 0000.0c07.ac**. The ** is the HSRP group number in hexadecimal, based on the respective interface. For example, HSRP group 1 uses the HSRP virtual MAC address of 0000.0c07.ac01. Hosts on the adjoining LAN segment use the normal Address Resolution Protocol (ARP) process in order to resolve the associated MAC addresses.

ICMP Redirects

HSRP peer routers that protect a subnet are able to provide access to all other subnets in the network. This is the basis of HSRP. Therefore, which router becomes the active HSRP router is irrelevant. In Cisco IOS software releases earlier than Cisco IOS Software Release 12.1(3)T, ICMP redirects are automatically disabled on an interface when HSRP is used on that interface. Without this configuration, the hosts can be redirected away from the HSRP virtual IP address and toward an interface IP and MAC address of a single router. Redundancy is lost.

Cisco IOS Software introduces a method to allow ICMP redirects with HSRP. This method filters outbound ICMP redirect messages through HSRP. The next hop IP address is changed to an HSRP virtual address. The gateway IP address in the outbound ICMP redirect message is compared to a list of HSRP active routers that are present on that network. If the router that corresponds to the gateway IP address is an active router for an HSRP group, the gateway IP address is replaced with that group virtual IP address. This solution allows hosts to learn optimal routes to remote networks and, at the same time, maintain the resilience that HSRP provides.

HSRP Functionality Matrix

Refer to the Cisco IOS Release and HSRP Functionality Matrix section of Hot Standby Router Protocol Features and Functionality in order to learn about the features and Cisco IOS Software releases that support HSRP.

HSRP Features

Refer to Hot Standby Router Protocol Features and Functionality for information on most of the HSRP features. This document provides information on these HSRP features:

• Preemption

• Interface tracking

• Use of a BIA

• Multiple HSRP groups

• Configurable MAC addresses

• Syslog support

• HSRP debugging

• Enhanced HSRP debugging

• Authentication

• IP redundancy

• Simple Network Management Protocol (SNMP) MIB

• HSRP for Multiprotocol Label Switching (MPLS)

Note: You can use your browser Find feature in order to locate these sections within the document.

Packet Format

This table shows the format of the data portion of the UDP HSRP frame:

This table describes each of the fields in the HSRP packet:

HSRP States

HSRP Timers

Each router only uses three timers in HSRP. The timers time hello messages. The HSRP converges, when a failure occurs, depend on how the HSRP hello and hold timers are configured. By default, these timers are set to 3 and 10 seconds, respectively, which means that a hello packet is sent between the HSRP standby group devices every 3 seconds, and the standby device becomes active when a hello packet has not been received for 10 seconds. You can lower these timer settings to speed up the failover or preemption, but, to avoid increased CPU usage and unnecessary standby state flapping, do not set the hello timer lower than one (1) second or the hold timer lower than 4 seconds. Note that, if you use the HSRP tracking mechanism and the tracked link fails, the failover or preemption occurs immediately, regardless of the hello and hold timers. When a timer expires, the router transitions to a new HSRP state. The timers can be changed with this command: standby [group-number] timers hellotime holdtime. For example, standby 1 timers 5 15.

This table provides more information on these timers:

HSRP Events

This table provides the events in the HSRP finite state machine:

HSRP Actions

This table specifies the actions to be taken as part of the state machine:

HSRP State Table

The diagram in this section shows the state transitions of the HSRP state machine. Each time that an event occurs, the associated action results, and the router transitions to the next HSRP state. In the diagram, numbers designate events, and letters designate the associated action. The table in the section HSRP Events defines the numbers, and the table in the section HSRP Actions defines the letters. Use this diagram only as a reference. The diagram is detailed and is not necessary for general troubleshoot purposes.

For a high resolution image of the diagram, see HSRP States Operation.

Packet Flow

Router A Configuration (Active Router)

interface GigabitEthernet 0/0
 ip address 10.1.1.2 255.255.255.0
 mac-address 4000.0000.0010
 standby 1 ip 10.1.1.1
 standby 1 priority 200

interface GigabitEthernet 0/1
 ip address 10.1.2.2 255.255.255.0
 mac-address 4000.0000.0011
 standby 1 ip 10.1.2.1
 standby 1 priority 200

Router B Configuration (Standby Router)

interface GigabitEthernet 0/0
    ip address 10.1.1.3 255.255.225.0    
    mac-address 4000.0000.0020
    standby 1 ip 10.1.1.1

interface GigabitEthernet 0/1
     ip address 10.1.2.3 255.255.255.0
     mac-address 4000.0000.0021
     standby 1 ip 10.1.2.1

Note: These examples configure static MAC addresses for illustration purposes only. Do not configure static MAC addresses unless you are required to do so.

You must understand the concept behind packet flow when you obtain sniffer traces to troubleshoot HSRP problems. Router A uses the priority of 200 and becomes the active router on both interfaces. In the example in this section, packets from the router that are destined for a host workstation have the source MAC address of the router physical MAC address (BIA). Packets from the host machines that are destined for the HSRP IP address have the destination MAC address of the HSRP virtual MAC address. Note that the MAC addresses are not the same for each flow between the router and the host.

This table shows the respective MAC and IP address information per flow on the basis of a sniffer trace that is taken from Switch X.

Troubleshoot HSRP Case Studies

Case Study #1: HSRP Standby IP Address Is Reported as a Duplicate IP Address

These error messages can appear:

Oct 12 13:15:41: %STANDBY-3-DUPADDR: Duplicate address 10.25.0.1 
  on Vlan25, sourced by 0000.0c07.ac19 
Oct 13 16:25:41: %STANDBY-3-DUPADDR: Duplicate address 10.25.0.1 
  on Vlan25, sourced by 0000.0c07.ac19 
Oct 15 22:31:02: %STANDBY-3-DUPADDR: Duplicate address 10.25.0.1 
  on Vlan25, sourced by 0000.0c07.ac19 
Oct 15 22:41:01: %STANDBY-3-DUPADDR: Duplicate address 10.25.0.1 
  on Vlan25, sourced by 0000.0c07.ac19 

These error messages do not necessarily indicate an HSRP problem. Rather, the error messages indicate a possible Spanning Tree Protocol (STP) loop or router/switch configuration issue. The error messages are just symptoms of another problem.

In addition, these error messages do not prevent the proper operation of HSRP. The duplicate HSRP packet is ignored. These error messages are throttled at 30-second intervals. But, slow network performance and packet loss can result from the network instability that causes the STANDBY-3-DUPADDR error messages of the HSRP address.

These messages specifically indicate that the router received a data packet that was sourced from the HSRP IP address on VLAN 25 with the MAC addresses 0000.0c07.ac19. Since the HSRP MAC address is 0000.0c07.ac19, either the router in question received its own packet back or both routers in the HSRP group went into the active state. Because the router received its own packet, the problem most likely is with the network rather than the router. A variety of problems can cause this behavior. Among the possible network problems that cause the error messages are:

• Momentary STP loops

• EtherChannel configuration issues

• Duplicated frames

When you troubleshoot these error messages, see the steps to troubleshoot in the Troubleshooting HSRP in Catalyst Switches section of this document. All the troubleshoot modules are applicable to this section, which includes modules on configuration. In addition, note any errors in the switch log and reference additional case studies as necessary.

You can use an access list in order to prevent the active router from receiving its own multicast hello packet. But, this is only a workaround for the error messages and actually hides the symptom of the problem. The workaround is to apply an extended inbound access list to the HSRP interfaces. The access list blocks all traffic that is sourced from the physical IP address and that is destined to all routers multicast address 224.0.0.2.

access-list 101 deny ip host 172.16.12.3 host 224.0.0.2 
access-list 101 permit ip any any 
  
interface GigabitEthernet 0/0 
 ip address 172.16.12.3 255.255.255.0 
 standby 1 ip 172.16.12.1 
 ip access-group 101 in

Case Study #2: HSRP State Continuously Changes (Active, Standby, Speak) or %HSRP-6-STATECHANGE

These error messages can appear:

Jan 9 08:00:42.623: %STANDBY-6-STATECHANGE: Standby: 49: 
  Vlan149 state Standby -> Active
Jan 9 08:00:56.011: %STANDBY-6-STATECHANGE: Standby: 49: 
  Vlan149 state Active -> Speak
Jan 9 08:01:03.011: %STANDBY-6-STATECHANGE: Standby: 49: 
  Vlan149 state Speak -> Standby
Jan 9 08:01:29.427: %STANDBY-6-STATECHANGE: Standby: 49: 
  Vlan149 state Standby -> Active
Jan 9 08:01:36.808: %STANDBY-6-STATECHANGE: Standby: 49: 
  Vlan149 state Active -> Speak
Jan 9 08:01:43.808: %STANDBY-6-STATECHANGE: Standby: 49: 
  Vlan149 state Speak -> Standby

Jul 29 14:03:19.441: %HSRP-5-STATECHANGE: Vlan10 Grp 110 state Standby -> Active
Jul 29 16:27:04.133: %HSRP-5-STATECHANGE: Vlan10 Grp 110 state Active -> Speak
Jul 29 16:31:49.035: %HSRP-5-STATECHANGE: Vlan10 Grp 110 state Speak -> Standby

These error messages describe a situation in which a standby HSRP router did not receive three successive HSRP hello packets from its HSRP peer. The output shows that the standby router moves from the standby state to the active state. Shortly thereafter, the router returns to the standby state. Unless this error message occurs during the initial installation, an HSRP issue probably does not cause the error message. The error messages signify the loss of HSRP hellos between the peers. When you troubleshoot this issue, you must verify the communication between the HSRP peers. A random, momentary loss of data communication between the peers is the most common problem that results in these messages. HSRP state changes are often due to High CPU Utilization. If the error message is due to high CPU utilization, put a sniffer on the network and the trace the system that causes the high CPU utilization.

There are several possible causes for the loss of HSRP packets between the peers. The most common problems are physical layer problems, excessive network traffic caused by spanning tree issues or excessive traffic caused by each Vlan. As with Case Study #1, all the troubleshoot modules are applicable to the resolution of HSRP state changes, particularly the Layer 3 HSRP Debugging.

If the loss of HSRP packets between peers is due to excessive traffic caused by each VLAN as mentioned, you can tune or increase the SPD and hold the queue size to overcome the input queue drop problem.

To increase the Selective Packet Discard (SPD) size, go to the configuration mode and execute these commands on the Cat6500 switches:

(config)#ip spd queue max-threshold 600

!--- Hidden Command

(config)#ip spd queue min-threshold 500

!--- Hidden Command

In order to increase the hold queue size, go to the VLAN interface mode and execute this command:

(config-if)#hold-queue 500 in

After you increase the SPD and hold queue size, you can clear the interface counters if you execute the clear counter interfacecommand.

Case Study #3: HSRP Does Not Recognize Peer

The router output in this section shows a router that is configured for HSRP but does not recognize its HSRP peers. In order for this to occur, the router must fail to receive HSRP hellos from the neighbor router. When you troubleshoot this issue, see the Verify Physical Layer Connectivity section and the Verify HSRP Router Configuration section of this document. If the physical layer connectivity is correct, check for the mismatched VTP modes.

Vlan8 - Group 8
Local state is Active, priority 110, may preempt
Hellotime 3 holdtime 10
Next hello sent in 00:00:01.168
Hot standby IP address is 10.1.2.2 configured
Active router is local
Standby router is unknown expired
Standby virtual mac address is 0000.0c07.ac08
5 state changes, last state change 00:05:03

Case Study #4: HSRP State Changes and Switch Reports SYS-4-P2_WARN: 1/Host <mac_address> Is Flapping Between Port <port_1> and Port <port_2> in Syslog

These error messages can appear:

2001 Jan 03 14:18:43 %SYS-4-P2_WARN: 1/Host 00:00:0c:14:9d:08 
  is flapping between port 2/4 and port 2/3

Feb 4 07:17:44 AST: %SW_MATM-4-MACFLAP_NOTIF: Host 0050.56a9.1f28 in vlan 1027 is flapping between port Te1/0/7 and port Te2/0/2

In Catalyst Switches, the switch reports a host MAC address that moves if the host MAC address moves twice within 15 seconds. A possible cause is an STP loop. The switch discards packets from this host for about 15 seconds in an effort to minimize the impact of an STP loop. If the MAC address move between two ports that is reported is the HSRP virtual MAC address, the problem is most likely an issue in which both HSRP routers go into the active state.

If the MAC address that is reported is not the HSRP virtual MAC address, the issue can indicate the loop, duplication, or reflection of packets in the network. These types of conditions can contribute to HSRP problems. The most common causes for the move of MAC addresses are spanning tree problems or physical layer problems.

When you troubleshoot this error message, complete these steps: 

Note: Also, complete the steps in the Troubleshooting HSRP in Catalyst Switches section of this document.

1. Determine the correct source (port) of the host MAC address.

2. Disconnect the port that must not source the host MAC address.

3. Document the STP topology on a per-VLAN basis and check for STP failure.

4. Verify the port channeling configuration.

   1. An incorrect port channel configuration can result in the flap of error messages by the host MAC address. This is because of the load-balancing nature of port channeling.

Case Study #5: Asymmetric Routing and HSRP (Excessive Flooding of Unicast Traffic in Network with Routers that run HSRP)

With asymmetric routing, transmit and receive packets use different paths between a host and the peer with which it communicates. This packet flow is a result of the configuration of load-balancing between HSRP routers, based on HSRP priority, which set the HSRP to active or standby. This type of packet flow in a switching environment can result in excessive unknown unicast flooding. Also, Multilayer Switching (MLS) entries can be absent. Unknown unicast flooding occurs when the switch floods a unicast packet out of all ports. The switch floods the packet because there is no entry for the destination MAC address. This behavior does not break connectivity because packets are still forwarded. But, the behavior does account for the flood of extra packets on host ports. This case studies the behavior of asymmetric routing and why unicast flooding results.

Symptoms of asymmetric routing include:

• Excessive unicast packet flooding

• Absent MLS entry for flows

• Sniffer trace that shows that packets on the host port are not destined for the host

• Increased network latency with L2-based packet rewrite engines, such as server load balancers, web cache devices, and network appliances

  Examples include the Cisco LocalDirector and Cisco Cache Engine.

• Dropped packets on connected hosts and workstations that cannot handle the additional unicast-flooding traffic load

Note: The default ARP cache aging time on a router is four hours. The default aging time of the switch content-addressable memory (CAM) entry is five minutes. The ARP aging time of the host workstations is not significant for this discussion. but, the example sets the ARP aging time to four hours.

This diagram illustrates this issue. This topology example includes Catalyst 6500s with Multilayer Switch Feature Cards (MSFCs) in each switch. Although this example uses MSFCs, you can use any router instead of the MSFC. Example routers that you can use include the Route Switch Module (RSM), Gigabit Switch Router (GSR), and Cisco 7500. The hosts are directly connected to ports on the switch. The switches are interconnected through a trunk which carries traffic for VLAN 1 and VLAN 2.

These outputs are excerpts from the show standby command configuration from each MSF.

MSFC1

interface Vlan 1 
 mac-address 0003.6bf1.2a01 
 ip address 10.1.1.2 255.255.255.0 
 no ip redirects 
 standby 1 ip 10.1.1.1 
 standby 1 priority 110 
	
interface Vlan 2 
 mac-address 0003.6bf1.2a01 
 ip address 10.1.2.2 255.255.255.0 
 no ip redirects 
 standby 2 ip 10.1.2.1 
	
MSFC1#show standby
Vlan1 - Group 1
Local state is Active, priority 110
Hellotime 3 holdtime 10
Next hello sent in 00:00:00.696
Hot standby IP address is 10.1.1.1 configured
Active router is local
Standby router is 10.1.1.3 expires in 00:00:07
Standby virtual mac address is 0000.0c07.ac01
2 state changes, last state change 00:20:40
Vlan2 - Group 2
Local state is Standby, priority 100
Hellotime 3 holdtime 10
Next hello sent in 00:00:00.776
Hot standby IP address is 10.1.2.1 configured
Active router is 10.1.2.3 expires in 00:00:09, priority 110
Standby router is local
4 state changes, last state change 00:00:51
MSFC1#exit
Console> (enable)

MSFC2

interface Vlan 1 
 mac-address 0003.6bf1.2a02 
 ip address 10.1.1.3 255.255.255.0 
 no ip redirects 
 standby 1 ip 10.1.1.1 
    
interface Vlan 2 
 mac-address 0003.6bf1.2a02 
 ip address 10.1.2.3 255.255.255.0 
 no ip redirects 
 standby 2 ip 10.1.2.1 
 standby 2 priority 110  
	 
MSFC2#show standby
Vlan1 - Group 1
Local state is Standby, priority 100
Hellotime 3 holdtime 10
Next hello sent in 00:00:01.242
Hot standby IP address is 10.1.1.1 configured
Active router is 10.1.1.2 expires in 00:00:09, priority 110
Standby router is local
7 state changes, last state change 00:01:17 
Vlan2 - Group 2
Local state is Active, priority 110
Hellotime 3 holdtime 10
Next hello sent in 00:00:00.924
Hot standby IP address is 10.1.2.1 configured
Active router is local
Standby router is 10.1.2.2 expires in 00:00:09
Standby virtual mac address is 0000.0c07.ac02
2 state changes, last state change 00:40:08
MSFC2#exit

Note: On MSFC1, VLAN 1 is in the HSRP active state, and VLAN 2 is in the HSRP standby state. On MSFC2, VLAN 2 is in the HSRP active state, and VLAN 1 is in the HSRP standby state. The default gateway of each host is the respective standby IP address.

1. Initially, all caches are empty. Host A uses MSFC1 as its default gateway. Host B uses MSFC2.

   ARP and MAC Address Tables Before Ping Is Initiated

   Host A ARP Table	Switch 1 MAC Address Table MAC VLAN Port	MSFC1 ARP Table	MSFC2 ARP Table	Switch 2 MAC Address Table MAC VLAN Port	Host B ARP Table
   	0003.6bf1.2a01 1 15/1			0003.6bf1.2a02 1 15/1
   	0003.6bf1.2a01 2 15/1			0003.6bf1.2a02 2 15/1
   	0000.0c07.ac01 1 15/1			0000.0c07.ac01 1 1/1
   	0000.0c07.ac02 2 1/1			0000.0c07.ac02 2 15/1
   	0003.6bf1.2a02 1 1/1			0003.6bf1.2a01 1 1/1
   	0003.6bf1.2a02 2 1/1			0003.6bf1.2a01 2 1/1

   Note: For brevity, the Switch 1 MAC address for the router HSRP and MAC address are not included in the other tables that appear in this section.

2. Host A pings host B, which means that host A sends an ICMP echo packet. Because each host resides on a separate VLAN, host A forwards its packets that are destined for host B to its default gateway. In order for that process to occur, host A must send an ARP in order to resolve its default gateway MAC address, 10.1.1.1.

   ARP and MAC Address Tables After Host A Sends ARP for Default Gateway

   Host A ARP Table	Switch 1 MAC Address Table MAC VLAN Port	MSFC1 ARP Table	MSFC2 ARP Table	Switch 2 MAC Address Table MAC VLAN Port	Host B ARP Table
   10.1.1.1 : 0000.0c07.ac01	0000.0c00.0001 1 2/1	10.1.1.10 : 0000.0c00.0001

3. MSFC1 receives the packet, rewrites the packet, and forwards the packet to host B. In order to rewrite the packet, MSFC1 sends an ARP request for host B because the host resides off a directly connected interface. MSFC2 has yet to receive any packets in this flow. When MSFC1 receives the ARP reply from host B, both switches learn the source port that is associated with host B.

   ARP and MAC Address Tables After Host A Sends Packet to Default Gateway and MSFC1 Sends ARP for Host B

   Host A ARP Table	Switch 1 MAC Address Table MAC VLAN Port	MSFC1 ARP Table	MSFC2 ARP Table	Switch 2 MAC Address Table MAC VLAN Port	Host B ARP Table
   10.1.1.1 : 0000.0c07.ac01	0000.0c00.0001 1 2/1	10.1.1.10 : 0000.0c00.0001		0000.0c00.0002 2 2/1	10.1.2.2 : 0003.6bf1.2a01
   	0000.0c00.0002 2 1/1	10.1.2.10 : 0000.0c00.0002

4. Host B receives the echo packet from host A, through MSFC1. Host B must now send an echo reply to host A. Since host A resides on a different VLAN, host B forwards the reply through its default gateway, MSFC2. In order to forward the packet throughMSFC2, host B must send an ARP for its default gateway IP address, 10.1.2.1.

   ARP and MAC Address Tables After Host B Sends ARP for Its Default Gateway

   Host A ARP Table	Switch 1 MAC Address Table MAC VLAN Port	MSFC1 ARP Table	MSFC2 ARP Table	Switch 2 MAC Address Table MAC VLAN Port	Host B ARP Table
   10.1.1.1 : 0000.0c07.ac01	0000.0c00.0001 1 2/1	10.1.1.10 : 0000.0c00.0001	10.1.2.10 0000.0c00.0002	0000.0c00.0002 2 2/1	10.1.2.2 (0003.6bf1.2a01)
   	0000.0c00.0002 2 1/1	10.1.2.10 : 0000.0c00.0001			10.1.2.1 (0000.0c07.ac02)

5. Host B now forwards the echo reply packet to MSFC2. MSFC2 sends an ARP request for host A because it is directly connected on VLAN 1. Switch 2 populates its MAC address table with the MAC address of host B.

   ARP and MAC Address Tables After Echo Packet Has Been Received by Host A

   Host A ARP Table	Switch 1 MAC Address Table MAC VLAN Port	MSFC1 ARP Table	MSFC2 ARP Table	Switch 2 MAC Address Table MAC VLAN Port	Host B ARP Table
   10.1.1.1 : 0000.0c07.ac01	0000.0c00.0001 1 2/1	10.1.1.10 : 0000.0c00.0001	10.1.2.10 0000.0c00.0002	0000.0c00.0002 2 2/1	10.1.2.2 ( 0003.6bf1.2a01)
   10.1.1.3 : 0003.6bf1.2a0	0000.0c00.0002 2 1/1	10.1.2.10 : 0000.0c00.0001	10.1.1.10 0000.0c00.0001	0000.0c00.00001 1 1/1	10.1.2.1 (0000.0c07.ac02)

6. The echo reply reaches host A and the flow is complete.

Consequences of Asymmetric Routing

Consider the case of the continuous ping of host B by host A. Remember that host A sends the echo packet to MSFC1, and host B sends the echo reply to MSFC2, which is in an asymmetric routing state. The only time that Switch 1 learns the source MAC of host B is when host B replies to an ARP request from MSFC1. This is because host B uses MSFC2 as its default gateway and does not send packets to MSFC1 and, consequently, Switch 1. Since the ARP timeout is four hours by default, Switch 1 ages the MAC address of host B after five minutes by default. Switch 2 ages host A after five minutes. As a result, Switch 1 must treat any packet with a destination MAC of host B as an unknown unicast. The switch floods the packet that comes from host A and is destined for host B out all ports. In addition, because there is no MAC address entry host B in Switch 1, there is no MLS entry as well.

ARP and MAC Address Tables After 5 Minutes of Continuous Ping of Host B by Host A

The echo reply packets that come from host B experience the same issue after the MAC address entry for host A ages on Switch 2. Host B forwards the echo reply to MSFC2, which in turn routes the packet and sends it out on VLAN 1. The switch does not have an entry host A in the MAC address table and must flood the packet out all ports in VLAN 1.

Asymmetric routing issues do not break connectivity. But, asymmetric routing can cause excessive unicast flooding and MLS entries that are missing. There are three configuration changes that can remedy this situation:

• Adjust the MAC aging time on the respective switches to 14,400 seconds (four hours) or longer.

• Change the ARP timeout on the routers to five minutes (300 seconds).

• Change the MAC aging time and ARP timeout to the same timeout value.

The preferable method is to change the MAC aging time to 14,400 seconds. These are the configuration guidelines:

• Cisco IOS Software:

  mac address-table aging-time <seconds> vlan <vlan_id>

Case Study #6: HSRP Virtual IP Address Is Reported as a Different IP Address

The STANDBY-3-DIFFVIP1 error message occurs when there is interVLAN leakage because of bridging loops in the switch.

If you get this error message and there is interVLAN leakage because of bridging loops in the switch, complete these steps in order to resolve the error:

1. Identify the path that the packets take between end nodes.

   If there is a router on this path, complete these steps:

   1. Troubleshoot the path from the first switch to the router.

   2. Troubleshoot the path from the router to the second switch.

2. Connect to each switch on the path and check the status of the ports that are used on the path between end nodes. 

Case Study #7: HSRP Causes MAC Violation on a Secure Port

When port security is configured on the switch ports that are connected to the HSRP enabled routers, it causes a MAC violation, since you cannot have the same secure MAC address on more than one interface. A security violation occurs on a secure port in one of these situations:

• The maximum number of secure MAC addresses is added to the address table, and a station whose MAC address is not in the address table attempts to access the interface.

• An address that is learned or configured on one secure interface is seen on another secure interface in the same VLAN.

By default, a port security violation causes the switch interface to become error-disabled and to shutdown immediately, which blocks the HSRP status messages between the routers.

Workaround

• Issue the standby use-bia command on the routers. This forces the routers to use a burned-in address for HSRP instead of the virtual MAC address.

• Disable port security on the switch ports that connect to the HSRP enabled routers.

Case Study #9: %Interface Hardware Cannot Support Multiple Groups

If multiple HSRP groups are created on the interface, this error message is received:

%Interface hardware cannot support multiple groups

This error message is received due to the hardware limitation on some Routers or switches. It is not possible to overcome the limitation by any software methods. The problem is that each HSRP group uses one additional MAC address on interface, so the Ethernet MAC chip must support multiple programmable MAC addresses in order to enable several HSRP groups.

The workaround is to use the standby use-bia interface configuration command, which uses the Burned-In Address (BIA) of the interface as its virtual MAC address, instead of the preassigned MAC address.

Troubleshoot HSRP in Catalyst Switches

A. Verify HSRP Router Configuration

1. Verify Unique Router Interface IP Address

Verify that each HSRP router has a unique IP address for each subnet on a per-interface basis. Also, verify that each interface has the line protocol up. In order to quickly verify the current state of each interface, issue the show ip interface brief command. Here is an example:

Router_1#show ip interface brief
Interface              IP-Address      OK? Method Status                Protocol
Vlan1                  192.168.1.1     YES manual up                    up      
Vlan10                 192.168.10.1    YES manual up                    up      
Vlan11                 192.168.11.1    YES manual up                    up    

Router_2#show ip interface brief 
Interface              IP-Address      OK? Method Status                Protocol
Vlan1                  192.168.1.2     YES manual up                    up      
Vlan10                 192.168.10.2    YES manual up                    up      
Vlan11                 192.168.11.2    YES manual up  up

2. Verify Standby (HSRP) IP Addresses and Standby Group Numbers

Verify that the configured standby (HSRP) IP addresses and standby group numbers match each HSRP-participating router. A mismatch of standby groups or HSRP standby addresses can cause HSRP problems. The show standby command details the standby group and standby IP address configuration of each interface. Here is an example:

Router_1#show standby  
Vlan10 - Group 110
  State is Active
    2 state changes, last state change 00:01:34
  Virtual IP address is 192.168.10.100
  Active virtual MAC address is 0000.0c07.ac6e (MAC In Use)
    Local virtual MAC address is 0000.0c07.ac6e (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 0.144 secs
  Preemption enabled
  Active router is local
  Standby router is 192.168.10.2, priority 109 (expires in 10.784 sec)
  Priority 110 (configured 110)
  Group name is "hsrp-Vl10-110" (default)
  FLAGS: 0/1
Vlan11 - Group 111
  State is Active
    2 state changes, last state change 00:00:27
  Virtual IP address is 192.168.11.100
  Active virtual MAC address is 0000.0c07.ac6f (MAC In Use)
    Local virtual MAC address is 0000.0c07.ac6f (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 2.096 secs
  Preemption enabled
  Active router is local
  Standby router is 192.168.11.2, priority 109 (expires in 8.944 sec)
  Priority 110 (configured 110)
  Group name is "hsrp-Vl11-111" (default)
  FLAGS: 0/1

Router_2#show standby 
Vlan10 - Group 110
  State is Standby
    1 state change, last state change 00:03:15
  Virtual IP address is 192.168.10.100
  Active virtual MAC address is 0000.0c07.ac6e (MAC Not In Use)
    Local virtual MAC address is 0000.0c07.ac6e (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 1.088 secs
  Preemption disabled
  Active router is 192.168.10.1, priority 110 (expires in 11.584 sec)
  Standby router is local
  Priority 109 (configured 109)
  Group name is "hsrp-Vl10-110" (default)
  FLAGS: 0/1
Vlan11 - Group 111
  State is Standby
    1 state change, last state change 00:02:53
  Virtual IP address is 192.168.11.100
  Active virtual MAC address is 0000.0c07.ac6f (MAC Not In Use)
    Local virtual MAC address is 0000.0c07.ac6f (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 2.352 secs
  Preemption disabled
  Active router is 192.168.11.1, priority 110 (expires in 9.120 sec)
  Standby router is local
  Priority 109 (configured 109)
  Group name is "hsrp-Vl11-111" (default)
  FLAGS: 0/1

3. Verify That Standby (HSRP) IP Address Is Different per Interface

Verify that the standby (HSRP) IP address is unique from the configured IP address on each interface. The show standby command is a quick reference in order to view this information. Here is an example:

Router_1#show standby  
Vlan10 - Group 110
  State is Active
    2 state changes, last state change 00:01:34
  Virtual IP address is 192.168.10.100
  Active virtual MAC address is 0000.0c07.ac6e (MAC In Use)
    Local virtual MAC address is 0000.0c07.ac6e (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 0.144 secs
  Preemption enabled
  Active router is local
  Standby router is 192.168.10.2, priority 109 (expires in 10.784 sec)
  Priority 110 (configured 110)
  Group name is "hsrp-Vl10-110" (default)
  FLAGS: 0/1
Vlan11 - Group 111
  State is Active
    2 state changes, last state change 00:00:27
  Virtual IP address is 192.168.11.100
  Active virtual MAC address is 0000.0c07.ac6f (MAC In Use)
    Local virtual MAC address is 0000.0c07.ac6f (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 2.096 secs
  Preemption enabled
  Active router is local
  Standby router is 192.168.11.2, priority 109 (expires in 8.944 sec)
  Priority 110 (configured 110)
  Group name is "hsrp-Vl11-111" (default)
  FLAGS: 0/1

Router_2#show standby 
Vlan10 - Group 110
  State is Standby
    1 state change, last state change 00:03:15
  Virtual IP address is 192.168.10.100
  Active virtual MAC address is 0000.0c07.ac6e (MAC Not In Use)
    Local virtual MAC address is 0000.0c07.ac6e (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 1.088 secs
  Preemption disabled
  Active router is 192.168.10.1, priority 110 (expires in 11.584 sec)
  Standby router is local
  Priority 109 (configured 109)
  Group name is "hsrp-Vl10-110" (default)
  FLAGS: 0/1
Vlan11 - Group 111
  State is Standby
    1 state change, last state change 00:02:53
  Virtual IP address is 192.168.11.100
  Active virtual MAC address is 0000.0c07.ac6f (MAC Not In Use)
    Local virtual MAC address is 0000.0c07.ac6f (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 2.352 secs
  Preemption disabled
  Active router is 192.168.11.1, priority 110 (expires in 9.120 sec)
  Standby router is local
  Priority 109 (configured 109)
  Group name is "hsrp-Vl11-111" (default)
  FLAGS: 0/1 

4. When to Use the standy use-bia Command

Unless HSRP is configured on a Token Ring interface, only use the standby use-bia command in special circumstances. This command tells the router to use its BIA instead of the virtual HSRP MAC address for the HSRP group. On a Token Ring network, if source-route bridging (SRB) is in use, the standby use-bia command allows the new active router to update the host Routing Information Field (RIF) cache with a gratuitous ARP. But, not all of the host implementations handle the gratuitous ARP correctly. Another caveat for the standby use-bia command involves proxy ARP. A standby router cannot cover for the lost proxy ARP database of the failed active router.

5. Verify Access List Configuration

Verify that the access lists that are configured on all of the HSRP peers do not filter any HSRP addresses that are configured on their interfaces. Specifically, verify the multicast address that is used in order to send traffic to all of the routers on a subnet (224.0.0.2). Also, verify that the UDP traffic that is destined for the HSRP port 1985 is not filtered. HSRP uses this address and port to send hello packets between peers. Issue the show access-lists command as a quick reference to note the access lists that are configured on the router. Here is an example:

Router_1#show access-lists
Standard IP access list 77
    deny 10.19.0.0, wildcard bits 0.0.255.255 
    permit any
Extended IP access list 144
    deny pim 238.0.10.0 0.0.0.255 any 
    permit ip any any (58 matches)

B. Verify Catalyst Fast EtherChannel and Trunking Configuration

1. Verify Trunking Configuration

If a trunk is used in order to connect the HSRP routers, verify the trunking configurations on the routers and switches. There are five possible trunking modes:

• on

• desirable

• auto

• off

• nonegotiate

Verify that the trunking modes that are configured provide the desired trunking method. 

Use the desirable configuration for switch-to-switch connections when you troubleshoot HSRP issues. This configuration can isolate issues where switch ports are unable to establish trunks correctly. Set a router-to-switch configuration as nonegotiate because most Cisco IOS routers do not support negotiation of a trunk.

For IEEE 802.1Q (dot1q) trunking mode, verify that both sides of the trunk are configured to use the same native VLAN and encapsulation. Because Cisco products do not tag the native VLAN by default, a mismatch of native VLAN configurations results in no connectivity on mismatched VLANs. Lastly, verify that the trunk is configured to carry the VLANs that are configured on the router, and verify that the VLANs are not pruned and in the STP state for router-connected ports. Issue the show interfaces <interface> trunk  command for a quick reference that shows this information. Here is an example:

L2Switch_1#show interfaces gigabitEthernet1/0/13 trunk

Port        Mode             Encapsulation  Status        Native vlan
Gi1/0/13    on               802.1q         trunking      1

Port        Vlans allowed on trunk
Gi1/0/13    1-4094

Port        Vlans allowed and active in management domain
Gi1/0/13    1,10-11,70,100,300-309

Port        Vlans in spanning tree forwarding state and not pruned
Gi1/0/13    1,10-11,70,100,300-309

Router_1#show interfaces gigabitEthernet1/0/1 trunk 

Port        Mode             Encapsulation  Status        Native vlan  
Gi1/0/1     on               802.1q         trunking      1

Port        Vlans allowed on trunk
Gi1/0/1     1-4094

Port        Vlans allowed and active in management domain
Gi1/0/1     1,10-11,100,206,301,307,401,900,3001-3002

Port        Vlans in spanning tree forwarding state and not pruned
Gi1/0/1     1,10-11,100,206,301,307,401,900,3001-3002

2. Verify Fast EtherChannel (Port Channeling) Configuration

If a port channel is used in order to connect the HSRP routers, verify the EtherChannel configuration on both routers and switches. Configure a switch-to-switch port channel as desirable on at least one side. The other side can be in any of these modes:

• on

• desirable

• auto

However, in this example intefaces are not member of a port-channel:

Router_1#show etherchannel summary 
Flags:  D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator

        M - not in use, minimum links not met
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port

        A - formed by Auto LAG


Number of channel-groups in use: 0
Number of aggregators:           0

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------

Router_1#

Router_2#show etherchannel summary 
Flags:  D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator

        M - not in use, minimum links not met
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port

        A - formed by Auto LAG


Number of channel-groups in use: 0
Number of aggregators:           0

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------

Router_2#

3. Investigate Switch MAC Address Forwarding Table

Verify that the MAC address table entries exist on the switch for the HSRP routers for the HSRP virtual MAC address and the physical BIAs. The show standby command on the router provides the virtual MAC address. The show interface command provides the physical BIA. Here are sample outputs:

Router_1#show standby 
Vlan10 - Group 110
  State is Active
    2 state changes, last state change 00:37:03
  Virtual IP address is 192.168.10.100
  Active virtual MAC address is 0000.0c07.ac6e (MAC In Use)
    Local virtual MAC address is 0000.0c07.ac6e (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 0.768 secs
  Preemption enabled
  Active router is local
  Standby router is 192.168.10.2, priority 109 (expires in 10.368 sec)
  Priority 110 (configured 110)
  Group name is "hsrp-Vl10-110" (default)
  FLAGS: 0/1
Vlan11 - Group 111
  State is Active
    2 state changes, last state change 00:35:56
  Virtual IP address is 192.168.11.100
  Active virtual MAC address is 0000.0c07.ac6f (MAC In Use)
    Local virtual MAC address is 0000.0c07.ac6f (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 1.472 secs
  Preemption enabled
  Active router is local
  Standby router is 192.168.11.2, priority 109 (expires in 8.336 sec)
  Priority 110 (configured 110)
  Group name is "hsrp-Vl11-111" (default)
  FLAGS: 0/1

Router_1#show interfaces vlan 10
Vlan10 is up, line protocol is up , Autostate Enabled
  Hardware is Ethernet SVI, address is d4e8.801f.4846 (bia d4e8.801f.4846)
  Internet address is 192.168.10.1/24
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not supported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:01, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     9258 packets input, 803066 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles 
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     3034 packets output, 368908 bytes, 0 underruns
     Output 0 broadcasts (0 IP multicasts)
     0 output errors, 2 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out

L2Switch_1#show mac address-table address 0000.0c07.ac6e
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
  10    0000.0c07.ac6e    DYNAMIC     Gi1/0/13
Total Mac Addresses for this criterion: 1

L2Switch_1#show mac address-table address 0000.0c07.ac6f
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
  11    0000.0c07.ac6f    DYNAMIC     Gi1/0/13
Total Mac Addresses for this criterion: 1

Be sure to check the CAM aging time in order to determine how quickly the entries are aged. If the time equals the configured value for STP forward delay, which is 15 seconds by default, there is a strong possibility that there is an STP loop in the network. Here is sample command output:

L2Switch_1#show mac address-table aging-time  vlan 10
Global Aging Time:  300
Vlan    Aging Time
----    ----------
  10     300

L2Switch_1#show mac address-table aging-time  vlan 11
Global Aging Time:  300
Vlan    Aging Time
----    ----------
  11     300

C. Verify Physical Layer Connectivity

If more than one router in an HSRP group becomes active, those routers do not consistently receive the hello packets from fellow HSRP peers. Physical layer problems can prevent the consistent pass of traffic between peers and cause this scenario. Be sure to verify physical connectivity and IP connectivity between HSRP peers when you troubleshoot HSRP. Issue the show standby command in order to verify connectivity. Here is an example:

Router_1#show standby 
Vlan10 - Group 110
  State is Active
    2 state changes, last state change 00:54:03
  Virtual IP address is 192.168.10.100
  Active virtual MAC address is 0000.0c07.ac6e (MAC In Use)
    Local virtual MAC address is 0000.0c07.ac6e (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 0.848 secs
  Preemption enabled
  Active router is local
  Standby router is unknown
  Priority 110 (configured 110)
  Group name is "hsrp-Vl10-110" (default)
  FLAGS: 0/1
Vlan11 - Group 111
  State is Active
    2 state changes, last state change 00:52:56
  Virtual IP address is 192.168.11.100
  Active virtual MAC address is 0000.0c07.ac6f (MAC In Use)
    Local virtual MAC address is 0000.0c07.ac6f (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 0.512 secs
  Preemption enabled
  Active router is local
  Standby router is unknown
  Priority 110 (configured 110)
  Group name is "hsrp-Vl11-111" (default)
  FLAGS: 0/1

Router_2#show standby 
Vlan10 - Group 110
  State is Init (interface down)
    2 state changes, last state change 00:00:42
  Virtual IP address is 192.168.10.100
  Active virtual MAC address is unknown (MAC Not In Use)
    Local virtual MAC address is 0000.0c07.ac6e (v1 default)
  Hello time 3 sec, hold time 10 sec
  Preemption disabled
  Active router is unknown
  Standby router is unknown
  Priority 109 (configured 109)
  Group name is "hsrp-Vl10-110" (default)
  FLAGS: 0/1
Vlan11 - Group 111
  State is Init (interface down)
    2 state changes, last state change 00:00:36
  Virtual IP address is 192.168.11.100
  Active virtual MAC address is unknown (MAC Not In Use)
    Local virtual MAC address is 0000.0c07.ac6f (v1 default)
  Hello time 3 sec, hold time 10 sec
  Preemption disabled
  Active router is unknown
  Standby router is unknown
  Priority 109 (configured 109)
  Group name is "hsrp-Vl11-111" (default)
  FLAGS: 0/1

1. Check Interface Status

Check the interfaces. Verify that all HSRP-configured interfaces are up/up, as this example shows:

Router_1#show ip interface brief 
Interface              IP-Address      OK? Method Status                Protocol
Vlan1                  192.168.1.1     YES manual up                    up      
Vlan10                 192.168.10.1    YES manual up                    up      
Vlan11                 192.168.11.1    YES manual up                    up      

Router_2#show ip interface brief 
Interface              IP-Address      OK? Method Status                Protocol
Vlan1                  192.168.1.2     YES manual up                    up      
Vlan10                 192.168.10.2    YES manual administratively down down    
Vlan11                 192.168.11.2    YES manual administratively down down    

If any interfaces are administratively down/down, enter the configuration mode on the router and issue the no shutdown interface-specific command. Here is an example:

Router_2#configure terminal 
Enter configuration commands, one per line.  End with CNTL/Z. 
Router_2(config)#interface vlan 10 
Router_2(config-if)#no shutdown 
Router_2(config-if)#end 

Router_2#configure terminal 
Enter configuration commands, one per line.  End with CNTL/Z. 
Router_2(config)#interface vlan 11 
Router_2(config-if)#no shutdown 
Router_2(config-if)#end 

Router_2#show ip interface brief 
Interface              IP-Address      OK? Method Status                Protocol
Vlan1                  192.168.1.2     YES manual up                    up      
Vlan10                 192.168.10.2    YES manual up                    down      
Vlan11                 192.168.11.2    YES manual up                    up   

If any interfaces are down/down or up/down, review the log for any interface change notifications. For Cisco IOS Software-based switches, these messages appear for link up/down situations:

%LINK-3-UPDOWN: Interface "interface", changed state to up
%LINK-3-UPDOWN: Interface "interface", changed state to down
  
Router_1#show log 
3d04h: %STANDBY-6-STATECHANGE: Standby: 0: Vlan10 state Active-> Speak 
3d04h: %LINK-5-CHANGED: Interface Vlan10, changed state to down 
3d04h: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan10, changed state to down 

Inspect the ports, cables, and any transceivers or other devices that are between the HSRP peers. Has anyone removed or loosened any connections? Are there any interfaces that lose a link repeatedly? Are the proper cable types used? Check the interfaces for any errors, as this example shows:

Router_2#show interface vlan 10
Vlan10 is down, line protocol is down , Autostate Enabled
  Hardware is Ethernet SVI, address is 1880.90d8.5946 (bia 1880.90d8.5946)
  Internet address is 192.168.10.2/24
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not supported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:10, output 00:00:08, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     1243 packets input, 87214 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles 
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     23 packets output, 1628 bytes, 0 underruns
     Output 0 broadcasts (0 IP multicasts)
     0 output errors, 2 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out

2. Link Change and Port Errors

Check the switch ports link changes and other errors. Issue these commands and review the output:

• show logging

• show interfaces <interface> counters

• show interfaces <interface> status

These commands help you determine if there is a problem with connectivity between switches and other devices.

These messages are normal for link up/down situations:

L2Switch_1#show logging       
Syslog logging: enabled (0 messages dropped, 5 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.



No Inactive Message Discriminator.


    Console logging: level informational, 319 messages logged, xml disabled,
                     filtering disabled
    Monitor logging: level debugging, 0 messages logged, xml disabled,
                     filtering disabled
    Buffer logging:  level debugging, 467 messages logged, xml disabled,
                    filtering disabled
    Exception Logging: size (4096 bytes)
    Count and timestamp logging messages: disabled
    File logging: disabled
    Persistent logging: disabled

No active filter modules.

    Trap logging: level informational, 327 message lines logged
        Logging Source-Interface:       VRF Name:

Log Buffer (10000 bytes):
*Jul 26 17:52:07.526: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/13, changed state to up
*Jul 26 17:52:09.747: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/13, changed state to down
*Jul 26 17:57:11.716: %SPANTREE-7-RECV_1Q_NON_TRUNK: Received 802.1Q BPDU on non trunk GigabitEthernet1/0/16 VLAN307.
*Jul 26 17:57:11.716: %SPANTREE-7-BLOCK_PORT_TYPE: Blocking GigabitEthernet1/0/16 on VLAN0307. Inconsistent port type.
*Jul 26 17:57:13.583: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/16, changed state to up
*Jul 26 17:57:16.237: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/16, changed state to down
*Jul 26 18:02:16.481: %SPANTREE-7-RECV_1Q_NON_TRUNK: Received 802.1Q BPDU on non trunk GigabitEthernet1/0/16 VLAN307.
*Jul 26 18:02:16.481: %SPANTREE-7-BLOCK_PORT_TYPE: Blocking GigabitEthernet1/0/16 on VLAN0307. Inconsistent port type.
*Jul 26 18:02:18.367: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/16, changed state to up
*Jul 26 18:02:20.561: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/16, changed state to down

Issue the show interfaces <interface> status command in order to determine the general health of a port. Here is an example:

L2Switch_1#show interfaces gigabitEthernet 1/0/13 status

Port      Name               Status       Vlan       Duplex  Speed Type 
Gi1/0/13                     connected    trunk      a-full a-1000 10/100/1000BaseTX

Is the interface status connected, notconnect, or errdisable? If the status is notconnect, check that the cable is plugged in on both sides. Check that the proper cable is used. If the status is errdisable, review the counters for excessive errors. Refer to Recover Errdisable Port State on Cisco IOS Platforms for more information.

For what VLAN is this port configured? Be sure that the other side of the connection is configured for the same VLAN. If the link is configured to be a trunk, be sure that both sides of the trunk carry the same VLANs.

What is the speed and duplex configuration? If the setting is preceded by a-, the port is configured to autonegotiate the speed and duplex. Otherwise, the network administrator has predetermined this configuration. For configuration of the speed and duplex for a link, the settings on both sides of the link must match. If one switch port is configured for autonegotiation, the other side of the link must also be configured for autonegotiation. If one side is hard coded to a specific speed and duplex, the other side must be hard coded as well. If you leave one side to autonegotiate while the other side is hard coded, you break the autonegotiation process.

L2Switch_1#show interfaces gi1/0/13 counters errors 

Port        Align-Err     FCS-Err    Xmit-Err     Rcv-Err  UnderSize  OutDiscards 
Gi1/0/13            0           0           0           0          0            0 

Port      Single-Col  Multi-Col   Late-Col  Excess-Col  Carri-Sen      Runts 
Gi1/0/13           0          0          0           0          0          0 

Are there a lot of Align-Err, FCS-Err, or Runts? These indicate a speed or duplex mismatch between the port and the connecting device. Change the speed and duplex settings for that port in order to help correct these errors.

Issue the show mac command in order to verify that the port is passing traffic. The In and Out columns indicate the number of unicast, multicast, and broadcast packets that are received and transmitted on a particular port. The bottom counters reveal how many packets are discarded or lost and whether these packets are a part of inbound or outbound traffic. Lrn-Discrd, In-Lost, and Out-Lost count the number of packets that are mistakenly forwarded or dropped due to insufficient buffers.

L2Switch_1#show interfaces gi1/0/13 counters              

Port            InOctets    InUcastPkts    InMcastPkts    InBcastPkts 
Gi1/0/13       304933333        1180453        1082538          14978 

Port           OutOctets   OutUcastPkts   OutMcastPkts   OutBcastPkts 
Gi1/0/13       282752538         276716         824562         588960 

3. Verify IP Connectivity

Verify IP connectivity. Issue an IP ping from the associated router to the remote HSRP device. This helps expose any momentary losses of connectivity. An extended ping is only available in enable mode. Here is sample command output:

Router_1#show run interface vlan 10
Building configuration...

Current configuration : 141 bytes
!
interface Vlan10
 ip address 192.168.10.1 255.255.255.0
 standby 110 ip 192.168.10.100
 standby 110 priority 110
 standby 110 preempt
end

Router_2#show run interface vlan 10
Building configuration...

Current configuration : 120 bytes
!
interface Vlan10
 ip address 192.168.10.2 255.255.255.0
 standby 110 ip 192.168.10.100
 standby 110 priority 109
end

Router_1#ping 192.168.10.2 repeat 1500
Type escape sequence to abort.
Sending 1500, 100-byte ICMP Echos to 192.168.10.2, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (1500/1500), round-trip min/avg/max = 1/2/9 ms

Issue the ping from each HSRP router to its peers in order to determine the location of the failure in connectivity.

4. Check for Unidirectional Link

Check the switch for unidirectional links between HSRP peers. A unidirectional link occurs whenever traffic that a local device transmits over a link is received by the neighbor, but traffic that the neighbor transmits is not received by the local device. This feature is known as UniDirectional Link Detection (UDLD) aggressive mode. Use of UDLD is only possible if both sides of the connection support the feature. UDLD aggressive mode operates at L2 to determine if a link is connected correctly and if traffic flows bidirectionally between the right neighbors. Here are sample command outputs:

Note: Navigate to the next link to Understand and Configure UDLD Feature  It depends on which platform is used.

Another option that can help to verify a Unidirectional Link if UDLD is not available is with the use of Cisco Discovery Protocol (CDP). Enablement of CDP is another way to detect if a unidirectional link exists. If only one side of a link can see its neighbor device, replace the cable between the devices and check for faulty interfaces. 

Router_1#show cdp
Global CDP information:
        Sending CDP packets every 60 seconds
        Sending a holdtime value of 180 seconds
        Sending CDPv2 advertisements is  enabled

Router_1#show cdp neighbors gi1/0/1 detail 
-------------------------
Device ID: L2Switch_1.cisco.com
Entry address(es): 
  IP address: 192.168.70.1
  IPv6 address: 2001:420:140E:2101::1  (global unicast)
  IPv6 address: FE80::2FE:C8FF:FED3:86C7  (link-local)
Platform: cisco WS-C3650-12X48UR,  Capabilities: Router Switch IGMP 
Interface: GigabitEthernet1/0/1,  Port ID (outgoing port): GigabitEthernet1/0/13
Holdtime : 173 sec

Version :
Cisco IOS Software [Denali], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.3.8, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Wed 13-Feb-19 03:00 by mcpre

advertisement version: 2
VTP Management Domain: 'CALOnet'
Native VLAN: 1
Duplex: full
Management address(es): 
  IP address: 192.168.70.1
Spare Pair PoE: Yes, Spare Pair Detection Required: No
Spare Pair PD Config: Disable, Spare Pair PSE Operational: No


Total cdp entries displayed : 1

5. Additional Physical Layer Troubleshooting References

Refer to these documents:

• Configuring and Troubleshooting Ethernet 10/100/1000Mb Half/Full Duplex Auto-Negotiation

• Recover Errdisable Port State on Cisco IOS Platforms

• Troubleshooting Cisco Catalyst Switches to NIC Compatibility Issues

• The Understanding Data Link Errors section of Troubleshooting Cisco Catalyst Switches to NIC Compatibility Issues

• Troubleshooting Switch Port and Interface Problems

D. Layer 3 HSRP Debugging

If the HSRP state changes are frequent, use the HSRP debug commands (in enable mode) on the router in order to watch HSRP activity. This information helps you determine what HSRP packets are received and sent by the router. Gather this information if you create a service request with Cisco Technical Support. The debug output also shows HSRP state information, along with detailed HSRP hello packet accounts.

1. Standard HSRP Debugging

In Cisco IOS enable the HSRP debug capability with the command debug standby. This information is useful where problems are intermittent and affect only a few interfaces. The debug enables you to determine if the HSRP router in question receives and transmits HSRP hello packets at specific intervals. If the router does not receive hello packets, you can infer that either the peer does not transmit the hello packets or the network drops the packets.

Here is sample command output:

Router_1#debug standby 
HSRP debugging is on
Jul 29 16:12:16.889: HSRP: Vl10 Grp 110 Hello  out 192.168.10.1 Active  pri 110 vIP 192.168.10.100
Jul 29 16:12:16.996: HSRP: Vl11 Grp 111 Hello  in  192.168.11.2 Standby pri 109 vIP 192.168.11.100
Jul 29 16:12:17.183: HSRP: Vl10 Grp 110 Hello  in  192.168.10.2 Standby pri 109 vIP 192.168.10.100
Jul 29 16:12:17.366: HSRP: Vl11 Grp 111 Hello  out 192.168.11.1 Active  pri 110 vIP 192.168.11.100
Jul 29 16:12:18.736: HSRP: Vl10 Interface adv in, Passive, active 0, passive 1, from 192.168.10.2
Jul 29 16:12:19.622: HSRP: Vl10 Grp 110 Hello  out 192.168.10.1 Active  pri 110 vIP 192.168.10.100

2. Conditional HSRP Debugging (Limiting Output Based on Standby Group and/or VLAN)

Cisco IOS Software Release 12.0(3) introduced a debug condition to allow the output from the debug standby command to be filtered based on interface and group number. The command utilizes the debug condition paradigm that was introduced in Cisco IOS Software Release 12.0.

The interface must be a valid interface that can support HSRP. The group can be any group, from 0 through 255. A debug condition can be set for groups that do not exist. This allows debugs to be captured during the initialization of a new group. Debug standby must be enabled in order to produce any debug output. If no standby debug conditions exist, debug output is produced for all groups on all interfaces. If at least one standby debug condition exists, standby debug output is filtered based on all of the standby debug conditions. Here is sample command output:

Router_1#debug condition standby vlan 10 110 
Condition 1 set 
Router_1# 
Jul 29 16:16:20.284: Vl10 HSRP110 Debug: Condition 1, hsrp Vl10 HSRP110 triggered, count 1
Router_1#debug standby 
HSRP debugging is on 
Router_1# 
Jul 29 16:16:44.797: HSRP: Vl10 Grp 110 Hello out 192.168.10.1 Active pri 110 vIP 192.168.10.100
Jul 29 16:16:45.381: HSRP: Vl10 Grp 110 Hello in 192.168.10.2 Standby pri 109 vIP 192.168.10.100
Jul 29 16:16:47.231: HSRP: Vl10 Grp 110 Hello out 192.168.10.1 Active pri 110 vIP 192.168.10.100
Jul 29 16:16:48.248: HSRP: Vl10 Grp 110 Hello in 192.168.10.2 Standby pri 109 vIP 192.168.10.100  

3. Enhanced HSRP Debugging

Cisco IOS Software Release 12.1(1) added enhanced HSRP debugging. In order to help find useful information, enhanced HSRP debugging limits the noise from periodic hello messages and includes additional state information. This information is particularly useful when you work with a Cisco Technical Support engineer if you create a service request.

Here is sample command output:

Router_2#debug standby terse 
HSRP:
  HSRP Errors debugging is on
  HSRP Events debugging is on
    (protocol, neighbor, redundancy, track, ha, arp, interface)
  HSRP Packets debugging is on
    (Coup, Resign)
Router_2#
*Jul 29 16:49:35.416: HSRP: Vl10 Grp 110 Resign in  192.168.10.1 Active  pri 110 vIP 192.168.10.100
*Jul 29 16:49:35.416: HSRP: Vl10 Grp 110 Standby: i/Resign rcvd (110/192.168.10.1)
*Jul 29 16:49:35.416: HSRP: Vl10 Grp 110 Active router is local, was 192.168.10.1
*Jul 29 16:49:35.416: HSRP: Vl10 Nbr 192.168.10.1 no longer active for group 110 (Standby)
*Jul 29 16:49:35.417: HSRP: Vl10 Nbr 192.168.10.1 Was active or standby - start passive holddown
*Jul 29 16:49:35.417: HSRP: Vl10 Grp 110 Standby router is unknown, was local
*Jul 29 16:49:35.417: HSRP: Vl10 Grp 110 Standby -> Active
*Jul 29 16:49:35.418: %HSRP-5-STATECHANGE: Vlan10 Grp 110 state Standby -> Active
*Jul 29 16:49:35.418: HSRP: Peer not present
*Jul 29 16:49:35.418: HSRP: Vl10 Grp 110 Redundancy "hsrp-Vl10-110" state Standby -> Active
*Jul 29 16:49:35.419: HSRP: Vl10 Grp 110 Added 192.168.10.100 to ARP (0000.0c07.ac6e)
*Jul 29 16:49:35.420: HSRP: Vl10 IP Redundancy "hsrp-Vl10-110" standby, local -> unknown
*Jul 29 16:49:35.421: HSRP: Vl10 IP Redundancy "hsrp-Vl10-110" update, Standby -> Active
*Jul 29 16:49:38.422: HSRP: Vl10 IP Redundancy "hsrp-Vl10-110" update, Active -> Active

You can use interface and/or HSRP group conditional debugging in order to filter this debug output.

In this example, the router joins a preexisting HSRP group:

Rotuer_2#debug condition standby vlan 10 110
Condition 1 set
Router_2#debug condition interface gigabitEthernet 1/0/1 vlan-id 10 
Condition 2 set
Router_2#debug standby 
HSRP debugging is on
Router_2#
*Jul 29 16:54:12.496: HSRP: Vl10 Grp 110 Hello  out 192.168.10.2 Active  pri 109 vIP 192.168.10.100
*Jul 29 16:54:15.122: HSRP: Vl10 Grp 110 Hello  out 192.168.10.2 Active  pri 109 vIP 192.168.10.100
*Jul 29 16:54:17.737: HSRP: Vl10 Grp 110 Hello  out 192.168.10.2 Active  pri 109 vIP 192.168.10.100
*Jul 29 16:54:18.880: HSRP: Vl10 Nbr 192.168.10.1 is passive
*Jul 29 16:54:20.316: HSRP: Vl10 Grp 110 Hello  out 192.168.10.2 Active  pri 109 vIP 192.168.10.100
*Jul 29 16:54:20.322: HSRP: Vl10 Grp 110 Coup   in  192.168.10.1 Listen  pri 110 vIP 192.168.10.100
*Jul 29 16:54:20.323: HSRP: Vl10 Grp 110 Active: j/Coup rcvd from higher pri router (110/192.168.10.1)
*Jul 29 16:54:20.323: HSRP: Vl10 Grp 110 Active router is 192.168.10.1, was local
*Jul 29 16:54:20.323: HSRP: Vl10 Nbr 192.168.10.1 is no longer passive
*Jul 29 16:54:20.324: HSRP: Vl10 Nbr 192.168.10.1 active for group 110
*Jul 29 16:54:20.324: HSRP: Vl10 Grp 110 Active -> Speak
*Jul 29 16:54:20.325: %HSRP-5-STATECHANGE: Vlan10 Grp 110 state Active -> Speak
*Jul 29 16:54:20.325: HSRP: Peer not present
*Jul 29 16:54:20.325: HSRP: Vl10 Grp 110 Redundancy "hsrp-Vl10-110" state Active -> Speak
*Jul 29 16:54:20.326: HSRP: Vl10 Grp 110 Removed 192.168.10.100 from ARP
*Jul 29 16:54:20.326: HSRP: Vl10 Grp 110 Deactivating MAC 0000.0c07.ac6e
*Jul 29 16:54:20.327: HSRP: Vl10 Grp 110 Removing 0000.0c07.ac6e from MAC address filter
*Jul 29 16:54:20.328: HSRP: Vl10 Grp 110 Hello  out 192.168.10.2 Speak   pri 109 vIP 192.168.10.100
*Jul 29 16:54:20.328: HSRP: Vl10 Grp 110 Hello  in  192.168.10.1 Active  pri 110 vIP 192.168.10.100
*Jul 29 16:54:23.104: HSRP: Vl10 Grp 110 Hello  out 192.168.10.2 Speak   pri 109 vIP 192.168.10.100
*Jul 29 16:54:23.226: HSRP: Vl10 Grp 110 Hello  in  192.168.10.1 Active  pri 110 vIP 192.168.10.100
*Jul 29 16:54:25.825: HSRP: Vl10 Grp 110 Hello  in  192.168.10.1 Active  pri 110 vIP 192.168.10.100
*Jul 29 16:54:25.952: HSRP: Vl10 Grp 110 Hello  out 192.168.10.2 Speak   pri 109 vIP 192.168.10.100
*Jul 29 16:54:28.427: HSRP: Vl10 Grp 110 Hello  in  192.168.10.1 Active  pri 110 vIP 192.168.10.100
*Jul 29 16:54:28.772: HSRP: Vl10 Grp 110 Hello  out 192.168.10.2 Speak   pri 109 vIP 192.168.10.100
*Jul 29 16:54:30.727: HSRP: Vl10 Grp 110 Speak: d/Standby timer expired (unknown)
*Jul 29 16:54:30.727: HSRP: Vl10 Grp 110 Standby router is local
*Jul 29 16:54:30.727: HSRP: Vl10 Grp 110 Speak -> Standby
*Jul 29 16:54:30.727: %HSRP-5-STATECHANGE: Vlan10 Grp 110 state Speak -> Standby
*Jul 29 16:54:30.728: HSRP: Peer not present
*Jul 29 16:54:30.728: HSRP: Vl10 Grp 110 Redundancy "hsrp-Vl10-110" state Speak -> Standby
*Jul 29 16:54:30.728: HSRP: Vl10 Grp 110 Hello  out 192.168.10.2 Standby pri 109 vIP 192.168.10.100
*Jul 29 16:54:31.082: HSRP: Vl10 Grp 110 Hello  in  192.168.10.1 Active  pri 110 vIP 192.168.10.100
*Jul 29 16:54:33.459: HSRP: Vl10 Grp 110 Hello  out 192.168.10.2 Standby pri 109 vIP 192.168.10.100
*Jul 29 16:54:33.811: HSRP: Vl10 Grp 110 Hello  in  192.168.10.1 Active  pri 110 vIP 192.168.10.100
*Jul 29 16:54:36.344: HSRP: Vl10 Grp 110 Hello  out 192.168.10.2 Standby pri 109 vIP 192.168.10.100
*Jul 29 16:54:36.378: HSRP: Vl10 Grp 110 Hello  in  192.168.10.1 Active  pri 110 vIP 192.168.10.100
*Jul 29 16:54:38.856: HSRP: Vl10 Grp 110 Hello  in  192.168.10.1 Active  pri 110 vIP 192.168.10.100
*Jul 29 16:54:38.876: HSRP: Vl10 Grp 110 Hello  out 192.168.10.2 Standby pri 109 vIP 192.168.10.100
*Jul 29 16:54:41.688: HSRP: Vl10 Grp 110 Hello  out 192.168.10.2 Standby pri 109 vIP 192.168.10.100
*Jul 29 16:54:41.717: HSRP: Vl10 Grp 110 Hello  in  192.168.10.1 Active  pri 110 vIP 192.168.10.100

E. Spanning Tree Troubleshooting

STP loop conditions or instability in a network can prevent proper communication of HSRP peers. Because of this improper communication, each peer becomes an active router. STP loops can cause broadcast storms, duplicated frames, and MAC table inconsistency. All of these problems affect the entire network, and especially HSRP. HSRP error messages can be the first indication of an STP issue.

When you troubleshoot STP, you must understand the STP topology of the network on each VLAN. You must determine what switch is the root bridge and which ports on the switch are on blocking and forwarding. Because each VLAN has its own STP topology, this information is very important on each VLAN.

1. Verify Spanning Tree Configuration

Be sure that STP is configured on every switch and bridging device in the network. Take note of where each switch believes the root bridge is located. Also, note the values of these timers:

• Root Max Age

• Hello Time

• Forward Delay

Issue the show spanning-tree command in order to see all of this information. By default, the command shows this information for all VLANs. But, you can also filter other VLAN information if you supply the VLAN number with the command. This information is very useful when you troubleshoot STP issues.

These three timers that you note in the show spanning-tree output are learned from the root bridge. These timers do not need to match the timers that are set on that specific bridge. But, be sure that the timers match the root bridge in the case that this switch becomes the root bridge at any point. This match of the timers to the root bridge helps maintain continuity and ease of administration. The match also prevents a switch with incorrect timers from crippling the network.

Note: Enable STP for all VLANs at all times, regardless of whether there are redundant links in the network. If you enable STP in nonredundant networks, you prevent a breakage. A breakage can occur if someone bridges switches together with hubs or other switches and accidentally creates a physical loop. STP is also very useful in the isolation of specific problems. If the enablement of STP affects the operation of something in the network, there can be an existing problem that you need to isolate.

Here is sample output of the show spanning-tree command:

L2Switch_1#show spanning-tree vlan 10

VLAN0010
  Spanning tree enabled protocol rstp
  Root ID    Priority    32778
             Address     00fe.c8d3.8680
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32778  (priority 32768 sys-id-ext 10)
             Address     00fe.c8d3.8680
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/3             Desg FWD 4         128.3    P2p 
Gi1/0/10            Desg FWD 4         128.10   P2p Edge 
Gi1/0/11            Desg FWD 4         128.11   P2p 
Gi1/0/13            Desg FWD 4         128.13   P2p 
Gi1/0/14            Desg FWD 4         128.14   P2p 
Gi1/0/15            Desg FWD 4         128.15   P2p 
Gi1/0/16            Desg FWD 4         128.16   P2p 
Gi1/0/35            Desg FWD 4         128.35   P2p 


L2Switch_1#show spanning-tree vlan 11

VLAN0011
  Spanning tree enabled protocol rstp
  Root ID    Priority    32779
             Address     00fe.c8d3.8680
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32779  (priority 32768 sys-id-ext 11)
             Address     00fe.c8d3.8680
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/3             Desg FWD 4         128.3    P2p 
Gi1/0/10            Desg FWD 4         128.10   P2p Edge 
Gi1/0/11            Desg FWD 4         128.11   P2p 
Gi1/0/13            Desg FWD 4         128.13   P2p 
Gi1/0/14            Desg FWD 4         128.14   P2p 
Gi1/0/15            Desg FWD 4         128.15   P2p 
Gi1/0/16            Desg FWD 4         128.16   P2p 
Gi1/0/35            Desg FWD 4         128.35   P2p 

Switch L2Switch_1 is the root of VLAN 10 and VLAN 11.

2. Spanning Tree Loop Conditions

In order for an STP loop to occur, there must be L2 physical redundancy in the network. An STP does not occur if there is no possibility of a physical loop condition. Symptoms of an STP loop condition are:

• Total network outage

• Loss of connectivity

• The report by network equipment of high process and system utilization

A single VLAN that experiences an STP loop condition can congest a link and starve the other VLANs of bandwidth. The show interfaces <interface> controller command notes which ports transmit or receive an excessive number of packets. Excessive broadcast and multicast can indicate ports that are part of an STP loop. As a general rule, suspect a link of an STP loop condition any time that multicast or broadcast exceeds the number of unicast packets.

Note: The switch also counts STP bridge protocol data units (BPDUs) that are received and transmitted as multicast frames. A port that is in the STP blocking state still transmits and receives STP BPDUs.

Router_2#show interfaces gi1/0/1 controller 
GigabitEthernet1/0/1 is up, line protocol is up (connected) 
  Hardware is Gigabit Ethernet, address is 1880.90d8.5901 (bia 1880.90d8.5901)
  Description: PNP STARTUP VLAN
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is on, output flow-control is unsupported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:04, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 33000 bits/sec, 31 packets/sec
  5 minute output rate 116000 bits/sec, 33 packets/sec
     9641686 packets input, 1477317083 bytes, 0 no buffer
     Received 1913802 broadcasts (1151766 multicasts)
     0 runts, 0 giants, 0 throttles 
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 1151766 multicast, 0 pause input
     0 input packets with dribble condition detected
     10702696 packets output, 4241534645 bytes, 0 underruns
     Output 3432 broadcasts (0 multicasts)
     0 output errors, 0 collisions, 2 interface resets
     9582 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
Transmit                  GigabitEthernet1/0/1          Receive                 
   4241534645 Total bytes                  1477317083 Total bytes              
     10562003 Unicast frames                  7727884 Unicast frames           
   4229489212 Unicast bytes                1291270617 Unicast bytes            
       137261 Multicast frames                1151766 Multicast frames         
     11812065 Multicast bytes                91096867 Multicast bytes          
         3432 Broadcast frames                 762036 Broadcast frames         
       233368 Broadcast bytes                94949599 Broadcast bytes          
            0 System FCS error frames               0 IpgViolation frames      
            0 MacUnderrun frames                    0 MacOverrun frames        
            0 Pause frames                          0 Pause frames             
            0 Cos 0 Pause frames                    0 Cos 0 Pause frames       
            0 Cos 1 Pause frames                    0 Cos 1 Pause frames       
            0 Cos 2 Pause frames                    0 Cos 2 Pause frames       
            0 Cos 3 Pause frames                    0 Cos 3 Pause frames       
            0 Cos 4 Pause frames                    0 Cos 4 Pause frames       
            0 Cos 5 Pause frames                    0 Cos 5 Pause frames       
            0 Cos 6 Pause frames                    0 Cos 6 Pause frames       
            0 Cos 7 Pause frames                    0 Cos 7 Pause frames       
            0 Oam frames                            0 OamProcessed frames      
            0 Oam frames                            0 OamDropped frames        
        38144 Minimum size frames             4165201 Minimum size frames      
      4910833 65 to 127 byte frames           3126489 65 to 127 byte frames    
      1237675 128 to 255 byte frames           750243 128 to 255 byte frames   
      1029126 256 to 511 byte frames          1279281 256 to 511 byte frames   
      2205966 512 to 1023 byte frames          103668 512 to 1023 byte frames  
      1280952 1024 to 1518 byte frames         205229 1024 to 1518 byte frames 
            0 1519 to 2047 byte frames          11575 1519 to 2047 byte frames 
            0 2048 to 4095 byte frames              0 2048 to 4095 byte frames 
            0 4096 to 8191 byte frames              0 4096 to 8191 byte frames 
            0 8192 to 16383 byte frames             0 8192 to 16383 byte frames
            0 16384 to 32767 byte frame             0 16384 to 32767 byte frame
            0 > 32768 byte frames                   0 > 32768 byte frames      
            0 Late collision frames                 0 SymbolErr frames         
            0 Excess Defer frames                   0 Collision fragments      
            0 Good (1 coll) frames                  0 ValidUnderSize frames    
            0 Good (>1 coll) frames                 0 InvalidOverSize frames   
            0 Deferred frames                       0 ValidOverSize frames     
            0 Gold frames dropped                   0 FcsErr frames            
            0 Gold frames truncated    
            0 Gold frames successful   
            0 1 collision frames       
            0 2 collision frames       
            0 3 collision frames       
            0 4 collision frames       
            0 5 collision frames       
            0 6 collision frames       
            0 7 collision frames       
            0 8 collision frames       
            0 9 collision frames       
            0 10 collision frames      
            0 11 collision frames      
            0 12 collision frames      
            0 13 collision frames      
            0 14 collision frames      
            0 15 collision frames      
            0 Excess collision frames  

LAST UPDATE 2384 msecs AGO

3. Toplogy Change Notification

Another command that is vital to the diagnosis of STP issues is the show spanning-tree detail command. This command tracks Topology Change Notification (TCN) messages back to the originator. These messages, sent as special BPDUs between switches, indicate that there has been a topology change on a switch. That switch sends a TCN out its root port. The TCN moves upstream to the root bridge. The root bridge then sends another special BPDU, a Topology Change Acknowledgement (TCA), out all of its ports. The root bridge sets the TCN bit in the configuration BPDU. This causes all nonroot bridges to set their MAC address table aging timer to the configuration STP forward delay.

In order to isolate this problem, access the root bridge for each VLAN and issue the show spanning-tree <interface> detail command for the switch-connected ports. The last change occurred entry gives the time that the last TCN was received. In this situation, you are too late to see who issued the TCNs that can have caused the possible STP loop. The Number of topology changes entry gives you an idea about the number of TCNs that occur. During an STP loop, this counter can increment every minute. Refer to Spanning Tree Protocol Problems and Related Design Considerations for more information.

Other useful information includes:

• Port of the last TCN

• Time of last TCN

• Current count of TCNs

Here is sample command output:

L2Switch_1#show spanning-tree vlan 10 detail 

 VLAN0010 is executing the rstp compatible Spanning Tree protocol
  Bridge Identifier has priority 32768, sysid 10, address 00fe.c8d3.8680
  Configured hello time 2, max age 20, forward delay 15, transmit hold-count 6
  We are the root of the spanning tree
  Topology change flag not set, detected flag not set
  Number of topology changes 8 last change occurred 03:21:48 ago
          from GigabitEthernet1/0/35
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15 
  Timers: hello 0, topology change 0, notification 0, aging 300

 Port 3 (GigabitEthernet1/0/3) of VLAN0010 is designated forwarding 
   Port path cost 4, Port priority 128, Port Identifier 128.3.
   Designated root has priority 32778, address 00fe.c8d3.8680
   Designated bridge has priority 32778, address 00fe.c8d3.8680
   Designated port id is 128.3, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 1
   Link type is point-to-point by default
   BPDU: sent 6066, received 0

 Port 10 (GigabitEthernet1/0/10) of VLAN0010 is designated forwarding 
   Port path cost 4, Port priority 128, Port Identifier 128.10.
   Designated root has priority 32778, address 00fe.c8d3.8680
   Designated bridge has priority 32778, address 00fe.c8d3.8680
   Designated port id is 128.10, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 1
   The port is in the portfast mode by portfast trunk configuration
   Link type is point-to-point by default
   BPDU: sent 6063, received 0

 Port 11 (GigabitEthernet1/0/11) of VLAN0010 is designated forwarding 
   Port path cost 4, Port priority 128, Port Identifier 128.11.
   Designated root has priority 32778, address 00fe.c8d3.8680
   Designated bridge has priority 32778, address 00fe.c8d3.8680
   Designated port id is 128.11, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 1
   Link type is point-to-point by default
   BPDU: sent 6066, received 0

 Port 13 (GigabitEthernet1/0/13) of VLAN0010 is designated forwarding 
   Port path cost 4, Port priority 128, Port Identifier 128.13.
   Designated root has priority 32778, address 00fe.c8d3.8680
   Designated bridge has priority 32778, address 00fe.c8d3.8680
   Designated port id is 128.13, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 1
   Link type is point-to-point by default
   BPDU: sent 6066, received 3

 Port 14 (GigabitEthernet1/0/14) of VLAN0010 is designated forwarding 
   Port path cost 4, Port priority 128, Port Identifier 128.14.
   Designated root has priority 32778, address 00fe.c8d3.8680
   Designated bridge has priority 32778, address 00fe.c8d3.8680
   Designated port id is 128.14, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 1
   Link type is point-to-point by default
   BPDU: sent 6066, received 3

 Port 15 (GigabitEthernet1/0/15) of VLAN0010 is designated forwarding 
   Port path cost 4, Port priority 128, Port Identifier 128.15.
   Designated root has priority 32778, address 00fe.c8d3.8680
   Designated bridge has priority 32778, address 00fe.c8d3.8680
   Designated port id is 128.15, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 1
   Link type is point-to-point by default
   BPDU: sent 6067, received 0

 Port 16 (GigabitEthernet1/0/16) of VLAN0010 is designated forwarding 
   Port path cost 4, Port priority 128, Port Identifier 128.16.
   Designated root has priority 32778, address 00fe.c8d3.8680
   Designated bridge has priority 32778, address 00fe.c8d3.8680
   Designated port id is 128.16, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 1
   Link type is point-to-point by default
   BPDU: sent 6067, received 0

 Port 35 (GigabitEthernet1/0/35) of VLAN0010 is designated forwarding 
   Port path cost 4, Port priority 128, Port Identifier 128.35.
   Designated root has priority 32778, address 00fe.c8d3.8680
   Designated bridge has priority 32778, address 00fe.c8d3.8680
   Designated port id is 128.35, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 1
   Link type is point-to-point by default
   BPDU: sent 6067, received 0

This output shows that the last topology change occurred from device connected off interface GigabitEthernet1/0/35. Next, issue the same show spanning-tree detail command from this device in order to try to track the issue. If this switch that generates the TCNs is only attached to PC or endpoints, be sure that STP PortFast is enabled on these ports. STP PortFast suppresses STP TCNs when a port transitions states.

Refer to these documents for information about STP and how to troubleshoot link transitions that are associated with network interface cards (NICs):

• Using PortFast and Other Commands to Fix Workstation Startup Connectivity Delays

• Understand Rapid Spanning Tree Protocol (802.1w)

• STP Problems and Related Design Considerations

4. Disconnected Blocked Ports

Because of the load-balancing nature of Fast EtherChannel (FEC) (port-channeling), FEC issues can contribute to both HSRP and STP problems. When you troubleshoot STP or HSRP, you can remove the configuration for any FEC connections. After the configuration changes are in place, issue the show spanning-tree blockedports command on both switches. Ensure that at least one of the ports starts blocking on either side of the connection. 

Refer to these documents for information about Fast EtherChannel:

• Understand EtherChannel Load-Balancing and Redundancy on Catalyst Switches

• Configuring EtherChannels

5. Broadcast Suppression

Enable broadcast suppression in order to help cut down the impact from a broadcast storm. A broadcast storm is one of the main side effects of an STP loop. Here is sample command output:

L2Switch_1#show run interface TenGigabitEthernet1/1/5
Building configuration...

Current configuration : 279 bytes
!
interface TenGigabitEthernet1/1/5
 switchport trunk allowed vlan 300-309
 switchport mode trunk
 storm-control broadcast level 30.00
 storm-control multicast level 30.00
 storm-control unicast level 30.00
 spanning-tree guard root
end

L2Switch_1#show storm-control broadcast 
Key: U - Unicast, B - Broadcast, M - Multicast
Interface  Filter State   Upper        Lower        Current     Action     Type
---------  -------------  -----------  -----------  ----------  ---------  ----
Te1/1/5    Forwarding          30.00%       30.00%       0.00%  None       B   
Te1/1/7    Link Down           30.00%       30.00%       0.00%  None       B   
Te1/1/8    Forwarding          10.00%       10.00%       0.00%  None       B   
    
L2Switch_1#show storm-control multicast 
Key: U - Unicast, B - Broadcast, M - Multicast
Interface  Filter State   Upper        Lower        Current     Action     Type
---------  -------------  -----------  -----------  ----------  ---------  ----
Te1/1/5    Forwarding          30.00%       30.00%       0.00%  None       M   
Te1/1/7    Link Down           30.00%       30.00%       0.00%  None       M   

6. Console and Telnet Access

Console or Telnet traffic to the switch often becomes too sluggish to properly track down an offending device during an STP loop. In order to force the network to recover instantly, remove all redundant physical links. After STP is allowed to reconverge on the new nonredundant topology, reattach one redundant link at a time. If the STP loop returns after you add one particular segment, you have identified the offending devices.

7. Spanning Tree Features: Portfast, UplinkFast, and BackboneFast

Verify that PortFast, UplinkFast, and BackboneFast are configured properly. When you troubleshoot STP issues, disable all advanced STP (UplinkFast and BackboneFast). In addition, verify that STP PortFast is only enabled on ports that are directly connected to nonbridging hosts. Nonbridging hosts include user workstations and routers without bridge groups. Do not enable PortFast on ports that are connected to hubs or other switches. Here are some documents to help understand and configure these features:

Configure Spanning Tree PortFast, BPDU Guard, BPDU Filter, UplinkFast, BackboneFast, and Loop Guard

Understand and Configure the Cisco UplinkFast Feature

8. BPDU Guard

When you enable PortFast BPDU guard, a nontrunking, PortFast-enabled port is moved into an errdisable state at the receipt of a BPDU on that port. This feature helps you find ports that are incorrectly configured for PortFast. The feature also detects where devices reflect packets or inject STP BPDUs into the network. When you troubleshoot STP issues, you can enable this feature to help isolate the STP issue.

L2Switch_1#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
L2Switch_1(config)#spanning-tree portfast bpduguard
L2Switch_1(config)#end

9. VTP Pruning

When VTP Pruning is enabled in the network, it can cause the devices of an HSRP group to go active. This results in IP conflicts among the gateways and cause traffic issues. Make sure the VLAN of any HSRP group is not pruned by VTP in the network.

F. Divide and Conquer

If all other attempts to isolate or resolve HSRP fail, the "divide and conquer" method is the next approach. This method helps isolate the network and components that make up the network. Divide and conquer involves any one of the guidelines in this list:

Note: This list repeats some guidelines from other sections of this document.

• Create a test VLAN for HSRP and isolated VLAN to switch with HSRP routers.

• Disconnect all redundant ports.

• Break FEC ports into single connected ports.

• Reduce HSRP group members to only two members.

• Prune trunk ports such that only necessary VLANs propagate across those ports.

• Disconnect connected switches in the network until the problems cease.

Known Issues

HSRP State Flapping/Unstable When You Use Cisco 2620/2621, Cisco 3600 with Fast Ethernet

This problem can occur with Fast Ethernet interfaces at the disruption of network connectivity or at the addition of an HSRP router with higher priority to a network. When the HSRP state changes from active to speaking, the router resets the interface in order to remove the HSRP MAC address from the interfaces MAC address filter. Only specific hardware that is used on the Fast Ethernet interfaces for Cisco 2600s, 3600s, and 7500s have this issue. The router interface reset causes a link state change on Fast Ethernet interfaces, and the switch detects the change. If the switch runs STP, the change causes an STP transition. The STP takes 30 seconds to transition the port into the forwarding state. This time is twice the default forward delay time of 15 seconds. At the same time, the speaking router transitions to the standby state after 10 seconds, which is the HSRP hold time. STP is not forwarding yet, so no HSRP hello messages are received from the active router. This causes the standby router to become active after about 10 seconds. Both routers are now active. When the STP ports become forwarding, the lower-priority router changes from active to speaking, and the whole process repeats.

Platform	Description	Cisco Bug ID	Fix	Workaround
Cisco 2620/2621	Fast Ethernet interface starts to flap when HSRP is configured and the cable is unplugged.		A software upgrade; refer to the bug for revision details.	Enables spanning tree PortFast on the connected switch port.
Cisco 2620/2621	HSRP state is flapping on 2600 with Fast Ethernet.		Cisco IOS Software Release 12.1.3	Enables spanning tree PortFast on the connected switch port.
Cisco 3600 with NM-1FE-TX1	HSRP state is flapping on 2600 and 3600 Fast Ethernet.		Cisco IOS Software Release 12.1.3	Enables spanning tree PortFast on the connected switch port.
Cisco 4500 with Fast Ethernet interface	HSRP state is flapping on 4500 Fast Ethernet.	Cisco bug ID CSCds16055	Cisco IOS Software Release 12.1.5	Enables spanning tree PortFast on the connected switch port.

¹NM-1FE-TX = one-port Fast Ethernet (10/100BASE-TX interface) network module.

An alternative workaround is to adjust the HSRP timers so that the STP forward delay is less than half of the default HSRP hold time. The default STP forward delay is 15 seconds, and the default HSRP hold time is 10 seconds.

When you use the track command under the HSRP process, Cisco recommends that you use a particular decrement value in order to avoid the HSRP flap.

Here is a sample configuration in an HSRP active router when you use the track command:

standby 1 ip 10.0.0.1
standby 1 priority 105
standby 1 preempt delay minimum 60
standby 1 name TEST
standby 1 track <object> decrement 15

Where 15 is the decrement value when the object flaps. In order to know more about the track command, please navigate to the document Track Option in HSRPv2 Configuration Example.

Related Information

• Campus LAN Catalyst Switches - Access
• LAN Switching
• Technical Support & Documentation - Cisco Systems