Troubleshooting High Availability Issues

This chapter describes how to troubleshoot issues related to high availability (HA).

This chapter includes the following sections:

Information About Cisco VSG High Availability

Cisco VSG high availability (HA) is a subset of the Cisco NX-OS HA. The following HA features minimize or prevent traffic disruption in the event of a failure:

Redundancy

Cisco VSG redundancy is equivalent to HA pairing. The two possible redundancy states are active and standby. An active Cisco VSG is always paired with a standby Cisco VSG. HA pairing is based on the Cisco VSG HA-ID. Two Cisco VSGs that are assigned an identical ID are automatically paired. All processes running in the Cisco VSG are data path critical. If one process fails in an active Cisco VSG, a failover to the standby Cisco VSG occurs instantly and automatically.

Isolation of Processes

The Cisco VSG software contains independent processes known as services. These services perform a function or set of functions for a subsystem or feature set of the Cisco VSG. Each service and service instance runs as an independent, protected process. This operational process ensures a highly fault-tolerant software infrastructure and fault isolation between services. Any failure in a service instance does not affect any other services running at that time. Additionally, each instance of a service runs as an independent process. For example, two instances of a routing protocol run as separate processes.

Cisco VSG Failovers

The Cisco VSG HA pair configuration allows uninterrupted traffic forwarding by using stateful failovers when a failure occurs.

Problems with High Availability

The following key problems are found in Cisco VSG HA. In addition to these issues, some of the common Cisco NX-OS HA symptoms are listed in Table 7-1. The symptoms that are related to high availability, their possible causes, and recommended solutions are as follows.

  • Cisco VSG pair communication problems.
  • Cisco VSGs do not reach an active/standby status.
  • Sometimes, Cisco VSG reboots continuously when tenants share the management network (for example, collisions of the domain ID).

– In a multitenant environment, if there is a shared management network and a collision occurs in the domain ID (two or more tenants using the same domain ID) spontaneous reboots of the Cisco VSGs that are involved in the collision are triggered. There is no workaround for this issue. When domain IDs are provisioned, they must be unique across all tenants.

  • Cisco VSGs in the HA pair get stuck in bash# prompt mode during reboots/upgrades/switchovers.

 

Table 7-1 Problems with High Availability

Symptom
Possible Cause
Solution

The active Cisco VSG does not see the standby Cisco VSG.

Roles are not configured properly:

  • primary
  • secondary

Do the following:

1. Verify the role of each Cisco VSG by entering the show system redundancy status command.

2. Update an incorrect role by entering the system redundancy role command.

3. Save the configuration by entering the copy run start command.

Network connectivity problems are occurring between the Cisco VSG and the upstream and virtual switches. The problem could be in the control or management VLAN.

Restore connectivity as follows:

1. From the Hyper-V client, shut down Cisco VSG, which should be in standby mode.

2. From the Hyper-V client, power on the standby Cisco VSG after network connectivity is restored.

The active Cisco VSG does not complete synchronization with the standby Cisco VSG.

A version mismatch between Cisco VSGs might be occurring.

Do the following:

1. Verify the software version on both Cisco VSGs by entering the show version command.

2. Reinstall the secondary Cisco VSG with the same version used in the primary.

Fatal errors occur during the gsync process.

Check the gsyncctrl log by entering the show system internal log sysmgr gsyncctrl command and look for fatal errors.

Reload the standby Cisco VSG by entering the reload module standby_module_number command.

See the “Reloading a Module” section.

The standby Cisco VSG reboots periodically.

The Cisco VSG has connectivity only through the management interface.

When a Cisco VSG is able to communicate through the management interface, but not through the control interface, the active Cisco VSG resets the standby to prevent the two Cisco VSGs from being in HA mode and out of sync.

Check control VLAN connectivity between the primary and secondary Cisco VSG by entering the show system internal redundancy info command.

In the output, degraded_mode flag = true.

If there is no connectivity, restore it through the control interface.

Both Cisco VSGs are in active mode.

The following network connectivity problems might be occurring:

  • Check for control and management VLAN connectivity between the Cisco VSG at the upstream and virtual switches.
  • When the Cisco VSG cannot communicate through any of these two interfaces, they both try to become active.

If network problems exist, do the following:

1. From the Hyper-V client, shut down the Cisco VSG, which should be in standby mode.

2. From the Hyper-V client, bring up the standby Cisco VSG after network connectivity is restored.

Active and standby Cisco VSGs are not synchronized.

Incompatible versions

The boot variables for active and standby Cisco VSGs are set to different image names, or if image names are the same, the files are not the correct files.

When active and standby Cisco VSG devices are running different versions that are not HA compatible, they are unable to synchronize.

Update the software version or the boot variables as follows:

1. From each Cisco VSG (active and standby), verify the software version by entering the show version command.

2. Reload the standby Cisco VSG with the version that is running the active Cisco VSG by doing one of the following:

– Correct the boot variable names.

– Replace the incorrect software files.

See the “Reloading a Module” section.

Broadcast traffic problem

The broadcast traffic from the standby to the active Cisco VSG might prevent the Cisco VSGs from synchronizing. The standby Cisco VSG tries to contact the active Cisco VSG periodically, but if broadcast traffic problems persist for over a minute when the standby device is booting up, the system cannot synchronize.

Fix the traffic problem and reload the standby Cisco VSG as follows.

1. From the standby Cisco VSG, verify the broadcast traffic problem by entering the show system internal log sysmgr verctrl command.

The following message appears:

standby_verctrl: no response from the active System Manager

2. Fix network connectivity.

3. Reload the standby Cisco VSG by using the reload module standby_module_number command.

See the “Reloading a Module” section.

False standby removal

The active Cisco VSG falsely detects a disconnect with the standby device, which is removed and reinserted and synchronization does not occur.

Verify redundancy states and reload the standby Cisco VSG as follows:

1. Verify active Cisco VSG redundancy by using the show system internal redundancy status command. The output is as follows:

RDN_DRV_ST_AC_NP

2. Verify the standby Cisco VSG redundancy by using the show system internal redundancy status command. The output is as follows:

RDN_DRV_ST_SB_AC

3. Reload the standby Cisco VSG by using the reload module standby_module_number command.

See the “Reloading a Module” section.

The Cisco VSG HA pair reboots continuously in headless mode (VSMs are down).

The nonsystem VLAN Cisco VSG ports are down after they reconnect post reboot of the Cisco VSG because the VSM is not present to bring them up.

 

Check if the service and HA VLANs are configured as system VLANs. If they are not system VLANs and the Cisco VSG pair reboots for any reason, they do not come back up until the VSM comes up.

High-Availability Troubleshooting Commands

This section lists commands that you can use to troubleshoot problems related to high availability.

This section includes the following topics:

Checking the HA-ID of the Cisco VSG

You can display the HA-ID information by entering the show vsg command.

This example shows how to display the HA-ID information:

VSG_Fire(config)# show vsg
Model: VSG
HA ID: 142
VSG software version: 5.2(1)VSG2(1.2b) build [5.2(1)VSG2(1.2b)]
PNSC IP: 10.2.71.139
NSC PA version: 2.1(1e)-vsg

Checking Redundancy

This section includes the following topics:

Checking the System Redundancy Status

You can check the system redundancy status by entering the show system redundancy status command.

This example shows how to display the system redundancy status:

vsg# show system redundancy status
Redundancy role
---------------
administrative: primary <-- Configured redundancy role
operational: primary <-- Current operational redundancy role
 
Redundancy mode
---------------
administrative: HA
operational: HA
 
This supervisor (sup-1)
-----------------------
Redundancy state: Active <-- Redundancy state of this VSG
Supervisor state: Active
Internal state: Active with HA standby
 
Other supervisor (sup-2)
------------------------
Redundancy state: Standby <-- Redundancy state of the other VSG
Supervisor state: HA standby
Internal state: HA standby <-- The standby VSG is in HA mode and in sync

Checking the System Internal Redundancy Status

You can check the system internal redundancy status by entering the show system internal redundancy info command.

This example shows how to display the system internal redundancy status information:

vsg# show system internal redundancy info
My CP:
slot: 0
domain: 184 <-- Domain id used by this VSG
role: primary <-- Redundancy role of this VSG
status: RDN_ST_AC <-- Indicates redundancy state (RDN_ST) of the this VSG is Active (AC)
state: RDN_DRV_ST_AC_SB
intr: enabled
power_off_reqs: 0
reset_reqs: 0
Other CP:
slot: 1
status: RDN_ST_SB <-- Indicates redundancy state (RDN_ST) of the other VSG is Standby (SB)
active: true
ver_rcvd: true
degraded_mode: false <-- When true, it indicates that communication through the control interface is faulty
Redun Device 0: <-- This device maps to the control interface
name: ha0
pdev: ad7b6c60
alarm: false
mac: 00:50:56:b7:4b:59
tx_set_ver_req_pkts: 11590
tx_set_ver_rsp_pkts: 4
tx_heartbeat_req_pkts: 442571
tx_heartbeat_rsp_pkts: 6
rx_set_ver_req_pkts: 4
rx_set_ver_rsp_pkts: 1
rx_heartbeat_req_pkts: 6
rx_heartbeat_rsp_pkts: 442546 <-- Counter should be increasing, as this indicates that communication between VSG is working properly.
rx_drops_wrong_domain: 0
rx_drops_wrong_slot: 0
rx_drops_short_pkt: 0
rx_drops_queue_full: 0
rx_drops_inactive_cp: 0
rx_drops_bad_src: 0
rx_drops_not_ready: 0
rx_unknown_pkts: 0
Redun Device 1: <-- This device maps to the mgmt interface
name: ha1
pdev: ad7b6860
alarm: true
mac: ff:ff:ff:ff:ff:ff
tx_set_ver_req_pkts: 11589
tx_set_ver_rsp_pkts: 0
tx_heartbeat_req_pkts: 12
tx_heartbeat_rsp_pkts: 0
rx_set_ver_req_pkts: 0
rx_set_ver_rsp_pkts: 0
rx_heartbeat_req_pkts: 0
rx_heartbeat_rsp_pkts: 0 <-- When communication between VSG through the control interface is interrupted but continues through the mgmt interface, the rx_heartbeat_rsp_pkts will increase.
rx_drops_wrong_domain: 0
rx_drops_wrong_slot: 0
rx_drops_short_pkt: 0
rx_drops_queue_full: 0
rx_drops_inactive_cp: 0
rx_drops_bad_src: 0
rx_drops_not_ready: 0
rx_unknown_pkts: 0

Checking the System Manager State

You can check the internal system manager state by entering the show system internal sysmgr state command.

This example shows how to display the internal system manager state information:

vsg# show system internal sysmgr state
 
The master System Manager has PID 1988 and UUID 0x1.
Last time System Manager was gracefully shutdown.
The state is SRV_STATE_MASTER_ACTIVE_HOTSTDBY entered at time Tue Apr 28 13:09:13 2009.
 
The '-b' option (disable heartbeat) is currently disabled.
 
The '-n' (don't use rlimit) option is currently disabled.
 
Hap-reset is currently enabled.
 
Watchdog checking is currently disabled.
 
Watchdog kgdb setting is currently enabled.
 
 
Debugging info:
 
The trace mask is 0x00000000, the syslog priority enabled is 3.
The '-d' option is currently disabled.
The statistics generation is currently enabled.
 
 
HA info:
 
slotid = 1 supid = 0
cardstate = SYSMGR_CARDSTATE_ACTIVE.
cardstate = SYSMGR_CARDSTATE_ACTIVE (hot switchover is configured enabled).
Configured to use the real platform manager.
Configured to use the real redundancy driver.
Redundancy register: this_sup = RDN_ST_AC, other_sup = RDN_ST_SB.
EOBC device name: eth0.
Remote addresses: MTS - 0x00000201/3 IP - 127.1.1.2
MSYNC done.
Remote MSYNC not done.
Module online notification received.
Local super-state is: SYSMGR_SUPERSTATE_STABLE
Standby super-state is: SYSMGR_SUPERSTATE_STABLE
Swover Reason : SYSMGR_SUP_REMOVED_SWOVER <-- Reason for the last switchover
Total number of Switchovers: 0 <-- Number of switchovers
>> Duration of the switchover would be listed, if any.
 
Statistics:
 
Message count: 0
Total latency: 0 Max latency: 0
Total exec: 0 Max exec: 0

Reloading a Module

You can reload a module by entering the reload module command.

Note Using the reload command without specifying a module reloads the whole system.

This example shows how to reload a module:

vsg# reload module 2
This command will reboot standby supervisor module. (y/n)? [n]

Attaching to the Standby Cisco VSG Console

The standby Cisco VSG console is not accessible externally. You can access the standby Cisco VSG console through the active Cisco VSG console by entering the attach module module-number command.

This example shows how to access the standby Cisco VSG console through the active Cisco VSG console:

vsg# attach module 2
Attaching to module 2...
To exit type 'exit', to abort type '$.'
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

Checking for the Event History Errors

You can check for errors in the event history by entering the show system internal sysmgr event-history errors command.

This example shows how to display errors that have been logged in the event history:

vsg# show system internal sysmgr event-history errors
Event:E_DEBUG, length:122, at 370850 usecs after Thu Feb 3 09:45:28 2011
[101] sysmgr_sdb_set_standby_state: Setting standby super state in sdb for vdc 1 to SYSMGR_SUPERSTATE_STABLE, returned
0x0
 
Event:E_DEBUG, length:73, at 408277 usecs after Thu Feb 3 09:44:52 2011
[101] active_gsyncctrl_info_parse: UUID 0xB6 in vdc 1 service not running
 
Event:E_DEBUG, length:73, at 593428 usecs after Thu Feb 3 09:44:49 2011
[101] active_gsyncctrl_info_parse: UUID 0xE0 in vdc 1 service not running
 
Event:E_DEBUG, length:80, at 624613 usecs after Thu Feb 3 09:44:40 2011
[101] process_plugin_load_complete_msg: Start done rcvd for all plugins in vdc 1
 
Event:E_DEBUG, length:89, at 624611 usecs after Thu Feb 3 09:44:40 2011
[101] process_plugin_load_complete_msg: Received plugin start done for plugin 1 for vdc 1
 
Event:E_DEBUG, length:99, at 518152 usecs after Thu Feb 3 09:44:39 2011
[101] perform_bootup_plugin_manager_interactions: all bootup plugins have now been loaded in vdc: 1
 
Event:E_DEBUG, length:79, at 518150 usecs after Thu Feb 3 09:44:39 2011
[101] perform_bootup_plugin_manager_interactions:incrementing number of plugins
 
Event:E_DEBUG, length:118, at 518020 usecs after Thu Feb 3 09:44:39 2011
[101] perform_bootup_plugin_manager_interactions: plugin has been loaded in vdc 1 - sending response to Plugin Manager
 
Event:E_DEBUG, length:58, at 631599 usecs after Thu Feb 3 09:44:38 2011
[101] process_reparse_request: on vdc 1, plugin start rcvd

Standby Synchronization

This section includes the following topic:

Synchronization Fails

If the standby Cisco VSG is stuck in the synchronization stage, follow these steps on the active Cisco VSG:

Step 1 Enter the show system internal sysmgr state command and check for a line similar to the following:

Gsync in progress for uuid: xxxx

If this entry is present and shows the same xxxx value for a long time, the system has trouble synchronizing the state for one of the processes.

Step 2 Identify the process by entering the show system internal sysmgr service running | grep xxxx command.

This message appears in the first few lines of the output:


BL-bash# show system internal sysmgr state
The master System Manager has PID 1350 and UUID 0x1.
Last time System Manager was gracefully shutdown.
Gsync in progress for uuid: 0x18
The state is SRV_STATE_MASTER_ACTIVE_HOTSTDBY entered at time Mon Feb 21 17:56:3
9 2013.

The '-b' option (disable heartbeat) is currently disabled.
…

If the synchronization for each process occurs quickly, you might not have the chance to see the line (you might have to enter the command repeatedly as the standby Cisco VSG). If gsync for a particular process gets stuck, the line stays in the output for a while.
 

Step 3 If you are able to login to the standby Cisco VSG console (you might need to press Ctrl-C after giving the password), check the output of these two commands:

  • show system internal sysmgr state
  • show system internal sysmgr service running | grep xxxx
    where xxxx is from the line “Gsync in progress for uuid: xxxx” (found by using the show system internal sysmgr state command)

Step 4 If access to the system is available only after the standby server has booted up and synchronized with the active server, use the following commands:

  • Enter the show system internal sysmgr bootupstats command and look for processes that took much longer than the rest, in the time that the system took to boot.
  • On the standby console, enter the show system internal sysmgr gsyncstats command and look for processes with large Gsync time values.