Troubleshooting High Availability Issues

This chapter describes how to troubleshoot issues related to high availability (HA).

This chapter includes the following sections:

•Information About Cisco VSG High Availability

•Problems with High Availability

•High Availability Troubleshooting Commands

•Standby Synchronization

Information About Cisco VSG High Availability

Cisco VSG high availability (HA) is a subset of the Cisco NX-OS HA. The following HA features minimize or prevent traffic disruption in the event of a failure:

•Redundancy

•Isolation of Processes

•Cisco VSG Failovers

Redundancy

Cisco VSG redundancy is equivalent to HA pairing. The two possible redundancy states are active and standby. An active Cisco VSG is always paired with a standby Cisco VSG. HA pairing is based on the Cisco VSG ID. Two Cisco VSGs that are assigned an identical ID are automatically paired. All processes running in the Cisco VSG are data path critical. If one process fails in an active Cisco VSG, a failover to the standby Cisco VSG occurs instantly and automatically.

Isolation of Processes

The Cisco VSG software contains independent processes known as services. These services perform a function or set of functions for a subsystem or feature set of the Cisco VSG. Each service and service instance runs as an independent, protected process. This operational process ensures a highly fault-tolerant software infrastructure and fault isolation between services. Any failure in a service instance does not affect any other services running at that time. Additionally, each instance of a service runs as an independent process. For example, two instances of a routing protocol run as separate processes.

Cisco VSG Failovers

The Cisco VSG HA pair configuration allows uninterrupted traffic forwarding by using stateful failovers when a failure occurs.

Problems with High Availability

The following key problems are found in Cisco VSG HA. In addition to these issues, some of the common Cisco NX-OS HA symptoms are listed in Table 7-1. The symptoms that are related to high availability, their possible causes, and recommended solutions are as follows.

•Cisco VSG pair communication problems

•Cisco VSGs do not reach an active/standby status

•Sometimes, the Cisco VSG reboots continuously when tenants share the management network (for example, collisions of the domain ID):

–In a multitenant environment, if there is a shared management network and a collision occurs in the domain ID (two or more tenants using the same domain ID) spontaneous reboots of the Cisco VSGs that are involved in the collision are triggered. There is no workaround for this issue. When domain IDs are provisioned, they must be unique across all tenants

•Cisco VSGs in the HA pair get stuck in bash# prompt mode during reboots/upgrades/switchovers

•Cisco VSGs in the HA pair get stuck in boot# prompt mode during reboots/upgrades/switchovers

Table 7-1 Problems with High Availability  

Symptom
Possible Cause
Solution

The active Cisco VSG does not see the standby Cisco VSG.

Roles are not configured properly:

•primary

•secondary

Do the following:

1. Verify the role of each Cisco VSG by entering the show system redundancy status command.

2. Update an incorrect role by entering the system redundancy role command.

3. Save the configuration by entering the copy run start command.

Network connectivity problems are occurring between the Cisco VSG and the upstream and virtual switches. The problem could be in the control or management VLAN.

Restore connectivity as follows:

1. From the vSphere client, shut down the Cisco VSG, which should be in standby mode.

2. From the vSphere client, bring up the standby Cisco VSG after network connectivity is restored.

The active Cisco VSG does not complete synchronization with the standby Cisco VSG.

A version mismatch between Cisco VSGs might be occurring.

Do the following:

1. Verify the software version on both Cisco VSGs by entering the show version command.

2. Reinstall the secondary Cisco VSG with the same version used in the primary.

Fatal errors occur during the gsync process.

Check the gsyncctrl log by entering the show system internal log sysmgr gsyncctrl command and look for fatal errors.

Reload the standby Cisco VSG by entering the reload module standby_module_number command.

See the "Reloading a Module" section.

The standby Cisco VSG reboots periodically.

The Cisco VSG has connectivity only through the management interface.

When a Cisco VSG is able to communicate through the management interface, but not through the control interface, the active Cisco VSG resets the standby to prevent the two Cisco VSGs from being in HA mode and out of sync.

Check control VLAN connectivity between the primary and secondary Cisco VSG by entering the show system internal redundancy info command.

In the output, degraded_mode flag = true.

If there is no connectivity, restore it through the control interface.

Both Cisco VSGs are in active mode.

The following network connectivity problems might be occurring:

•Check for control and management VLAN connectivity between the Cisco VSG at the upstream and virtual switches.

•When the Cisco VSG cannot communicate through any of these two interfaces, they both try to become active.

If network problems exist, do the following:

1. From the vSphere client, shut down the Cisco VSG, which should be in standby mode.

2. From the vSphere client, bring up the standby Cisco VSG after network connectivity is restored.

Active and standby Cisco VSGs are not synchronized.

Incompatible versions

The boot variables for active and standby Cisco VSGs are set to different image names, or if image names are the same, the files are not the correct files.

When active and standby Cisco VSGs are running different versions that are not HA compatible, they are unable to synchronize.

Update the software version or the boot variables as follows:

1. From each Cisco VSG (active and standby), verify the software version by entering the show version command.

2. Reload the standby Cisco VSG with the version that is running the active Cisco VSG by doing one of the following:

–Correct the boot variable names.

–Replace the incorrect software files.

See the "Reloading a Module" section.

Broadcast traffic problem

The broadcast traffic from the standby to the active Cisco VSG might prevent the Cisco VSGs from synchronizing. The standby Cisco VSG tries to contact the active Cisco VSG periodically, but if broadcast traffic problems persist for over a minute when the standby is booting up, the system cannot synchronize.

Fix the traffic problem and reload the standby Cisco VSG as follows.

1. From the standby Cisco VSG, verify the broadcast traffic problem by entering the show system internal log sysmgr verctrl command.

The following message appears:

standby_verctrl: no response from the active System Manager

2. Fix network connectivity.

3. Reload the standby Cisco VSG by using the reload module standby_module_number command.

See the "Reloading a Module" section.

False standby removal

The active Cisco VSG falsely detects a disconnect with the standby. The standby is removed and reinserted and synchronization does not occur.

Verify redundancy states and reload the standby Cisco VSG as follows:

1. Verify active Cisco VSG redundancy by using the show system internal redundancy status command. The output is as follows: 

RDN_DRV_ST_AC_NP

2. Verify the standby Cisco VSG redundancy by using the show system internal redundancy status command. The output is as follows: 

RDN_DRV_ST_SB_AC

3. Reload the standby Cisco VSG by using the reload module standby_module_number command.

See the "Reloading a Module" section.

The Cisco VSG HA pair reboots continuously in headless mode (VSMs are down).

The nonsystem VLAN Cisco VSG ports are down after they reconnect post reboot of the Cisco VSG because the VSM is not present to bring them up.

Check if the service and HA VLANs are configured as system VLANs. If they are not system VLANs and the Cisco VSG pair reboots for any reason, they do not come back up until the VSM comes up.


High Availability Troubleshooting Commands

This section lists commands that you can use to troubleshoot problems related to high availability.

This section includes the following topics:

•Checking the Domain ID of the Cisco VSG

•Checking Redundancy

•Checking the System Manager State

•Reloading a Module

•Attaching to the Standby Cisco VSG Console

•Checking for the Event History Errors

Checking the Domain ID of the Cisco VSG

You can display the domain ID information by entering the show vsg command.

This example shows how to display the domain ID information: 

vsg# show vsg

Model: VSG

HA ID: 3000

VSG Software Version: 4.2(1)VSG1(1) build [4.2(1)VSG1(1)]

VNMC IP: 10.193.75.145

Checking Redundancy

This section includes the following topics:

•Checking the System Redundancy Status

•Checking the System Internal Redundancy Status

Checking the System Redundancy Status

You can check the system redundancy status by entering the show system redundancy status command.

This example shows how to display the system redundancy status: 

vsg# show system redundancy status 

Redundancy role

---------------

      administrative:   primary <-- Configured redundancy role

         operational:   primary <-- Current operational redundancy role

 
   
 
    
   
 
    
Redundancy mode

 
   
 
    
   
 
    
---------------

 
   
 
    
   
 
    
      administrative:   HA

 
   
 
    
   
 
    
         operational:   HA

 
   
 
    
   
 
    
 
   
 
    
   
 
    
This supervisor (sup-1)

 
   
 
    
   
 
    
-----------------------

 
   
 
    
   
 
    
    Redundancy state:   Active <-- Redundancy state of this VSG

 
   
 
    
   
 
    
    Supervisor state:   Active

 
   
 
    
   
 
    
      Internal state:   Active with HA standby                  

 
   
 
    
   
 
    
 
   
 
    
   
 
    
Other supervisor (sup-2)

 
   
 
    
   
 
    
------------------------

 
   
 
    
   
 
    
    Redundancy state:   Standby <-- Redundancy state of the other VSG

 
   
 
    
   
 
    
    Supervisor state:   HA standby

 
   
 
    
   
 
    
      Internal state:   HA standby <-- The standby VSG is in HA mode and in sync

 
   
 
     
    
    
   
 Checking the System Internal Redundancy Status 
 
    
   
 You can check the system internal redundancy status by entering the show system internal redundancy info command. 
 
    
   
 This example shows how to display the system internal redundancy status information: 
 
    
   
 
    
vsg# show system internal redundancy info 

 
   
 
    
   
 
    
My CP:

 
   
 
    
   
 
    
  slot: 0

 
   
 
    
   
 
    
  domain: 184 <-- Domain id used by this VSG

 
   
 
    
   
 
    
  role:   primary <-- Redundancy role of this VSG

 
   
 
    
   
 
    
  status: RDN_ST_AC <-- Indicates redundancy state (RDN_ST) of the this VSG is Active (AC)

 
   
 
    
   
 
    
  state:  RDN_DRV_ST_AC_SB

 
   
 
    
   
 
    
  intr:   enabled

 
   
 
    
   
 
    
  power_off_reqs: 0

 
   
 
    
   
 
    
  reset_reqs:     0

 
   
 
    
   
 
    
Other CP:

 
   
 
    
   
 
    
  slot: 1

 
   
 
    
   
 
    
  status: RDN_ST_SB <-- Indicates redundancy state (RDN_ST) of the other VSG is Standby 
(SB)

 
   
 
    
   
 
    
  active: true

 
   
 
    
   
 
    
  ver_rcvd: true

 
   
 
    
   
 
    
  degraded_mode: false <-- When true, it indicates that communication through the control 
interface is faulty

 
   
 
    
   
 
    
Redun Device 0: <-- This device maps to the control interface

 
   
 
    
   
 
    
  name: ha0

 
   
 
    
   
 
    
  pdev: ad7b6c60

 
   
 
    
   
 
    
  alarm: false

 
   
 
    
   
 
    
  mac: 00:50:56:b7:4b:59

 
   
 
    
   
 
    
  tx_set_ver_req_pkts:   11590

 
   
 
    
   
 
    
  tx_set_ver_rsp_pkts:   4

 
   
 
    
   
 
    
  tx_heartbeat_req_pkts: 442571

 
   
 
    
   
 
    
  tx_heartbeat_rsp_pkts: 6

 
   
 
    
   
 
    
  rx_set_ver_req_pkts:   4

 
   
 
    
   
 
    
  rx_set_ver_rsp_pkts:   1

 
   
 
    
   
 
    
  rx_heartbeat_req_pkts: 6

 
   
 
    
   
 
    
  rx_heartbeat_rsp_pkts: 442546 <-- Counter should be increasing, as this indicates that 
communication between VSG is working properly.

 
   
 
    
   
 
    
  rx_drops_wrong_domain: 0

 
   
 
    
   
 
    
  rx_drops_wrong_slot:   0

 
   
 
    
   
 
    
  rx_drops_short_pkt:    0

 
   
 
    
   
 
    
  rx_drops_queue_full:   0

 
   
 
    
   
 
    
  rx_drops_inactive_cp:  0

 
   
 
    
   
 
    
  rx_drops_bad_src:      0

 
   
 
    
   
 
    
  rx_drops_not_ready:    0

 
   
 
    
   
 
    
  rx_unknown_pkts:       0

 
   
 
    
   
 
    
Redun Device 1: <-- This device maps to the mgmt interface

 
   
 
    
   
 
    
  name: ha1

 
   
 
    
   
 
    
  pdev: ad7b6860

 
   
 
    
   
 
    
  alarm: true

 
   
 
    
   
 
    
  mac: ff:ff:ff:ff:ff:ff

 
   
 
    
   
 
    
  tx_set_ver_req_pkts:   11589

 
   
 
    
   
 
    
  tx_set_ver_rsp_pkts:   0

 
   
 
    
   
 
    
  tx_heartbeat_req_pkts: 12

 
   
 
    
   
 
    
  tx_heartbeat_rsp_pkts: 0

 
   
 
    
   
 
    
  rx_set_ver_req_pkts:   0

 
   
 
    
   
 
    
  rx_set_ver_rsp_pkts:   0

 
   
 
    
   
 
    
  rx_heartbeat_req_pkts: 0

 
   
 
    
   
 
    
  rx_heartbeat_rsp_pkts: 0 <-- When communication between VSG through the control 
interface is interrupted but continues through the mgmt interface, the 
rx_heartbeat_rsp_pkts will increase.

 
   
 
    
   
 
    
  rx_drops_wrong_domain: 0

 
   
 
    
   
 
    
  rx_drops_wrong_slot:   0

 
   
 
    
   
 
    
  rx_drops_short_pkt:    0

 
   
 
    
   
 
    
  rx_drops_queue_full:   0

 
   
 
    
   
 
    
  rx_drops_inactive_cp:  0

 
   
 
    
   
 
    
  rx_drops_bad_src:      0

 
   
 
    
   
 
    
  rx_drops_not_ready:    0

 
   
 
    
   
 
    
  rx_unknown_pkts:       0

 
   
 
     
    
    
   
 Checking the System Manager State 
 
    
   
 You can check the system internal sysmgr state by entering the show system internal sysmgr state command. 
 
    
   
 This example shows how to display the system internal sysmgr state information: 
 
    
   
 
    
vsg# show system internal sysmgr state 

 
   
 
    
   
 
    
 
   
 
    
   
 
    
The master System Manager has PID 1988 and UUID 0x1.

 
   
 
    
   
 
    
Last time System Manager was gracefully shutdown.

 
   
 
    
   
 
    
The state is SRV_STATE_MASTER_ACTIVE_HOTSTDBY entered at time Tue Apr 28 13:09:13 2009.

 
   
 
    
   
 
    
 
   
 
    
   
 
    
The '-b' option (disable heartbeat) is currently disabled.

 
   
 
    
   
 
    
 
   
 
    
   
 
    
The '-n' (don't use rlimit) option is currently disabled.

 
   
 
    
   
 
    
 
   
 
    
   
 
    
Hap-reset is currently enabled.

 
   
 
    
   
 
    
 
   
 
    
   
 
    
Watchdog checking is currently disabled.

 
   
 
    
   
 
    
 
   
 
    
   
 
    
Watchdog kgdb setting is currently enabled.

 
   
 
    
   
 
    
 
   
 
    
   
 
    
 
   
 
    
   
 
    
        Debugging info:

 
   
 
    
   
 
    
 
   
 
    
   
 
    
The trace mask is 0x00000000, the syslog priority enabled is 3.

 
   
 
    
   
 
    
The '-d' option is currently disabled.

 
   
 
    
   
 
    
The statistics generation is currently enabled.

 
   
 
    
   
 
    
 
   
 
    
   
 
    
 
   
 
    
   
 
    
        HA info:

 
   
 
    
   
 
    
 
   
 
    
   
 
    
slotid = 1    supid = 0

 
   
 
    
   
 
    
cardstate = SYSMGR_CARDSTATE_ACTIVE.

 
   
 
    
   
 
    
cardstate = SYSMGR_CARDSTATE_ACTIVE (hot switchover is configured enabled).

 
   
 
    
   
 
    
Configured to use the real platform manager.

 
   
 
    
   
 
    
Configured to use the real redundancy driver.

 
   
 
    
   
 
    
Redundancy register: this_sup = RDN_ST_AC, other_sup = RDN_ST_SB.

 
   
 
    
   
 
    
EOBC device name: eth0.

 
   
 
    
   
 
    
Remote addresses:  MTS - 0x00000201/3      IP - 127.1.1.2

 
   
 
    
   
 
    
MSYNC done.

 
   
 
    
   
 
    
Remote MSYNC not done.

 
   
 
    
   
 
    
Module online notification received.

 
   
 
    
   
 
    
Local super-state is: SYSMGR_SUPERSTATE_STABLE

 
   
 
    
   
 
    
Standby super-state is: SYSMGR_SUPERSTATE_STABLE

 
   
 
    
   
 
    
Swover Reason : SYSMGR_SUP_REMOVED_SWOVER <-- Reason for the last switchover

 
   
 
    
   
 
    
Total number of Switchovers: 0 <-- Number of switchovers 

 
   
 
    
   
 
    
								>> Duration of the switchover would be listed, if any.

 
   
 
    
   
 
    
 
   
 
    
   
 
    
        Statistics:

 
   
 
    
   
 
    
 
   
 
    
   
 
    
Message count:           0

 
   
 
    
   
 
    
Total latency:           0              Max latency:             0

 
   
 
    
   
 
    
Total exec:              0              Max exec:                0

 
   
 
     
    
    
   
 Reloading a Module 
 
    
   
 You can reload a module by entering the reload module command. 
 
   
 
     
   
 
   
 
    
   
 Note Using the reload command without specifying a module reloads the whole system. 
 
   
 
    
   
 This example shows how to reload a module: 
 
    
   
 
    
vsg# reload module 2

 
   
 
    
   
 
    
This command will reboot the system (y/n)? y

 
   
 
     
    
    
   
 Attaching to the Standby Cisco VSG Console 
 
    
   
 The standby Cisco VSG console is not accessible externally. You can access the standby Cisco VSG console through the active Cisco VSG by entering the attach module module-number command. 
 
    
   
 This example shows how to access the standby Cisco VSG through the active Cisco VSG: 
 
    
   
 
    
vsg# attach module 2

 
   
 
    
   
 
    
Attaching to module 2...

 
   
 
    
   
 
    
To exit type 'exit', to abort type '$.' 

 
   
 
    
   
 
    
Cisco Nexus Operating System (NX-OS) Software

 
   
 
    
   
 
    
TAC support: http://www.cisco.com/tac

 
   
 
    
   
 
    
Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved.

 
   
 
    
   
 
    
The copyrights to certain works contained in this software are

 
   
 
    
   
 
    
owned by other third parties and used and distributed under

 
   
 
    
   
 
    
license. Certain components of this software are licensed under

 
   
 
    
   
 
    
the GNU General Public License (GPL) version 2.0 or the GNU

 
   
 
    
   
 
    
Lesser General Public License (LGPL) Version 2.1. A copy of each

 
   
 
    
   
 
    
such license is available at

 
   
 
    
   
 
    
http://www.opensource.org/licenses/gpl-2.0.php and

 
   
 
    
   
 
    
http://www.opensource.org/licenses/lgpl-2.1.php

 
   
 
     
    
    
    
   
 Checking for the Event History Errors 
 
    
   
 You can check for errors in the event history by entering the show system internal sysmgr event-history errors command. 
 
    
   
 This example shows how to display errors that have been logged in the event history: 
 
    
   
 
    
vsg# show system internal sysmgr event-history errors 
Event:E_DEBUG, length:122, at 370850 usecs after Thu Feb  3 09:45:28 2011

 
   
 
    
   
 
    
[101] sysmgr_sdb_set_standby_state: Setting standby super state in sdb for vdc 1 to 
SYSMGR_SUPERSTATE_STABLE, returned

 
   
 
    
   
 
    
0x0

 
   
 
    
   
 
    
 
   
 
    
   
 
    
Event:E_DEBUG, length:73, at 408277 usecs after Thu Feb  3 09:44:52 2011

 
   
 
    
   
 
    
[101] active_gsyncctrl_info_parse: UUID 0xB6 in vdc 1 service not running

 
   
 
    
   
 
    
 
   
 
    
   
 
    
Event:E_DEBUG, length:73, at 593428 usecs after Thu Feb  3 09:44:49 2011

 
   
 
    
   
 
    
[101] active_gsyncctrl_info_parse: UUID 0xE0 in vdc 1 service not running

 
   
 
    
   
 
    
 
   
 
    
   
 
    
Event:E_DEBUG, length:80, at 624613 usecs after Thu Feb  3 09:44:40 2011

 
   
 
    
   
 
    
[101] process_plugin_load_complete_msg: Start done rcvd for all plugins in vdc 1

 
   
 
    
   
 
    
 
   
 
    
   
 
    
Event:E_DEBUG, length:89, at 624611 usecs after Thu Feb  3 09:44:40 2011

 
   
 
    
   
 
    
[101] process_plugin_load_complete_msg: Received plugin start done for plugin 1 for vdc 1

 
   
 
    
   
 
    
 
   
 
    
   
 
    
Event:E_DEBUG, length:99, at 518152 usecs after Thu Feb  3 09:44:39 2011

 
   
 
    
   
 
    
[101] perform_bootup_plugin_manager_interactions: all bootup plugins have now been loaded 
in vdc: 1

 
   
 
    
   
 
    
 
   
 
    
   
 
    
Event:E_DEBUG, length:79, at 518150 usecs after Thu Feb  3 09:44:39 2011

 
   
 
    
   
 
    
[101] perform_bootup_plugin_manager_interactions:incrementing number of plugins

 
   
 
    
   
 
    
 
   
 
    
   
 
    
Event:E_DEBUG, length:118, at 518020 usecs after Thu Feb  3 09:44:39 2011

 
   
 
    
   
 
    
[101] perform_bootup_plugin_manager_interactions: plugin has been loaded in vdc 1 - 
sending response to Plugin Manager

 
   
 
    
   
 
    
 
   
 
    
   
 
    
Event:E_DEBUG, length:58, at 631599 usecs after Thu Feb  3 09:44:38 2011

 
   
 
    
   
 
    
[101] process_reparse_request: on vdc 1, plugin start rcvd

 
   
 
     
    
    
    
   
 Standby Synchronization 
 
    
   
 This section includes the following topic: 
 
    
   
 •Synchronization Fails 
 
     
    
    
   
 Synchronization Fails 
 
    
   
 If the standby Cisco VSG is stuck in the synchronization stage, follow these steps on the active Cisco VSG: 
 
   
 
    
   
 Step 1 Enter the show system internal sysmgr state command and check for a line similar to the following: 
 
    
   
 
    
Gsync in progress for uuid: xxxx 


 
   
 
    
   
 If this entry is present and shows the same xxxx value for a long time, the system has trouble synchronizing the state for one of the processes. 
 
    
   
 Step 2 Identify the process by entering the show system internal sysmgr service running | grep xxxx command. 
 
    
   
 This message appears in the first few lines of the output: 
 
    
   
 
    
 
BL-bash# show system internal sysmgr state

 
   
 
    
   
 
    
The master System Manager has PID 1350 and UUID 0x1. 
Last time System Manager was gracefully shutdown. 
Gsync in progress for uuid: 0x18  
The state is SRV_STATE_MASTER_ACTIVE_HOTSTDBY entered at time Mon Feb 21 17:56:3 
9 2011. 
 
The '-b' option (disable heartbeat) is currently disabled. 
... 
 
If the synchronization for each process occurs quickly, you might not have the chance to 
see the line (you might have to enter the command repeatedly as the standby Cisco VSG). If 
gsync for a particular process gets stuck, the line stays in the output for a while. 

 
   
 
    
   
 
    
 
   
 
    
   
 Step 3 If you are able to login to the console of the standby Cisco VSG (you might need to press Ctrl-C after giving the password), check the output of these two commands: 
 
    
   
 •show system internal sysmgr state 
 
    
   
 •show system internal sysmgr service running | grep xxxx
where xxxx is from the line "Gsync in progress for uuid: xxxx" (found by using the show system internal sysmgr state command) 
 
    
   
 Step 4 If access to the system is available only after the standby server has booted up and synchronized with the active server, use the following commands: 
 
    
   
 •Enter the show system internal sysmgr bootupstats command and look for processes that took much longer than the rest, in the time that the system took to boot. 
 
    
   
 •On the standby console, enter the show system internal sysmgr gsyncstats command and look for processes with large Gsync time values.