Troubleshoot SVL on Catalyst 9000 Switches
Available Languages
Contents
Introduction
This document describes how to identify, collect useful logs, and solve problems that occur with StackWise-virtual (SVL) on Catalyst 9000 switches.
Prerequisites
Requirements
There are no specific requirements for this document.
Components Used
The information in this document is based on these software and hardware versions:
- StackWise-virtual (SVL)
- Catalyst 9000 Switches
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Background Information
This document also provides support matrix, restrictions, commands, and common issues seen with SVL.
When you troubleshoot SVL, there are basic criteria that you need to understand and adhere to when you deploy SVL with the Cisco Catalyst 9000 switches. These are criteria that need to be met:
- Ensure that SVL is supported for a given switch, platform, and software release.
- Ensure that SVL is configured as per the guidelines provided in the configuration guide and strictly keeps to the provided restrictions.
- Ensure that SVL links are physically connected between the switches.
Platform Support
Currently supported platforms listed together with software train that introduced support.
| Software Train | SVL Support Introduced On | Comments |
|
16.3 |
C3850-48XS |
|
|
16.6 |
C9500-24Q |
|
|
16.8 |
C9500-12Q, C9500-24Q, C9500-16X, C9500-40X C3850-12XS, C3850-24XS, C3850-48XS |
SVL/DAD link not supported on uplink module on C9500-16X, C9500-40X, C3850-12XS, C3850-24XS. |
|
16.9 |
C9404R, C9407R |
- SVL/DAD link supported on supervisor ports only- SVL supported only with SUP-1 or SUP-1XL. |
|
16.10 |
C9500-32C, C9500-32QC, C9500-24Y4C, C9500-48Y4C |
StackWise virtual support first introduced on C9500 High-performance models. |
|
16.11 |
C9500-NM-2Q, C9500-NM-8X, C9500-16X, C9500-40X C9410R, C9400-SUP-1XL-Y |
- SVL/DAD link now supported on uplink modules on C9500-16X, C9500-40X. -SVL supported on all C9400 chassis with SUP-1 or SUP-1XL or SUP-1XL-Y. - SUP XL-25G SVL SUPPORT |
|
16.12 |
C9606R |
- StackWise virtual support first introduced on C9600 chassis with C9600-LC-48YL and C9600-LC-24C. - SVL FIP support - UPLINKS FIPS SUPPORT ON SVL - Secure SVL support. |
|
17.1 |
C9606R |
New LC C9600-LC-48TX support on C9600 HA & SVL |
|
17.2 |
C9606R |
- Quad-Supervisor with Route Processor Redundancy (RPR) support first introduced on C9606R chassis. - Support for DAD links on C9600-LC-48S . |
Restrictions
Most of the restrictions can be found in the Cisco StackWise-virtual Configuration guide. Next, are some of the other restrictions which are more platform-specific and can not be explicitly called out in the configuration guide.
Common Restriction
Dual-Active Detection (DAD) and SVL configuration must be performed manually and the devices must be rebooted for configuration changes to take effect.
Restrictions on C9400 SVL
- SVL connections can be established through 10G, 40G or 25G (available only on C9400-SUP-1XL-Y) uplink ports on the supervisor modules and 10G downlink ports on the line cards.
- SVL configuration on 1G interfaces is not supported.
- SVL and DAD links in 16.9.1 are supported on supervisor ports. DAD ePAgP can be configured on line card, as well as supervisor ports.
- SVL and DAD on certain line card are controlled availability in 16.11.1.
For a complete list of restrictions on C9400, refer to the High Availability Configuration Guide, Catalyst 9400 Switches
Restrictions on C9500 SVL
- When SVL is configured on Cisco Catalyst 9500 Series Switches with C9500-NM-2Q (2x40G), you cannot use a combination of fixed downlink and modular uplink ports. SVLs must have the same speed on each member.
- The 40G ports on a C9500-NM-2Q cannot be combined with the downlink ports on a switch as they have different speeds.
- In a Cisco StackWise Virtual solution, interfaces that support 4X10G breakout cables and QSA can be used for data/control traffic but cannot be used to configure SVLs or DAD links.
Restrictions on C9500H SVL
- On C9500-32C switches, you can configure SVL and DAD only on interfaces numbered 1-16 on the front panel of the switch.
- On C9500-32QC, you can configure SVL and DAD only on native 100G and 40G interfaces (default configuration ports). You cannot configure SVL and DAD on converted 100G and 40G interfaces.
- SVL/DAD links cannot be configured on the 4X10G and 4X25G breakout interfaces on C9500-32C SVL. However, breakout interfaces can be used for data/control traffic when C9500 Series High-Performance Switches are configured in StackWise Virtual mode.
- On Cisco Catalyst C9500 Series High-Performance Switches, SVL link configurations on 1G interfaces are not supported.
- 9500H does not support the SVL feature on any 16.9 release. (Cisco removed this configuration option via Cisco bug ID CSCvt46115 in 16.9.6). Verify your 9500H is running 16.10 or later.
Restrictions on C9600 SVL
- On Cisco Catalyst C9600R Switches, SVL/DAD links cannot be configured on the 4X10 and 4X25G breakout interfaces. However, breakout interfaces can be used for data traffic when C9600R Switches are configured in StackWise Virtual mode.
- On Cisco Catalyst C9600R Switches, SVL link configurations on 1G interfaces are not supported.
Troubleshoot
Check the StackWise-virtual Configuration
Step 1. Use show running configuration to ensure StackWise-virtual configuration exists and correct in running-config.
Switch#show running-config
Building configuration...
Current configuration : 9047 bytes
!
! Last configuration change at 09:36:41 UTC Fri Nov 13 2020
!
version 16.11
[....]
stackwise-virtual
domain 1
!
[.....]
!
license boot level network-advantage addon dna-advantage
!
[....]
interface GigabitEthernet1/1/0/43
stackwise-virtual dual-active-detection
!
interface GigabitEthernet1/1/0/44
!
interface TenGigabitEthernet1/3/0/1
stackwise-virtual link 1
!
interface TenGigabitEthernet1/3/0/2
stackwise-virtual link 1
!
interface TenGigabitEthernet1/3/0/3
stackwise-virtual link 1
!
interface TenGigabitEthernet1/3/0/4
stackwise-virtual link 1
!
interface TenGigabitEthernet1/3/0/5
!
interface TenGigabitEthernet1/3/0/6
[....]
interface TenGigabitEthernet2/3/0/1
stackwise-virtual link 1
!
interface TenGigabitEthernet2/3/0/2
stackwise-virtual link 1
!
interface TenGigabitEthernet2/3/0/3
stackwise-virtual link 1
!
interface TenGigabitEthernet2/3/0/4
stackwise-virtual link 1
!
interface TenGigabitEthernet2/3/0/5
!
interface TenGigabitEthernet2/3/0/6
!
interface GigabitEthernet2/5/0/43
stackwise-virtual dual-active-detection
!
interface GigabitEthernet2/5/0/44
!
Step 2. Show romvar can be used to verify that the rommon variables display the correct values which correspond to the configured StackWise-virtual configuration.
Switch#show romvar
Switch 1
ROMMON variables:
SWITCH_NUMBER="1"
MODEL_NUM="C9400-SUP-1XL"
LICENSE_BOOT_LEVEL="network-advantage+dna-advantage,all:MACALLAN-CHASSIS;"
D_STACK_DISTR_STACK_LINK2=""
D_STACK_MODE="aggregation"
D_STACK_DOMAIN_NUM="1"
D_STACK_DISTR_STACK_LINK1="Te1/3/0/1,Te1/3/0/2,Te1/3/0/3,Te1/3/0/4,"
D_STACK_DAD="Gi1/1/0/43,"
Switch 2
ROMMON variables:
LICENSE_BOOT_LEVEL="network-advantage+dna-advantage,all:MACALLAN-CHASSIS;"
D_STACK_DISTR_STACK_LINK2=""
SWITCH_NUMBER="2"
D_STACK_MODE="aggregation"
D_STACK_DOMAIN_NUM="1"
D_STACK_DISTR_STACK_LINK1="Te2/3/0/1,Te2/3/0/2,Te2/3/0/3,Te2/3/0/4,"
D_STACK_DAD="Gi2/5/0/43,"
D_STACK_DISTR_STACK_LINK1 represents the rommon variable for SVL links from both switch1 & switch2.
D_STACK_DAD represents the rommon variable for DAD links.
D_STACK_DOMAIN_NUM represents the SVL domain number, ensure they are the same on both switch1 & switch2.
D_STACK_MODE represents that Cat9k switch is configured in SVL/distributed stack mode.
SVL Domain number, as well as SVL and DAD link configuration, is not only stored as part of running-config/startup-config but also as rommon variables.
The rommon variables and the related values of these rommon variables can be verified. Use CLI as shown previously.
The next output displays the rommon variable created and updated from a C9500H/C9500 switch configured in SVL mode where the interface name tracks 3 tuple model when configured in SVL mode.
Switch#show romvar | include D_STACK D_STACK_DISTR_STACK_LINK2="" D_STACK_MODE="aggregation" D_STACK_DOMAIN_NUM="100" D_STACK_DISTR_STACK_LINK1="Te1/0/3,Te1/0/5," D_STACK_DAD="Te1/0/4,"
D_STACK_DISTR_STACK_LINK2=""
D_STACK_MODE="aggregation"
D_STACK_DOMAIN_NUM="100"
D_STACK_DISTR_STACK_LINK1="Te1/0/3,Te1/0/5,"
D_STACK_DAD="Te1/0/4,"
Check Switch Status
The switch status of both the chassis is expected to be in Ready state. Also, check the show module command output to ensure all LCs are in OK state.
Switch#show switch
Switch/Stack Mac Address : 00a7.42d7.4620 - Local Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State
-------------------------------------------------------------------------------------
*1 Active 00a7.42d7.3680 1 V02 Ready
2 Standby 00a7.42d7.4620 1 V02 Ready
Switch States
| SVL Switch States | Description |
|
Ready |
Stack/switch is ready for use. |
|
mismatch |
Image version mismatched with the peer switch. |
|
|
The system is not ready for use, still under initialize phase. |
|
HA sync in progress |
The system is not ready for use, it waits for config sync to complete. |
The next is the show module output from a 9400 switch.
Switch#show module
Chassis Type: C9407R
Switch Number 1
Mod Ports Card Type Model Serial No.
---+-----+--------------------------------------+--------------+--------------
1 48 48-Port UPOE 10/100/1000 (RJ-45) WS-XL48U JAE201900TY
3 10 Supervisor 1 XL Module C9400-SUP-1XL JAE222805NB
7 24 24-Port 10 Gigabit Ethernet (SFP+) C9400-LC-24XS JAE22170EAG
Mod MAC addresses Hw Fw Sw Status
---+--------------------------------+----+------------+------------------+--------
1 0035.1A8D.2DEC to 0035.1A8D.2E1B 0.4 17.1.1r 16.11.01c ok
3 2C5A.0F1C.626C to 2C5A.0F1C.6275 2.0 17.1.1r 16.11.01c ok
7 780C.F02B.4D50 to 780C.F02B.4D67 1.0 17.1.1r 16.11.01c ok
Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode
---+-------------------+-------------------------+---------------------------
3 Active sso sso
Switch Number 2
Mod Ports Card Type Model Serial No.
---+-----+--------------------------------------+--------------+--------------
3 10 Supervisor 1 XL Module C9400-SUP-1XL JAE222805QB
5 48 48-Port 10/100/1000 (RJ-45) C9400-LC-48T JAE213207ZX
Mod MAC addresses Hw Fw Sw Status
---+--------------------------------+----+------------+------------------+--------
3 2C5A.0F1C.622C to 2C5A.0F1C.6235 2.0 17.1.1r 16.11.01c ok
5 E4AA.5D59.FB48 to E4AA.5D59.FB77 1.0 17.1.1r 16.11.01c ok
Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode
---+-------------------+-------------------------+---------------------------
3 Standby sso sso
Chassis MAC address range: 44 addresses from 2c5a.0f1c.6240 to 2c5a.0f1c.626b
Switch#
Check SVL Link Status
The Link-Status of SVL ports are expected to be in U (Up) state and the Protocol-Status has to be R (Ready)
Switch#show stackwise-virtual link
Stackwise Virtual Link(SVL) Information:
----------------------------------------
Flags:
------
Link Status
-----------
U-Up D-Down
Protocol Status
---------------
S-Suspended P-Pending E-Error T-Timeout R-Ready
-----------------------------------------------
Switch SVL Ports Link-Status Protocol-Status
------ --- ----- ----------- ---------------
1 1 FortyGigabitEthernet1/0/1 U R
FortyGigabitEthernet1/0/2 U R
2 1 FortyGigabitEthernet2/0/1 U R
FortyGigabitEthernet2/0/2 U R
| SVL Protocol Status | Description |
|
Suspended |
Protocol is in a suspended state, could be seen when SVL link is down. Check link status and ensure link is up. |
|
Pending |
Protocol is pending, could be seen when link is not bundled yet. Check the remote side of the link, if both sides are in pend check LMP health. |
|
Error |
Protocol is in error state, could be seen when LMP packets are exchanged with value errors. |
|
Timeout |
Protocol has timed out, could be seen when LMP messages are not transmitted or received over 16 second time window. |
|
Ready |
Protocol is in ready state, this is the desired state in normal operation. Both LMP messages as well as SDP messages are exchanged successfully. |
Switch#show stackwise-virtual link
Stackwise Virtual Link(SVL) Information:
----------------------------------------
Flags:
------
Link Status
-----------
U-Up D-Down
Protocol Status
---------------
S-Suspended P-Pending E-Error T-Timeout R-Ready
-----------------------------------------------
Switch SVL Ports Link-Status Protocol-Status
------ --- ----- ----------- ---------------
1 1 FortyGigabitEthernet1/1/0/3 U R
FortyGigabitEthernet1/1/0/5 U R
2 1 FortyGigabitEthernet2/1/0/3 U R
FortyGigabitEthernet2/1/0/5 U R
Check DAD Link Status
Switch#show stackwise-virtual dual-active-detection Dual-Active-Detection Configuration: ------------------------------------- Switch Dad port Status ------ ------------ --------- 1 FortyGigabitEthernet1/0/4 up 2 FortyGigabitEthernet2/0/4 up
Switch#show stackwise-virtual dual-active-detection Dual-Active-Detection Configuration: ------------------------------------- Switch Dad port Status ------ ------------ --------- 1 FortyGigabitEthernet1/1/0/4 up 2 FortyGigabitEthernet2/1/0/4 up
Check ASIC Core IFM Map
Switch#sh platform software fed sw active ifm mapp Interface IF_ID Inst Asic Core Port SubPort Mac Cntx LPN GPN Type Active
TwentyFiveGigE1/0/1 0x3c 1 0 1 20 0 16 4 1 97 NIF Y
TwentyFiveGigE1/0/2 0x3d 1 0 1 21 0 17 5 2 98 NIF Y Switch#sh platform software fed sw standby ifm mapp Interface IF_ID Inst Asic Core Port SubPort Mac Cntx LPN GPN Type Active
TwentyFiveGigE1/0/1 0x8 1 0 1 20 0 16 4 1 1 NIF Y
TwentyFiveGigE1/0/2 0x9 1 0 1 21 0 17 5 2 2 NIF Y
Switch#sh platform software fed sw active ifm mapp Interface IF_ID Inst Asic Core Port SubPort Mac Cntx LPN GPN Type Active FortyGigabitEthernet1/1/0/3 0xb 0 0 0 16 0 16 0 3 2360 NIF Y FortyGigabitEthernet1/1/0/5 0xd 1 0 1 8 0 14 1 5 2361 NIF Y Switch#sh platform software fed sw standby ifm mapp Interface IF_ID Inst Asic Core Port SubPort Mac Cntx LPN GPN Type Active
FortyGigabitEthernet2/1/0/3 0x6b 0 0 0 16 0 16 0 3 2361 NIF Y
FortyGigabitEthernet2/1/0/5 0x6d 1 0 1 8 0 14 1 5 2360 NIF Y
Check FED Channel Health
Switch#show platform software fed switch active fss counters
FSS Packet Counters
SDP LMP
TX | RX TX | RX
--------------------- ---------------------
72651 72666 1157750 1154641
OOB1 OOB2
TX | RX TX | RX
--------------------- ---------------------
8 8 7740057 7590208
EMP
TX | RX LOOPBACK
---------------------- -------------------
0 0 79
Switch#show platform software fed switch active fss err-pkt-counters latency
Switch#show platform software fed switch active fss err-pkt-counters seqerr
Switch#show platform software fed switch active fss registers | i group
Check LMP Health
Use this show commands to check the LMP health.
Switch#show platform software fed sw active fss bundle Stack Port (0-Based) 0
Control port 16
Next Probable Control port Unknown
Member Port LPN list
-------------------
LPN:Partner_LPN
1.16:1.16
Stack Port (0-Based) 1
Control port Unknown
Next Probable Control port Unknown
Member Port LPN list
-------------------
LPN:Partner_LPN
Switch#show platform software fed switch active fss sdp-packets
FED FSS SDP packets max 10:
------------------------
FED-> Nif Mgr
Timestamp Src Mac Dst Mac Seq Num
--------------------------------------------------------------------
Sun Nov 15 18:59:07 2020 bc26:c722:9ef8 ffff:ffff:ffff 51843
Sun Nov 15 18:59:11 2020 bc26:c722:9ef8 ffff:ffff:ffff 51844
Sun Nov 15 18:59:15 2020 bc26:c722:9ef8 ffff:ffff:ffff 51845
Sun Nov 15 18:59:19 2020 bc26:c722:9ef8 ffff:ffff:ffff 51846
Sun Nov 15 18:59:23 2020 bc26:c722:9ef8 ffff:ffff:ffff 51847
Sun Nov 15 18:59:27 2020 bc26:c722:9ef8 ffff:ffff:ffff 51848
Sun Nov 15 18:59:31 2020 bc26:c722:9ef8 ffff:ffff:ffff 51849
Sun Nov 15 18:59:35 2020 bc26:c722:9ef8 ffff:ffff:ffff 51850
Sun Nov 15 18:58:59 2020 bc26:c722:9ef8 ffff:ffff:ffff 51841
Sun Nov 15 18:59:03 2020 bc26:c722:9ef8 ffff:ffff:ffff 51842
Nif Mgr -> FED
Timestamp Src Mac Dst Mac Seq Num
--------------------------------------------------------------------
Sun Nov 15 18:59:29 2020 bc26:c722:dae0 ffff:ffff:ffff 51863
Sun Nov 15 18:59:33 2020 bc26:c722:dae0 ffff:ffff:ffff 51864
Sun Nov 15 18:59:37 2020 bc26:c722:dae0 ffff:ffff:ffff 51865
Sun Nov 15 18:59:01 2020 bc26:c722:dae0 ffff:ffff:ffff 51856
Sun Nov 15 18:59:05 2020 bc26:c722:dae0 ffff:ffff:ffff 51857
Sun Nov 15 18:59:09 2020 bc26:c722:dae0 ffff:ffff:ffff 51858
Sun Nov 15 18:59:13 2020 bc26:c722:dae0 ffff:ffff:ffff 51859
Sun Nov 15 18:59:17 2020 bc26:c722:dae0 ffff:ffff:ffff 51860
Sun Nov 15 18:59:21 2020 bc26:c722:dae0 ffff:ffff:ffff 51861
Sun Nov 15 18:59:25 2020 bc26:c722:dae0 ffff:ffff:ffff 51862
Switch#show platform software fed switch active fss lmp-packets
Interface:TenGigabitEthernet1/3/0/1 IFID:0x37
FED FSS LMP packets max 10:
FED --> Nif Mgr
Timestamp Local Peer Seq
LPN LPN Num
-------------------------------------------------------
Sun Nov 15 19:01:31 2020 1 1 206696
Sun Nov 15 19:01:32 2020 1 1 206697
Sun Nov 15 19:01:33 2020 1 1 206698
Sun Nov 15 19:01:34 2020 1 1 206699
Sun Nov 15 19:01:36 2020 1 1 206701
Sun Nov 15 19:01:37 2020 1 1 206702
Sun Nov 15 19:01:27 2020 1 1 206692
Sun Nov 15 19:01:28 2020 1 1 206693
Sun Nov 15 19:01:29 2020 1 1 206694
Sun Nov 15 19:01:30 2020 1 1 206695
Nif Mgr --> FED
Timestamp Local Peer Seq
LPN LPN Num
-------------------------------------------------------
Sun Nov 15 19:01:29 2020 1 1 206696
Sun Nov 15 19:01:30 2020 1 1 206697
Sun Nov 15 19:01:31 2020 1 1 206698
Sun Nov 15 19:01:32 2020 1 1 206699
Sun Nov 15 19:01:33 2020 1 1 20670
Sun Nov 15 19:01:34 2020 1 1 206701
Sun Nov 15 19:01:35 2020 1 1 206702
Sun Nov 15 19:01:36 2020 1 1 206703
Sun Nov 15 19:01:37 2020 1 1 206704
Sun Nov 15 19:01:28 2020 1 1 206695
Interface:TenGigabitEthernet1/3/0/2 IFID:0x38
FED FSS LMP packets max 10:
FED --> Nif Mgr
Timestamp Local Peer Seq
LPN LPN Num
-------------------------------------------------------
Sun Nov 15 19:01:32 2020 2 2 206697
Sun Nov 15 19:01:33 2020 2 2 206698
Sun Nov 15 19:01:34 2020 2 2 206699
Sun Nov 15 19:01:35 2020 2 2 206700
Sun Nov 15 19:01:36 2020 2 2 206701
Sun Nov 15 19:01:37 2020 2 2 206702
Sun Nov 15 19:01:28 2020 2 2 206693
Sun Nov 15 19:01:29 2020 2 2 206694
Sun Nov 15 19:01:30 2020 2 2 206695
Sun Nov 15 19:01:31 2020 2 2 206696
Nif Mgr --> FED
Timestamp Local Peer Seq
LPN LPN Num
-------------------------------------------------------
Sun Nov 15 19:01:33 2020 2 2 206700
Sun Nov 15 19:01:34 2020 2 2 206701
Sun Nov 15 19:01:35 2020 2 2 206702
Sun Nov 15 19:01:36 2020 2 2 206703
Sun Nov 15 19:01:37 2020 2 2 206704
Sun Nov 15 19:01:28 2020 2 2 206695
Sun Nov 15 19:01:29 2020 2 2 206696
Sun Nov 15 19:01:30 2020 2 2 206697
Sun Nov 15 19:01:31 2020 2 2 206698
Sun Nov 15 19:01:32 2020 2 2 206699
Switch#show platform software fed switch active fss interface-counters
Interface TenGigabitEthernet1/3/0/1 IFID: 0x37 Counters
LMP
TX | RX
---------------------
206125 204784
Interface TenGigabitEthernet1/3/0/2 IFID: 0x38 Counters
LMP
TX | RX
---------------------
207012 206710
Switch#test platform software nif_mgr lmp member-port 1 Member port LPN 1 details ---------------------------- Transmitting on LPN: 1 member_port idx: 0 Stack Port: 0 Connection Status: Ready Port Link Status: Up LMP HELLO disabled: FALSE LMP Tx count: 3864 LMP Tx seq no: 3864 LMP Rx count: 3856 LMP Timeout Rx count: 0 LMP Partner Platform Information: Blueshift Version:1 Distributed Stack Domain:100 Distributed Stack Mode:1 System Model String:C9500-24Q System Product ID:FCW2144A3KF System Version ID:V01 Stack MAC Address:0027:90be:1f00 System CMI Index:0 LMP Port LPN:1 System Switch Number:2 LMP PENDING Partner Platform Information: Blueshift Version:1 Distributed Stack Domain:100 Distributed Stack Mode:1 System Model String:C9500-24Q System Product ID:FCW2144A3KF System Version ID:V01 Stack MAC Address:0027:90be:1f00 System CMI Index:0 LMP Port LPN:1 Switch#test platform software nif_mgr lmp member-port 2 Member port LPN 2 details ---------------------------- Transmitting on LPN: 2 member_port idx: 1 Stack Port: 0 Connection Status: Ready Port Link Status: Up LMP HELLO disabled: FALSE LMP Tx count: 3873 LMP Tx seq no: 3873 LMP Rx count: 3870 LMP Timeout Rx count: 0 LMP Partner Platform Information: Blueshift Version:1 Distributed Stack Domain:100 Distributed Stack Mode:1 System Model String:C9500-24Q System Product ID:FCW2144A3KF System Version ID:V01 Stack MAC Address:0027:90be:1f00 System CMI Index:0 LMP Port LPN:2 System Switch Number:2 LMP PENDING Partner Platform Information: Blueshift Version:1 Distributed Stack Domain:100 Distributed Stack Mode:1 System Model String:C9500-24Q System Product ID:FCW2144A3KF System Version ID:V01 Stack MAC Address:0027:90be:1f00 System CMI Index:0 LMP Port LPN:2
Switch#test platform software nif_mgr lmp status
Switch#test platform software nif_mgr lmp stack-port 1
Stack port 1 details
----------------------------
stack_port idx:0
Stack Link status:Up
Number Member Ports:1
Member Port LPN List:
1/16,
Switch#test platform software nif_mgr lmp stack-port 2
Stack port 2 details
----------------------------
stack_port idx:1
Stack Link status:Down
Number Member Ports:0
Member Port LPN List:
Shut/Unshut SVL Port
Shut and no shut comands are disabled on SVL and DAD ports when the system configured in StackWise-virtual mode. For test purposes, if there is a need to shut/unshut SVL ports, try as shown here:
Switch#test platform software nif_mgr port ? disable shutdown port enable unshut port
Alternatively, use the next approach which can simulate SFP/QSFP OIR from software (for example, on Cat9400/Cat9500H/Cat9600). These are hidden commands and requires service internal to be configured:
Switch#test idprom interface <...> ? fake-insert Fake insert fake-remove Fake remove
Extract Trace Archive from the System
When SVL Active switch can communicate with the SVL Standby switch, then a trace archive for both switches can be generated. Use this CLI.
Switch#request platform software trace rotate all
Switch#request platform software trace archive
Unable to archive /tmp/udev_ng4k.vbd.log : file does not exist
Unable to archive /tmp/vbd_app_init.log : file does not exist
excuting cmd on chassis 1 ...
sending cmd to chassis 2 ...
Creating archive file [flash:Switch_1_RP_0_trace_archive-20190807-073924.tar.gz]
Done with creation of the archive file: [flash:Switch_1_RP_0_trace_archive-20190807-073924.tar.gz]
Switch#
Switch#dir flash: | in tar
180238 -rw- 7189863 Aug 7 2019 07:39:34 +00:00 Switch_1_RP_0_trace_archive-20190807-073924.tar.gz
Switch#
In the case of Quad SUP, you need to collect a trace archive for all the SUP separately.
request platform software trace slot switch active R0 archive
request platform software trace slot switch active R1 archive
request platform software trace slot switch standby R0 archive
request platform software trace slot switch standby R1 archive
In situations where SVL Active switch cannot communicate with the SVL Standby switch such as SVL Split, ensure you generate a trace archive on both units.
show log file crashinfo:tracelogs/<filename>.bin.gz internal
show tech-support stackwise-virtual switch [active|standby|all|#]
New LMP and SDP Counters
Support added from release 16.10.1.
SDP - Stack Discovery Protocol - Traffic for role negotiation and topology.
The stack manager component is responsible for Role negotiation between stack members and elects the Active and Standby role. Stack Manager sends and receives the SDP packets over the SVL and gets the view of all the switches which are part of the Stackwise Virtual.
LMP - Link Management Protocol
L2 traffic to maintain the SVL. The link Management protocol is a software component, which runs a hello between ends and decides if the physical link is eligible to be part of the StackWise Virtual. LMP also monitors each configured physical link while they are part of the SVL. LMP is a part of the Network Interface Manager (Nif Mgr) software process.
FSS - Front Side Stack - Another name for StackWise Virtual.
From a software (Nif-mgr = Network Interface Manager) perspective:
show platform software nif-mgr switch active <R0 | R1> switch-info
show platform software nif-mgr switch active <R0 | R1> counters
show platform software nif-mgr switch active <R0 | R1> counters lpn 1
show platform software nif-mgr switch active <R0 | R1> packets
set platform software nif_mgr switch active <R0 | R1> pak-cache 40 —> set the packet cache count per SVL port to 40 (default = 10)From a hardware (FED = Forward Engine Driver) perspective:
show platform software fed switch active fss lmp-packets interface <type & #>
show platform software fed switch active fss sdp-packets
show platform software fed switch active fss counters
show platform software fed switch active fss interface-counters <interface type & #>
set platform software fed switch active F1 active fss pak-cache 40 —> set the packet cache count per SVL port to 40 (default = 10)
Check Quad SUP Redundancy Status
Quad Sup system has a max of 4 Supervisors with 2 Supervisors present per chassis. Show module provides a complete view of both the chassis and the SUP present in the System.
You can see the Slots in which SUP are present and the Supervisor which is marked as In chassis Standby update with status as provisioned.
SG_SVL_QuadSup#show module
Chassis Type: C9606R
Switch Number 1
Mod Ports Card Type Model Serial No.
---+-----+--------------------------------------+--------------+--------------
1 48 48-Port 10GE / 25GE C9600-LC-48YL CAT2310L58W
2 24 24-Port 40GE/12-Port 100GE C9600-LC-24C CAT2310L4CP
3 0 Supervisor 1 Module C9600-SUP-1 CAT2319L302
4 0 Supervisor 1 Module C9600-SUP-1 CAT2319L301
5 48 48-Port 10GE / 25GE C9600-LC-48YL CAT2312L2G7
6 24 24-Port 40GE/12-Port 100GE C9600-LC-24C CAT2310L4D6
Mod MAC addresses Hw Fw Sw Status
---+--------------------------------+----+------------+------------------+--------
1 DC8C.371D.2300 to DC8C.371D.237F 1.0 17.3.1r[FC2] 2020-11-05_13.02_s ok
2 DC8C.371D.2080 to DC8C.371D.20FF 1.0 17.3.1r[FC2] 2020-11-05_13.02_s ok
3 DC8C.37CA.6500 to DC8C.37CA.657F 1.0 17.3.1r[FC2] 2020-11-05_13.02_s ok
4 -- -- N/A -- Provisioned
5 DC8C.37A0.6880 to DC8C.37A0.68FF 1.0 17.3.1r[FC2] 2020-11-05_13.02_s ok
6 DC8C.371D.1A80 to DC8C.371D.1AFF 1.0 17.3.1r[FC2] 2020-11-05_13.02_s ok
Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode
---+-------------------+-------------------------+---------------------------
3 Standby sso sso
4 InChassis-Standby rpr rpr
Switch Number 2
Mod Ports Card Type Model Serial No.
---+-----+--------------------------------------+--------------+--------------
1 24 24-Port 40GE/12-Port 100GE C9600-LC-24C CAT2313L2WE
3 0 Supervisor 1 Module C9600-SUP-1 CAT2321L553
4 0 Supervisor 1 Module C9600-SUP-1 CAT2319L309
5 48 48-Port 10GE / 25GE C9600-LC-48YL CAT2312L2C5
6 48 48-Port 10GE / 25GE C9600-LC-48YL CAT2312L2DW
Mod MAC addresses Hw Fw Sw Status
---+--------------------------------+----+------------+------------------+--------
1 DC8C.37A0.C480 to DC8C.37A0.C4FF 1.0 17.3.1r[FC2] 2020-11-05_13.02_s ok
3 DC8C.37CA.6D00 to DC8C.37CA.6D7F 1.0 17.3.1r[FC2] 2020-11-05_13.02_s ok
4 -- -- N/A -- Provisioned
5 DC8C.37A0.5F80 to DC8C.37A0.5FFF 1.0 17.3.1r[FC2] 2020-11-05_13.02_s ok
6 DC8C.37A0.5C80 to DC8C.37A0.5CFF 1.0 17.3.1r[FC2] 2020-11-05_13.02_s ok
Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode
---+-------------------+-------------------------+---------------------------
3 Active sso sso
4 InChassis-Standby rpr rpr
Chassis 1 MAC address range: 64 addresses from 2c4f.5204.c080 to 2c4f.5204.c0bf
Chassis 2 MAC address range: 64 addresses from 2c4f.5204.bec0 to 2c4f.5204.beff
To observe per sup detailed redundancy state, the next show CLI can be used. Details provided includes Current up time and image details.
SG_SVL_QuadSup#sh redundancy rpr
My Switch Id = 2
Peer Switch Id = 1
Last switchover reason = none
Configured Redundancy Mode = sso
Operating Redundancy Mode = sso
Switch 2 Slot 3 Processor Information:
---------------------------------------------
Current Software State = ACTIVE
Uptime in current state = 18 minutes
Image Version = Cisco IOS Software [Bengaluru], Catalyst L3 Switch Software (CAT9K_IOSXE), Experimental Version 17.5.20201101:045441 [s2c/polaris_dev-/nobackup/suhs/latest/polaris 104]
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Sun 01-Nov-20 10:28 by suhs
BOOT = bootflash:packages.conf;
Switch 2 Slot 4 Processor Information:
---------------------------------------------
Current Software State = InChassis-Standby (Ready)
Uptime in current state = 18 minutes
Image Version =
BOOT = bootflash:packages.conf;
Switch 1 Slot 3 Processor Information:
---------------------------------------------
Current Software State = STANDBY HOT
Uptime in current state = 18 minutes
Image Version = Cisco IOS® Software [Bengaluru], Catalyst L3 Switch Software (CAT9K_IOSXE), Experimental Version 17.5.20201101:045441 [s2c/polaris_dev-/nobackup/suhs/latest/polaris 104]
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Sun 01-Nov-20 10:28 by suhs
BOOT = bootflash:packages.conf;
Switch 1 Slot 4 Processor Information:
---------------------------------------------
Current Software State = InChassis-Standby (Ready)
Uptime in current state = 18 minutes
Image Version =
BOOT = bootflash:packages.conf;
Other Major Enabler for Quad SUP system is to make Global Active Sup have an access to all the SUP File system , as well as reachability.
This provides the means to replicare the Configuration, Images, License, and other configured parameters across all the SUP.
Use this CLI to validate the File system availability:
SG_SVL_QuadSup#dir ?
/all List all files
/recursive List files recursively
all-filesystems List files on all filesystems
bootflash-1-0: Directory or file name
bootflash-1-1: Directory or file name
bootflash-2-1: Directory or file name
bootflash: Directory or file name
cns: Directory or file name
crashinfo-1-0: Directory or file name
crashinfo-1-1: Directory or file name
crashinfo-2-1: Directory or file name
crashinfo: Directory or file name
disk0-1-0: Directory or file name
disk0-1-1: Directory or file name
disk0-2-1: Directory or file name
disk0: Directory or file name
flash: Directory or file name
null: Directory or file name
nvram: Directory or file name
revrcsf: Directory or file name
stby-bootflash: Directory or file name
stby-crashinfo: Directory or file name
stby-disk0: Directory or file name
stby-nvram: Directory or file name
stby-rcsf: Directory or file name
system: Directory or file name
tar: Directory or file name
tmpsys: Directory or file name
usbflash0: Directory or file name
webui: Directory or file name
| Output modifiers
<cr> <cr>
SG_SVL_QuadSup#sh file systems
File Systems:
Size(b) Free(b) Type Flags Prefixes
- - opaque rw system:
- - opaque rw tmpsys:
* 11250098176 8731799552 disk rw bootflash: flash:
11250171904 7888437248 disk rw bootflash-1-0:
1651314688 0 disk rw crashinfo:
1651507200 0 disk rw crashinfo-1-0:
944993665024 896891006976 disk rw disk0:
944994516992 896892141568 disk rw disk0-1-0:
15988776960 15988768768 disk rw usbflash0:
7663022080 7542669312 disk ro webui:
- - opaque rw null:
- - opaque ro tar:
- - network rw tftp:
33554432 33483313 nvram rw nvram:
- - opaque wo syslog:
- - network rw rcp:
- - network rw http:
- - network rw ftp:
- - network rw scp:
- - network rw sftp:
- - network rw https:
- - opaque ro cns:
11250171904 6551502848 disk rw bootflash-2-1:
1651507200 0 disk rw crashinfo-2-1:
944994516992 896136118272 disk rw disk0-2-1:
11250171904 6074400768 disk rw bootflash-1-1:
1651507200 0 disk rw crashinfo-1-1:
945128734720 896416088064 disk rw disk0-1-1:
33554432 33479217 nvram rw stby-nvram:
- - nvram rw stby-rcsf:
11250098176 7888244736 disk rw stby-bootflash:
1651314688 0 disk rw stby-crashinfo:
944993665024 896891629568 disk rw stby-disk0:
- - opaque rw revrcsf:
Migrate 2 Sup SVL System to Quad SUP SVL System
Refer to this link for the Migration steps.
Transfer Catalyst 9600 to Quad-Supervisor Stackwise Virtual Setup
Quad SUP Switchover Behavior
Quad SUP with all the four SUP populated in a system failure of Active SUP tracks a Z model for transition to New Active Supposition.
This image depicts the New Active SUP transition on the failure of the current Active SUP.
Use CLI mentioned as part of Section 9 to check for the current SUP status and health of each sup at any point of Switchover. In addition show redundancy CLI can be used to track the Global Active / Standby redundancy transition details along with the history of switchover.
Quad SUP ISSU Behavior
Quad SUP with all the four Sup populated in a system, ISSU to new V2 image tracks a minimal data path impact model.
One shot ISSU progression is captured in this image which starts with:
- Copy the new V2 image to all 4 SUP and activate both the ICS with New V2 image. This results in a reload of both the ICS to boot up with New V2 image.
- Activate the Global Standby with V2, which results in the switchover to In-chassis Standby Sup associated with the Global standby followed by Old Global standby come up as ICS with V2.
- Activate the Global Active with V2 image which results in Switchover to In-chassis Standby associated with the Global Active. On successful boot up of Old Global Active as ICS with V2, completes the commit operation.
Troubleshoot Common Scenarios/Issues
StackWise Commands Do not Work
StackWise virtual commands are visible/available to configure only when the license level is configured as Network Advantage in case of cat9k switches and IPBase or IPServices in case of Cat3k switches.
Possible Cause
These issues are mainly seen when network essentials are not configured properly.
- On Cat3k, check if a license is IPBase or IPServices, use show version command.
- On Cat9K, check if a license is Network Advantage, use show version command.
- Check if MODEL_NUM rommon variable is populated correctly and matches actual device model.
Tips to Troubleshoot
If the license is right and still commands are not available, it could be a config sync issue in which case use these commands to collect the information that helps to debug this issue:
1. show version
2. show license summary
3. show romvar | i MODEL_NUM
Possible Solution
1. Configure the right license level based on the platform and retry.
2. If the Model number is different, the mostly the SUP can be old protoboard, you can correct the model number through ROMMON.
StackWise Configuration is Missed
When the system comes up, you can see StackWise configuration missed from the running configuration.
Possible Cause
Check if the desired configuration exists in the running-config database. Most likely it is an iosd config sync issue or the License level is configured back to non network-advantage.
Tips to Troubleshoot
1. Check LICENSE LEVEL, to see if SVL is supported for that license level. Ensure license level is set to network-advantage or the previous show license summary.
2. Check running-config and startup-config show running-config and show startup-config.
If the issue is immediately after ISSU and there is a mismatch between output of startup-config and running-config, then it could be an ISSU issue. Refer to the next step.
3. Check the output for the ISSU impacted CLI. show redundancy config-sync failure mcl.
If the interested configuration is missed in both running and startup-config, navigate to this next step.
4. Check rommon variables, use show romvar | inc D_S.
If the interested configuration is missed or present in romvar, do the next step.
5. Check show issu state detail.
If you are unable to troubleshoot, then collect the next command outputs to investigate further.
1. show platform software fed switch active fss counters (3-4 Increments)
2. test platform software nif_mgr lmp member-port LPN
3. collect show tech-support stackwise-virtual
4. collect trace archive file
Possible Solution
Configure the right license level based on the platform and retry. If the issue occurs after ISSU, then mostly you have to clean up or reconfigured the configuration that is listed in MCL failures.
StackWise Links/Dual Active Detection Links are Down
Possible Cause
There are many reasons for SVL/DAD links to go down. It is hard to tell just with a look at show command outputs, it needs some specific btrace logs along with show command output to carefully analyze and find the root cause.
Some of the most common ones that can cause Link down are bad SFP or bad cable or it could be a simple display issue.
Tips to Troubleshoot
1. Be clear on what is the status of the link, use command show stackwise-virtual link.
2. Ensure the physical connectivity of the SVL ports and that their configuration is good. Check show stackwise-virtual.
If you are unable to troubleshoot, then provide the next command outputs to investigate.
1. show platform software fed switch active fss counters (3-4 Increments)
2. test platform software nif_mgr lmp member-port LPN
3. collect show tech-support stackwise-virtual
4. collect trace archive file
SVL Protocol Status Does not Show R
Be clear on what is the status of the link shown in the show stackwise-virtual command.
Possible Cause:
1. If Protocol is in S-Suspended state, then most of the times means that the link is Down.
2. If Protocol is in T-Timeout state, then it means that it is not able to receive/send LMP packets.
3. For Timeout/Pending state,it is important to capture Nif_mgr, Fed and also just in case stack_mgr btrace logs.
4. Check if switch recovers with shut/no shut of SVL Links ( just a data pointer and not a solution).
Tips to Troubleshoot
If you are unable to find the root cause, provide these next command outputs
1. Check SVL/DAD link status. Use show stackwise-virtual link.
2. Check show platform software fed switch active/standby fss counters (3-4 Increments)
3. test platform software nif_mgr lmp member-port LPN.
4. Check show platform software fed switch active/standby ifm map.
6. There could be a possible issue with a specific type of xcvr used on SVL/DAD link. Verify show interface <SVL/DAD link> status and show idprom <SVL/DAD link>.
7. Check show platform to ensure SUP/LC that has SVL/DAD links are in OK state.
If you are unable to troubleshoot, then provide these next command outputs for the Engineering team to investigate.
1. Collect trace archive file
2. Collect show tech-support stackwise-virtual
3. show platform software fed switch active/standby fss counters (3-4 Increments)
Possible Solution:
1. Shut/ No Shut SVL LINKS: test platform software nif_mgr port disable / enable LPN
2 Physical OIR of the transceiver on SVL/DAD links
SVL Link Flapping
Possible Cause
The encounter of link flap on an SV Testbed can be in multiple domains. The most common and basic information to be collected irrespective of the domain/area of the flap are listed in the next section
Tips to Troubleshoot
1. Generate trace archive.
2. Check FED channel Health.
3. Check LMP health.
4. Identify ASIC/Core map which corresponds to SVL link.
5. collect show tech-support stackwise-virtual
If you are unable to find the root cause, provide these command outputs for further troubleshooting from the Engineering team.
1. test platform software nif_mgr lmp member-port <slot> <lpn port>
2. show platform hardware iomd switch 1/2 1/0 lc-portmap brief | detail
3. show tech-support stackwise-virtual
4. collect trace archive file
Possible Solution
- Shut/ No Shut SVL LINKS: test platform software nif_mgr port disable / enable LPN
- Physical OIR of the transceiver or try to change the transceiver or cable on SVL/DAD links.
Error-Disable of the SVL/DAD Links
Possible Cause
The transceiver/SFP/QSFP used on the SVL/DAD links can be faulty at the hardware layer and the xcvrs can itself generate frequent link flaps and thereby error-disable the SVL/DAD links.
Tips to Troubleshoot
1. Verify show idprom interface <SVL/DAD link>
2. Verify if any of the interfaces are indeed err-disabled on the switch. Use show interfaces status err-disabled.
If you are unable to find the root cause, provide these command outputs
1. show errdisable flap-values
2. show errdisable recovery
3. show errdisable detect
4. show tech-support stackwise-virtual
5. collect trace archive file
Possible Solution
1. Shut/ No Shut SVL LINKS: test platform software nif_mgr port disable / enable LPN
2. Physical OIR of the transceiver or try to change the transceiver or cable on SVL/DAD links.
Stack Does not Get Discovered
Expected output upon system come up would be:
Waitingfor 120 seconds for other switches to boot
#####################################################
Switch number is 2
All switches in the stack have been discovered. Accelerating discovery
Possible Cause
Stack discovery failures can be because of multiple reasons. Some of them are listed here:
1. SVL links status can NOT be UP or protocol status can not be Ready.
2. SVL links flap.
3. One of the stack switches can have crashed or gone down.
4. Accidentally pull off cable or transceiver or shut down of SVL links.
Tips to Troubleshoot
In an accelerate discovery, if the stack does not get discovered, you can check the link status of all the SVL. It can be that none of the SVL Links were up. You can check this with the command show stackwise-virtual link.
If you are unable to find the root cause, provide these command outputs along with the trace archive
1. collect trace archive file
2. collect show tech-support stackwise-virtual
3. show platform software fed switch active/standby fss counters (3-4 Increments)
Switch is Not in Recovery Mode
When all the SVL links are down and DAD links are configured on the SVL, the active switch must be in recovery-mode.
Switch(recovery-mode)#
Possible Cause
DAD link can have gone down because of either physical removal or shut down of the port.
Tips to Troubleshoot
1. Check if the DAD links are UP, use show stackwise-virtual dual-active-detection [pagp].
2. If DAD Links are down, check why the link is down.
5. If DAD links are up and the still switch does not move to recovery mode, you must check the IleMstStateTable bits for the particular DAD ports as a single DAD packet is sent out and it can be missed.
If you are unable to find the root cause or troubleshoot, provide these command outputs along with the trace archive
1. test platform software nif_mgr lmp member-port <slot> <lpn port>
2. show interface status <SVL>
3. collect trace archive file
4. collect show tech-support stackwise-virtual
5. show platform software fed switch active/standby fss counters (3-4 Increments)
Switch Gets to Recovery Mode before Removal of all SVL Links
Possible Cause
Active switch gets into recovery mode could be several reasons. Listed next are a few of the possible causes.
1. SVL links can experience unexpected link flaps.
2. SVL links can experience unexpected remote/local link faults.
3. There could be a problem in LMP/SDP packets exchange over the SVL links.
Tips to Troubleshoot
1. If the switch goes to recovery mode just by the removal of half / less than complete SVL links > use the test command to shut/no shut SVL > check if it is possible to reproduce the issue with physical OIR.
2. If it still goes to recovery mode without shut command, then first probably the root cause → Control port can not be a shift.
3. SVL links can be time out, LMP packets can not get exchanged correctly between active/standby switches.
5. Fed channel health is important to check if other packets do go through.
If you are unable to find the root cause or troubleshoot, provide these command outputs along with the trace archive.
1. show stackwise-virtual link
2. show stackwise-virtual dual-active-detection
3. show platform software fed switch active fss counters (3-4 Increments)
4. test platform software nif_mgr lmp member-port <slot> <lpn port>
5. show platform software fed switch active fss bundle
6. collect trace archive file
7. collect show tech-support stackwise-virtual
Ports do not Get Err-Disabled in Recovery Mode
Tips to Troubleshoot
If you are unable to find the root cause or troubleshoot, provide these command outputs along with the trace archive
1. show interface status
2. show stackwise-virtual dual-active-detection
3. show stackwise-virtual dual-active-detection pagp
4. test platform software nif_mgr lmp member-port LPN
Direct Ping from a Device Connected to Active, no MEC
Tips to Troubleshoot
If ping over active SVL does not work, and cannot be SVL issue, increase size of ping packets for debuggability purpose, track packet drop path (request/response) and also break stack and try the same.
If you are unable to find the root cause or troubleshoot, provide these command outputs along with the trace archive
1. show stackwise-virtual link
2. show platform software fed switch active fss counters (3-4 Increments)
3. test platform software nif_mgr lmp member-port LPN
4. show platform software fed switch active ifm map
5. show tech-support stackwise-virtual
TRAFFIC DROP CHECK:
7. clear controllers ethernet-controller
8. show controllers ethernet-controller <Te/Fo interfaces if traffic path>
Direct Ping from a Device Connected to Standby, no MEC
Tips to Troubleshoot
1. if ping over standby does not work, increase the size of ping packets for debuggability purpose, track packet drop path (request/response).
If you are unable to find the root cause or troubleshoot, provide these command outputs along with the trace archive.
1. show stackwise-virtual link
2. show platform software fed switch active fss counters (3-4 Increments)
3. test platform software nif_mgr lmp member-port LPN
4. show platform software fed switch active fss ifm map
5. show tech-support stackwise-virtual
TRAFFIC DROP CHECK:
7. clear controllers ethernet-controller
8. show controllers ethernet-controller <Te/Fo interfaces if traffic path>
9. show interfaces counters errors
Ping over Port-Channel
Tips to Troubleshoot
1. To diagnose issue isolate problem to either active port-channel port or standby port-channel port.
2. Once the previous isolation is established, repeat actions for either active or standby as listed previously.
3. Ensure port-channel member ports are up on each end of port-channel.
If you are unable to find the root cause or troubleshoot, provide the previous command outputs along with the trace archive:
1. show stackwise-virtual link
2. show platform software fed switch active fss counters (3-4 Increments)
3. test platform software nif_mgr lmp member-port LPN
4. show platform software fed switch active fss ifm map
5. show tech-support stackwise-virtual
TRAFFIC DROP CHECK:
7. clear controllers ethernet-controller
8. show controllers ethernet-controller <Te/Fo interfaces if traffic path
9. show interfaces counters errors
FED Crash / IOSd Crash/ Stack_mgr
Tips to Troubleshoot
Collect and provide this information:
1. Collect console logs.
2. System report and core file decode if any.
Chassis Lost both Active and Standby
Tips to Troubleshoot
Collect and provide this information:
1. Complete system report.
2. Links State and Protocol State of SVLs.
Active Removed before Stdby became Ready
Tips to Troubleshoot
Collect and provide this information:
1. Complete system report mainly.
2. Links State and Protocol State of SVLs.
Traffic
Tips to Troubleshoot
1. Ensure source and destination mac addresses are set accordingly in traffic streams.
2. Ensure traffic path is in same vlan domain or trunk mode.
3. If drop is on active and traffic stream is not expected to traverse over SVL, cannot be SVL issue, break stack and try.
4. if drop is on standby and packet is expected to traverse over SVL, collect csv dump, ifm map.
5. Identify drop, use show controller ethernet-controller <interface>.
6. if port-channel is involved, try to isolate drop by shutting down one member of port-channel to determine if drop is on active or standby, repeat same steps listed previously.
If you are unable to find the root cause or troubleshoot, provide these command outputs along with the trace archive.
1. show stackwise-virtual link
2. show platform software fed switch active fss counters (3-4 Increments)
3. test platform software nif_mgr lmp member-port LPN
4. show platform software fed switch active ifm map
5. show tech-support stackwise-virtual
TRAFFIC DROP CHECK:
clear controllers ethernet-controller
show controllers ethernet-controller <Te/Fo interfaces if traffic path>
show interfaces counters errors
L3 Traffic
Tips to Troubleshoot
1. Ensure arp entries are resolved and necessary routes have been added.
2. Ensure source and destination IP addresses are set correctly.
3. If drop is on active and traffic stream is not expected to traverse over SVL, cannot be SVL issue, break stack and try.
4. if drop is on standby and packet is expected to traverse over SVL, collect csv dump, ifm map
5. Identify drop, use show controller ethernet-controller <interface>.
6. If port-channel is involved, try to isolate drop by shutting down one member of port-channel to determine if the drop is on active or standby, repeat the same steps listed previously.
If you are unable to find the root cause or troubleshoot, provide these command outputs along with the trace archive
- show stackwise-virtual link
- show platform software fed switch active fss counters (3-4 Increments)
- test platform software nif_mgr lmp member-port LPN
- show platform software fed switch active ifm map
- show tech-support stackwise-virtual
TRAFFIC DROP CHECK:
clear controllers ethernet-controller
show controllers ethernet-controller <Te/Fo interfaces if traffic path>
show interfaces counters errors
Traffic Drop on SVL
Tips to Troubleshoot
1. Identify where traffic is dropped such as
- ingress switch on ingress interface [not related to SVL]
- ingress switch on egress SVL interface
- egress switch on ingress SVL interface
- egress switch on egress interface [not related to SVL]
2. Generate trace archive.
3. Check FED channel Health.
4. Check LMP health.
5. Identify ASIC/Core map which corresponds to the SVL link.
6. Capture packets that get dropped on SVL.
FIPS on SVL
Possible Cause
There is always a chance that FIPS on the SVL key cannot have been configured individually on each switch. The same FIPS key must be configured individually on both switches which are part of the SVL.
1.The FIPS_KEY stored in rommon can differ even though you configured the same auth key on both sw-1 and sw-2. This is expected behavior.
2. Verify show fips status and ensure the switch is configured for fips mode.
Tips to Troubleshoot
1. Verify FIPS mode, use show fips status.
2. Verify show fips authorization-key.
3. Verify show romvar.
If you are unable to find the root cause or troubleshoot, provide these command outputs along with the trace archive
1. show fips status
2. show fips authorization-key
3. show platform software fed switch <active|standby> fss sesa-counters
4. show stackwise-virtual link
5. show stackwise-virtual
6. collect show tech-support stackwise-virtual
7. collect and provide trace archive
Secure SVL
Possible Cause
There is always a chance that a secure SVL key was not configured individually on each switch. Same Secure SVL key must be configured individually on both switches which are part of the SVL
1. The SSVL_KEY stored in rommon can differ even though you configured the same auth key on both sw-1 and sw-2. This is expected behavior.
2. Verify show secure-stackwise-virtual status and ensure its configured in SECURE-SVL mode.
Tips to Troubleshoot
1. The SSVL_KEY stored in rommon can differ even though you configured the same auth key on both sw-1 and sw-2. This is expected behavior. Use show romvar CLI.
2. If there is mismatch between Switch-1 and Switch-2 in the Secure SVL Key configuration, when you bring up SVL, you could potentially see an error messages like:
"stack_mgr: Reload due to reason SESA: SESA AER Request not received, Err: 31"
3. Verify show secure-stackwise-virtual status and ensure it configured in SECURE-SVL mode.
If you are unable to find the root cause or troubleshoot, provide these command outputs along with the trace archive
1. show secure-stackwise-virtual status
2. show secure-stackwise-virtual authorization-key
3. show secure-stackwise-virtual interface <SVL_LINK>
4. collect show tech-support stackwise-virtual
5. debug secure-stackwise-virtual
6. collect and provide trace archive
V-Mistmatch in the Switch
Possible Cause
A different version of the software booted on active and standby chassis leads to V-Mismatch in SVL
Tips to Troubleshoot
Verify show switch and check if there is any V-Mismatch.
Possible Solution
There is a feature called software auto-upgrade which is enabled by default on Cat9k platforms. This feature when enabled and detects software mismatch and pushes the software package which is present on the active to standby switch and the standby switch gets auto-upgraded to the same software version which is on the active switch.
If software auto-upgrade is disabled, enable it and then reboot the standby switch so that software auto-upgrade kicks in and both active and standby switches have the same software version used in SVL.
Software auto-upgrade works only in install mode on Cat9k platforms which is the recommended boot mode.
1. Verify software auto-upgrade is enabled on the SVL. Use the show CLI:
switch#show run all | i software
software auto-upgrade enable
Stack Split with the Removal of Just One SVL Link out of Multiple SVL Links
Possible Cause
This type of problem can happen when transceivers are half/partially inserted on the front panel ports, there could be a chance that IOMD process can get into busy states when read idprom of these xcvrs. This can lead to slowness in execution and completion of some of the typical show commands like show idprom <interface>, show interface status, and so on active or standby switches in SVL.
1. Verify show idprom interface <intf>. Verify on both active and standby switch interfaces idprom dump and check if there is slowness/delay in completion of this CLI.
2. Verify show interfaces <intf>. Verify on both active and standby switches interfaces and check if there is slowness/delay in completion of this CLI.
3. Also, check if the CPU usage is high on the SVL, use show processes cpu sorted.
Tips to Troubleshoot
If you are unable to find the root cause, provide these command outputs along with the trace archive.
1. Check if CPU is high, use show processes cpu sorted.
2. Check if TDL subslot ping iworks fine, use test platform software tdl ping subslot <>.
Possible Solution
One of the possible causes for this issue could be half or loosely inserted transceivers on the front panel port of the switches in SVL.
Ensure that transceivers are seated/inserted properly on the front panel ports by physical inspection of the transceivers/cables.
Once that is done, verify the next commands to ensure things function well.
1. Verify show idprom interface <intf>. Verify on both active and standby switch interfaces idprom dump and check CLI gets executed without any delay in completion of this CLI.
2. Verify show interfaces <intf>. Verify on both active and standby switches interfaces and check CLI gets executed without any delay in completion of this CLI.
3. Also, check if the CPU usage is not high on the SVL, use show processes cpu sorted.
If the problem still persists, collect these CLI outputs and logs.
1. Collect show tech-support stackwise-virtual.
2. Collect and provide trace archive.
Revision History
| Revision | Publish Date | Comments |
|---|---|---|
3.0 |
15-Jul-2024 |
Updated Alt Text, Branding Requirements, Link Targets, and Formatting. |
2.0 |
02-Jun-2023 |
Recertification |
1.0 |
15-Dec-2020 |
Initial Release |
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)