本產品的文件集力求使用無偏見用語。針對本文件集的目的,無偏見係定義為未根據年齡、身心障礙、性別、種族身分、民族身分、性別傾向、社會經濟地位及交織性表示歧視的用語。由於本產品軟體使用者介面中硬式編碼的語言、根據 RFP 文件使用的語言,或引用第三方產品的語言,因此本文件中可能會出現例外狀況。深入瞭解思科如何使用包容性用語。
思科已使用電腦和人工技術翻譯本文件,讓全世界的使用者能夠以自己的語言理解支援內容。請注意,即使是最佳機器翻譯,也不如專業譯者翻譯的內容準確。Cisco Systems, Inc. 對這些翻譯的準確度概不負責,並建議一律查看原始英文文件(提供連結)。
本文檔介紹如何在Nexus 7k中的硬體模組無響應或間歇性時對其進行故障排除。
步驟 1.對各種SNMP V3使用者和/或SNMP V2社群字串執行snmpwalk(即遍歷主機名mib)。
在連續循環中執行此操作。
步驟2.使用ssh連線到有問題的VDC,該VDC在步驟1中對主機名具有間歇性無響應掃描。
由於步驟1和步驟2在60秒週期內同時受到影響,這似乎是一個N7K控制平面內的硬體故障,因為N7K始終運行硬體診斷運行狀況檢查。當您看到響應時間為30秒,無響應時間為30秒,然後循環重複時,這清楚地表明硬體診斷運行狀況檢查掃描所有硬體。30秒的響應時間是掃描正常硬體,而30秒的不響應時間是故障硬體。
步驟 3.如果步驟2.清楚地描述了硬體故障,請執行以下步驟:
註:EOBC是N7K用於在SUP/交換矩陣模組/線卡之間通訊的內部控制平面進程。如果此EOBC流程受到任何影響,管理VDC-1日誌檔案中描述的關聯模組最可能是先前測試中看到的間歇響應的罪魁禍首,因為SUP丟失了與管理VDC-1日誌檔案中描述的關聯模組的100%一致通訊,並且正在嘗試與其恢復/通訊,從而導致與其它控制平面流程的間歇響應。
範例:
lab-sw01-admin-vdc-1# show logging logfile | inc EOBC
2022 Feb 22 19:46:15 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure on standby sup in device DEV_EOBC_MAC (device error 0xc0a0504f)
2022 Feb 22 19:46:15 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure in device DEV_EOBC_MAC (device error 0xc0a0514d)
2022 Feb 22 19:46:16 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure on standby sup in device DEV_EOBC_MAC (device error 0xc0a0504f)
2022 Feb 22 19:46:16 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure in device DEV_EOBC_MAC (device error 0xc0a0514d)
2022 Feb 22 19:46:21 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure in device DEV_EOBC_MAC (device error 0xc0a0514d)
2022 Feb 22 19:46:21 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure on standby sup in device DEV_EOBC_MAC (device error 0xc0a0504f)
2022 Feb 22 19:46:22 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure in device DEV_EOBC_MAC (device error 0xc0a0514d)
2022 Feb 22 19:46:23 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure on standby sup in device DEV_EOBC_MAC (device error 0xc0a0504f)
2022 Feb 22 19:46:23 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure in device DEV_EOBC_MAC (device error 0xc0a0514d)
2022 Feb 22 19:46:24 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure on standby sup in device DEV_EOBC_MAC (device error 0xc0a0504f)
2022 Feb 22 19:46:24 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure in device DEV_EOBC_MAC (device error 0xc0a0514d)
2022 Feb 22 19:46:26 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure on standby sup in device DEV_EOBC_MAC (device error 0xc0a0504f)
2022 Feb 22 19:46:26 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure in device DEV_EOBC_MAC (device error 0xc0a0514d)
此日誌輸出清楚地顯示,模組8具有待命SUP的EOBC心跳故障,並且處於不正常狀態,需要立即採取措施。
步驟 1.執行show模組並捕獲資料以供參考:
lab-sw01-admin-vdc-1# show module
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 12 100 Gbps Ethernet Module N77-F312CK-26 ok
2 12 100 Gbps Ethernet Module N77-F312CK-26 ok
3 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
4 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
5 0 Supervisor Module-2 N77-SUP2E active *
6 0 Supervisor Module-2 N77-SUP2E ha-standby
7 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
8 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
Mod Sw Hw
--- --------------- ------
1 8.4(4) 1.5
2 8.4(4) 1.5
3 8.4(4) 1.9
4 8.4(4) 1.9
5 8.4(4) 1.3
6 8.4(4) 1.3
7 8.4(4) 1.2
8 8.4(4) 1.2
註意:所有模組均線上(即ok),模組5是主用(即active *)SUP,模組6是高可用性備用(即ha-standby)SUP。 雖然管理VDC日誌檔案中存在有關模組8的EOBC警告,但此輸出將模組8描述為「正常」。
步驟 2.重新載入交換機或執行Supervisor切換(即在Admin VDC內):
lab-sw01-admin-vdc-1# reload
- system (ie supervisor) switchover - NOTE: preferred method as this is a non-impacting procedure to the box with regards to active data flows
lab-sw01-admin-vdc-1# system switchover
注意:在任何一種情況下,執行重新載入或系統切換之前,請確保您同時處於兩個Supervisor控制檯上,以便您可以親眼看到所有Supervisor輸出。
步驟 3.如果模組8是可疑的罪魁禍首,您可能會在系統(即Supervisor)切換時看到控制檯模組8出現故障:
lab-sw01-admin-vdc-1(standby) login: 2022 Feb 23 02:09:45 lab-sw01-admin-vdc-1 %$ VDC-1 %$ %KERN-2-SYSTEM_MSG: [12392164.927835] Switchover started by redundancy driver - kernel
2022 Feb 23 02:09:45 lab-sw01-admin-vdc-1 %$ VDC-1 %$ %SYSMGR-2-HASWITCHOVER_PRE_START: This supervisor is becoming active (pre-start phase).
2022 Feb 23 02:09:45 lab-sw01-admin-vdc-1 %$ VDC-1 %$ %SYSMGR-2-HASWITCHOVER_START: Supervisor 6 is becoming active.
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth0/8 return status No card found in slot
2022 Feb 23 02:09:46 lab-sw01-admin-vdc-1 %$ VDC-1 %$ %SYSMGR-2-SWITCHOVER_OVER: Switchover completed.
2022 Feb 23 02:09:47 lab-sw01-admin-vdc-1 %$ VDC-1 %$ %PLATFORM-1-PFM_ALERT: Disabling ejector based shutdown on sup in slot 6
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth1/8 return status No card found in slot
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth2/8 return status No card found in slot
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth3/8 return status No card found in slot
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth4/8 return status No card found in slot
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth5/8 return status No card found in slot
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth6/8 return status No card found in slot
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth7/8 return status No card found in slot
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth8/8 return status No card found in slot
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth9/8 return status No card found in slot
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth10/8 return status No card found in slot
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth11/8 return status No card found in slot
步驟 4.執行多個show模組並觀察模組8是否重新聯機/何時重新聯機:
Module 5 dropped out and is powered-up:
Module 8 dropped out and is powered-up:
lab-sw01-admin-vdc-1# show module
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 12 100 Gbps Ethernet Module N77-F312CK-26 ok
2 12 100 Gbps Ethernet Module N77-F312CK-26 ok
3 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
4 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
5 0 Supervisor Module-2 powered-up
6 0 Supervisor Module-2 N77-SUP2E active *
7 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
8 24 10/40 Gbps Ethernet Module powered-up
Mod Power-Status Reason
--- ------------ ---------------------------
8 powered-up Unknown. Issue show system reset mod ...
Mod Sw Hw
--- --------------- ------
1 8.4(4) 1.5
2 8.4(4) 1.5
3 8.4(4) 1.9
4 8.4(4) 1.9
6 8.4(4) 1.3
7 8.4(4) 1.2
lab-sw01-admin-vdc-1# 2022 Feb 23 02:11:11 lab-sw01-vdc-2 %$ VDC-2 %$ %PLATFORM-2-MOD_DETECT: Module 8 detected (Serial number JAA00000000) Module-Type 10/40 Gbps Ethernet Module Model N77-F324FQ-25
2022 Feb 23 02:11:11 lab-sw01-vdc-2 %$ VDC-2 %$ %PLATFORM-2-MOD_PWRUP: Module 8 powered up (Serial number JAA00000000)
2022 Feb 23 02:11:11 lab-sw01-admin-vdc-1 %$ VDC-1 %$ %PLATFORM-2-MOD_DETECT: Module 8 detected (Serial number JAA00000000) Module-Type 10/40 Gbps Ethernet Module Model N77-F324FQ-25
2022 Feb 23 02:11:11 lab-sw01-admin-vdc-1 %$ VDC-1 %$ %PLATFORM-2-MOD_PWRUP: Module 8 powered up (Serial number JAA00000000)
Module 8 is pwr-cycled:
lab-sw01-admin-vdc-1# show module
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 12 100 Gbps Ethernet Module N77-F312CK-26 ok
2 12 100 Gbps Ethernet Module N77-F312CK-26 ok
3 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
4 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
5 0 Supervisor Module-2 powered-up
6 0 Supervisor Module-2 N77-SUP2E active *
7 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
8 24 10/40 Gbps Ethernet Module pwr-cycld
Mod Power-Status Reason
--- ------------ ---------------------------
8 pwr-cycld Unknown. Issue show system reset mod ...
Mod Sw Hw
--- --------------- ------
1 8.4(4) 1.5
2 8.4(4) 1.5
3 8.4(4) 1.9
4 8.4(4) 1.9
6 8.4(4) 1.3
7 8.4(4) 1.2
lab-sw01-admin-vdc-1# show module
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 12 100 Gbps Ethernet Module N77-F312CK-26 ok
2 12 100 Gbps Ethernet Module N77-F312CK-26 ok
3 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
4 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
5 0 Supervisor Module-2 powered-up
6 0 Supervisor Module-2 N77-SUP2E active *
7 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
8 24 10/40 Gbps Ethernet Module N77-F324FQ-25 powered-up
Mod Sw Hw
--- --------------- ------
1 8.4(4) 1.5
2 8.4(4) 1.5
3 8.4(4) 1.9
4 8.4(4) 1.9
6 8.4(4) 1.3
7 8.4(4) 1.2
8 8.4(4) 1.2
Module 8 is checked by epld auto-upgrade and is good to go:
lab-sw01-admin-vdc-1# 2022 Feb 23 02:13:06 lab-sw01-admin-vdc-1 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: <<%EPLD_AUTO-2-AUTO_UPGRADE_CHECK>> Automatic EPLD upgrade check for module 8: EPLD versions are up to date. - epld_auto
lab-sw01-admin-vdc-1# show module
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 12 100 Gbps Ethernet Module N77-F312CK-26 ok
2 12 100 Gbps Ethernet Module N77-F312CK-26 ok
3 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
4 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
5 0 Supervisor Module-2 powered-up
6 0 Supervisor Module-2 N77-SUP2E active *
7 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
8 24 10/40 Gbps Ethernet Module N77-F324FQ-25 powered-up
Mod Sw Hw
--- --------------- ------
1 8.4(4) 1.5
2 8.4(4) 1.5
3 8.4(4) 1.9
4 8.4(4) 1.9
6 8.4(4) 1.3
7 8.4(4) 1.2
8 8.4(4) 1.2
Module 8 moves to testing by the hardware diagnostics:
lab-sw01-admin-vdc-1# show module
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 12 100 Gbps Ethernet Module N77-F312CK-26 ok
2 12 100 Gbps Ethernet Module N77-F312CK-26 ok
3 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
4 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
5 0 Supervisor Module-2 powered-up
6 0 Supervisor Module-2 N77-SUP2E active *
7 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
8 24 10/40 Gbps Ethernet Module N77-F324FQ-25 testing
Mod Sw Hw
--- --------------- ------
1 8.4(4) 1.5
2 8.4(4) 1.5
3 8.4(4) 1.9
4 8.4(4) 1.9
6 8.4(4) 1.3
7 8.4(4) 1.2
8 8.4(4) 1.2
Module 8 moves to initializing after passing hardware diagnostics:
lab-sw01-admin-vdc-1# show module
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 12 100 Gbps Ethernet Module N77-F312CK-26 ok
2 12 100 Gbps Ethernet Module N77-F312CK-26 ok
3 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
4 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
5 0 Supervisor Module-2 powered-up
6 0 Supervisor Module-2 N77-SUP2E active *
7 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
8 24 10/40 Gbps Ethernet Module N77-F324FQ-25 initializing
Mod Sw Hw
--- --------------- ------
1 8.4(4) 1.5
2 8.4(4) 1.5
3 8.4(4) 1.9
4 8.4(4) 1.9
6 8.4(4) 1.3
7 8.4(4) 1.2
8 8.4(4) 1.2
Module 8 comes online:
lab-sw01-admin-vdc-1# show module
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 12 100 Gbps Ethernet Module N77-F312CK-26 ok
2 12 100 Gbps Ethernet Module N77-F312CK-26 ok
3 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
4 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
5 0 Supervisor Module-2 powered-up
6 0 Supervisor Module-2 N77-SUP2E active *
7 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
8 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
Mod Sw Hw
--- --------------- ------
1 8.4(4) 1.5
2 8.4(4) 1.5
3 8.4(4) 1.9
4 8.4(4) 1.9
6 8.4(4) 1.3
7 8.4(4) 1.2
8 8.4(4) 1.2
Module 5 SUP going active:
lab-sw01-admin-vdc-1# show module
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 12 100 Gbps Ethernet Module N77-F312CK-26 ok
2 12 100 Gbps Ethernet Module N77-F312CK-26 ok
3 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
4 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
5 0 Supervisor Module-2 N77-SUP2E inserted
6 0 Supervisor Module-2 N77-SUP2E active *
7 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
8 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
Mod Sw Hw
--- --------------- ------
1 8.4(4) 1.5
2 8.4(4) 1.5
3 8.4(4) 1.9
4 8.4(4) 1.9
5 8.4(4) 1.3
6 8.4(4) 1.3
7 8.4(4) 1.2
8 8.4(4) 1.2
Module 5 SUP becomes ha-standby:
2022 Feb 23 02:16:38 lab-sw01-admin-vdc-1 %$ VDC-1 %$ %PLATFORM-1-PFM_ALERT: Enabling ejector based shutdown on sup in slot 6
lab-sw01-admin-vdc-1# show module
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 12 100 Gbps Ethernet Module N77-F312CK-26 ok
2 12 100 Gbps Ethernet Module N77-F312CK-26 ok
3 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
4 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
5 0 Supervisor Module-2 N77-SUP2E ha-standby
6 0 Supervisor Module-2 N77-SUP2E active *
7 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
8 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
Mod Sw Hw
--- --------------- ------
1 8.4(4) 1.5
2 8.4(4) 1.5
3 8.4(4) 1.9
4 8.4(4) 1.9
5 8.4(4) 1.3
6 8.4(4) 1.3
7 8.4(4) 1.2
8 8.4(4) 1.2
2022 Feb 23 02:15:44 lab-sw01-admin-vdc-1 %MODULE-5-MOD_OK: Module 8 is online (Serial number: JAA00000000)
2022 Feb 23 02:15:43 lab-sw01-admin-vdc-1 %SYSMGR-SLOT8-5-MODULE_ONLINE: System Manager has received notification of local module becoming online.
2022 Feb 23 02:15:44 lab-sw01-admin-vdc-1 %PLATFORM-5-MOD_STATUS: Module 8 current-status is MOD_STATUS_ONLINE/OK
2022 Feb 23 02:16:38 lab-sw01-admin-vdc-1 %MODULE-5-STANDBY_SUP_OK: Supervisor 5 is standby
註:所有模組均聯機(正常),模組6為活動(活動*)SUP,模組5為高可用性備用(即ha-standby)SUP。
步驟 5.所有模組均聯機後,重複步驟1。並驗證所有連線都已標準化。
修訂 | 發佈日期 | 意見 |
---|---|---|
1.0 |
24-Mar-2022 |
初始版本 |