此产品的文档集力求使用非歧视性语言。在本文档集中,非歧视性语言是指不隐含针对年龄、残障、性别、种族身份、族群身份、性取向、社会经济地位和交叉性的歧视的语言。由于产品软件的用户界面中使用的硬编码语言、基于 RFP 文档使用的语言或引用的第三方产品使用的语言,文档中可能无法确保完全使用非歧视性语言。 深入了解思科如何使用包容性语言。
思科采用人工翻译与机器翻译相结合的方式将此文档翻译成不同语言,希望全球的用户都能通过各自的语言得到支持性的内容。 请注意:即使是最好的机器翻译,其准确度也不及专业翻译人员的水平。 Cisco Systems, Inc. 对于翻译的准确性不承担任何责任,并建议您总是参考英文原始文档(已提供链接)。
本文档介绍Nexus 7000(N7K)硬件的故障排除技术。
此命令显示交换机上的风扇模块状态。
SITE1-AGG1# show environment fan Fan: ------------------------------------------------------ Fan Model Hw Status ------------------------------------------------------ Fan1(sys_fan1) N7K-C7010-FAN-S 1.1 Ok Fan2(sys_fan2) N7K-C7010-FAN-S 1.1 Ok Fan3(fab_fan1) N7K-C7010-FAN-F 1.1 Ok Fan4(fab_fan2) N7K-C7010-FAN-F 1.1 Ok Fan_in_PS1 -- -- Ok Fan_in_PS2 -- -- Ok Fan_in_PS3 -- -- Shutdown Fan Zone Speed: Zone 1: 0x78 Zone 2: 0x58 Fan Air Filter : Present
风扇状态可以是正常、故障或缺失。
“Fan module removed. Fan module has been absent for 120 seconds"
此命令显示交换机上已安装的电源、电源使用摘要和电源状态。
提供了命令和示例输出。
SITE1-AGG1# show environment power Power Supply: Voltage: 50 Volts Power Actual Total Supply Model Output Capacity Status (Watts ) (Watts ) ------- ------------------- ----------- ----------- -------------- 1 N7K-AC-6.0KW 1179 W 6000 W Ok 2 N7K-AC-6.0KW 1117 W 6000 W Ok 3 N7K-AC-6.0KW 0 W 0 W Shutdown Actual Power Module Model Draw Allocated Status (Watts ) (Watts ) ------- ------------------- ----------- ----------- -------------- 1 N7K-M148GT-11 N/A 400 W Powered-Up 3 N7K-M132XP-12 N/A 750 W Powered-Up 4 N7K-F132XP-15 318 W 385 W Powered-Up 5 N7K-SUP1 N/A 210 W Powered-Up 6 N7K-SUP1 N/A 210 W Powered-Up 10 N7K-M132XP-12L 535 W 750 W Powered-Up Xb1 N7K-C7010-FAB-1 N/A 80 W Powered-Up Xb2 N7K-C7010-FAB-1 N/A 80 W Powered-Up Xb3 N7K-C7010-FAB-1 N/A 80 W Powered-Up Xb4 xbar N/A 80 W Absent Xb5 xbar N/A 80 W Absent fan1 N7K-C7010-FAN-S 133 W 720 W Powered-Up fan2 N7K-C7010-FAN-S 133 W 720 W Powered-Up fan3 N7K-C7010-FAN-F 12 W 120 W Powered-Up fan4 N7K-C7010-FAN-F 12 W 120 W Powered-Up N/A - Per module power not available Power Usage Summary: -------------------- Power Supply redundancy mode (configured) PS-Redundant Power Supply redundancy mode (operational) Non-Redundant Total Power Capacity (based on configured mode) 12000 W Total Power of all Inputs (cumulative) 12000 W Total Power Output (actual draw) 2296 W Total Power Allocated (budget) 4785 W Total Power Available for additional modules 7215 W
电源状态可以是以下其中一种:
电源故障:
每个电源都有一个指示电源输出状态的LED。此LED由电源直接控制,红色表示电源故障。扫描系统日志时,可能会显示有关电源故障和恢复的交替消息,进一步指示电源相关问题。
机箱中的每个卡至少有两个温度传感器。每个温度传感器都配置有次要和主要阈值。此命令的输出示例显示如何从交换机检索温度信息:
SITE1-AGG1# show environment temperature Temperature: -------------------------------------------------------------------- Module Sensor MajorThresh MinorThres CurTemp Status (Celsius) (Celsius) (Celsius) -------------------------------------------------------------------- 1 Crossbar(s5) 105 95 46 Ok 1 CTSdev4 (s9) 115 105 56 Ok 1 CTSdev5 (s10) 115 105 57 Ok 1 CTSdev7 (s12) 115 105 56 Ok 1 CTSdev9 (s14) 115 105 53 Ok 1 CTSdev10(s15) 115 105 53 Ok 1 CTSdev11(s16) 115 105 52 Ok 1 CTSdev12(s17) 115 105 51 Ok 1 QEng1Sn1(s18) 115 105 51 Ok 1 QEng1Sn2(s19) 115 105 50 Ok 1 QEng1Sn3(s20) 115 105 48 Ok 1 QEng1Sn4(s21) 115 105 48 Ok 1 L2Lookup(s22) 120 110 47 Ok 1 L3Lookup(s23) 120 110 54 Ok 3 Crossbar(s5) 105 95 50 Ok 3 QEng1Sn1(s12) 115 110 69 Ok 3 QEng1Sn2(s13) 115 110 67 Ok 3 QEng1Sn3(s14) 115 110 66 Ok 3 QEng1Sn4(s15) 115 110 67 Ok 3 QEng2Sn1(s16) 115 110 70 Ok 3 QEng2Sn2(s17) 115 110 67 Ok 3 QEng2Sn3(s18) 115 110 66 Ok 3 QEng2Sn4(s19) 115 110 67 Ok 3 L2Lookup(s27) 115 105 51 Ok 3 L3Lookup(s28) 120 110 64 Ok 4 Crossbar1(s1) 105 95 69 Ok 4 Crossbar2(s2) 105 95 52 Ok 4 L2dev1(s3) 105 95 37 Ok 4 L2dev2(s4) 105 95 43 Ok 4 L2dev3(s5) 105 95 45 Ok 4 L2dev4(s6) 105 95 45 Ok 4 L2dev5(s7) 105 95 40 Ok 4 L2dev6(s8) 105 95 41 Ok 4 L2dev7(s9) 105 95 42 Ok 4 L2dev8(s10) 105 95 40 Ok 4 L2dev9(s11) 105 95 38 Ok 4 L2dev10(s12) 105 95 38 Ok 4 L2dev11(s13) 105 95 38 Ok 4 L2dev12(s14) 105 95 37 Ok 4 L2dev13(s15) 105 95 34 Ok 4 L2dev14(s16) 105 95 33 Ok 4 L2dev15(s17) 105 95 33 Ok 4 L2dev16(s18) 105 95 32 Ok 5 Intake (s3) 60 42 24 Ok 5 EOBC_MAC(s4) 105 95 42 Ok 5 CPU (s5) 105 95 42 Ok 5 Crossbar(s6) 105 95 47 Ok 5 Arbiter (s7) 110 100 55 Ok 5 CTSdev1 (s8) 115 105 44 Ok 5 InbFPGA (s9) 105 95 43 Ok 5 QEng1Sn1(s10) 115 105 48 Ok 5 QEng1Sn2(s11) 115 105 46 Ok 5 QEng1Sn3(s12) 115 105 44 Ok 5 QEng1Sn4(s13) 115 105 44 Ok 6 Intake (s3) 60 42 24 Ok 6 EOBC_MAC(s4) 105 95 40 Ok 6 CPU (s5) 105 95 36 Ok 6 Crossbar(s6) 105 95 45 Ok 6 Arbiter (s7) 110 100 52 Ok 6 CTSdev1 (s8) 115 105 43 Ok 6 InbFPGA (s9) 105 95 43 Ok 6 QEng1Sn1(s10) 115 105 53 Ok 6 QEng1Sn2(s11) 115 105 51 Ok 6 QEng1Sn3(s12) 115 105 48 Ok 6 QEng1Sn4(s13) 115 105 48 Ok 10 Crossbar(s5) 105 95 46 Ok 10 QEng1Sn1(s12) 115 110 65 Ok 10 QEng1Sn2(s13) 115 110 62 Ok 10 QEng1Sn3(s14) 115 110 64 Ok 10 QEng1Sn4(s15) 115 110 65 Ok 10 QEng2Sn1(s16) 115 110 65 Ok 10 QEng2Sn2(s17) 115 110 63 Ok 10 QEng2Sn3(s18) 115 110 64 Ok 10 QEng2Sn4(s19) 115 110 65 Ok 10 L2Lookup(s27) 115 105 51 Ok 10 L3Lookup(s28) 120 110 71 Ok xbar-1 Intake (s2) 60 42 27 Ok xbar-1 Crossbar(s3) 105 95 55 Ok xbar-2 Intake (s2) 60 42 25 Ok xbar-2 Crossbar(s3) 105 95 49 Ok xbar-3 Intake (s2) 60 42 26 Ok xbar-3 Crossbar(s3) 105 95 47 Ok
进气传感器放置在气流进气处,是卡温度的最关键指示器。所有软件操作都基于进气传感器的严重温度违规。
这会导致系统日志消息、callhome事件和简单网络管理协议(SNMP)陷阱。此优先级1或2消息在系统日志中显示 — 模块1报告了严重温度警报(传感器指数1温度76)。
线路卡立即关闭,并显示此优先级为0的系统日志消息 — 模块1因严重温度警报而关闭。
冗余Supervisor立即关闭。这将导致切换或备用关闭,具体取决于违反阈值的特定Supervisor。显示此优先级0系统日志消息 — 模块1由于严重温度警报而关闭电源。
有时,温度传感器发生故障,无法访问。此情况不会执行任何明确的软件操作。此优先级4系统日志消息已打印 — 模块1温度传感器失败。
调试交换机/管理引擎级别重置/重新加载通常需要查看存储在管理引擎上非易失性随机访问存储器(NVRAM)上的调试/日志信息。NVRAM中有3种调试/日志信息,可能包含一些重要信息。
1.1重置原因
重置原因存储在每个Supervisor的Supervisor NVRAM中。每个Supervisor存储其自己的重置原因。交换机恢复后,可使用此CLI命令转储重置原因。提供了示例输出。
SITE1-AGG1# show system reset-reason ----- reset reason for Supervisor-module 5 (from Supervisor in slot 5) --- 1) No time Reason: Unknown Service: Version: 6.1(2) 2) No time Reason: Unknown Service: Version: 6.1(1) 3) At 246445 usecs after Wed Nov 7 21:26:59 2012 Reason: Reset triggered due to Switchover Request by User Service: SAP(93): Swover due to install Version: 6.1(2) 4) At 36164 usecs after Tue Nov 6 01:18:15 2012 Reason: Reset Requested by CLI command reload Service: Version: 5.2(1) ----- reset reason for Supervisor-module 5 (from Supervisor in slot 6) --- 1) At 939785 usecs after Wed Nov 7 22:28:36 2012 Reason: Reset due to upgrade Service: Version: 6.1(1) 2) At 687128 usecs after Thu Mar 29 18:06:34 2012 Reason: Reset of standby by active sup due to sysmgr timeout Service: Version: 6.0(2) 3) At 10012 usecs after Thu Mar 29 17:56:13 2012 Reason: Reset of standby by active sup due to sysmgr timeout Service: Version: 6.0(2) 4) At 210045 usecs after Thu Mar 29 17:45:51 2012 Reason: Reset of standby by active sup due to sysmgr timeout Service: Version: 6.0(2) ----- reset reason for Supervisor-module 6 (from Supervisor in slot 5) --- 1) At 50770 usecs after Wed Nov 7 21:12:19 2012 Reason: Reset due to upgrade Service: Version: 6.1(2) 2) At 434294 usecs after Mon Nov 5 22:10:16 2012 Reason: Reset due to upgrade Service: Version: 5.2(1) 3) At 518 usecs after Mon Nov 5 21:21:51 2012 Reason: Reset Requested by CLI command reload Service: Version: 5.2(7) 4) At 556934 usecs after Mon Nov 5 21:12:15 2012 Reason: Reset due to upgrade Service: Version: 5.2(1) ----- reset reason for Supervisor-module 6 (from Supervisor in slot 6) --- 1) No time Reason: Unknown Service: Version: 6.1(2) 2) At 462775 usecs after Wed Nov 7 22:38:44 2012 Reason: Reset triggered due to Switchover Request by User Service: SAP(93): Swover due to install Version: 6.1(1) 3) No time Reason: Unknown Service: Version: 6.1(2) 4) No time Reason: Unknown Service: Version: 5.2(1)
最后4个重置原因已保存并显示。重置原因包含:
有时会显示“未知”的重置原因。软件或软件控制范围之外未知的重置原因分类为未知。这些通常包括:
1.2 NVRAM系统日志
优先级为0、1和2的系统日志消息也记录到Supervisor的NVRAM中。交换机重新联机后,可使用此命令显示NVRAM中的系统日志消息。将显示命令和示例输出:
SITE1-AGG1# show log nvram 2012 Nov 17 05:59:51 SITE1-AGG1 %$ VDC-1 %$ %SYSMGR-STANDBY-2-LAST_CORE_BASIC_TRACE: : PID 15681 with message 'Core detected due to hwclock crash'. 2012 Nov 17 12:07:11 SITE1-AGG1 %$ VDC-1 %$ %CMPPROXY-2-LOG_CMP_UP: Connectivity Management processor(on module 5) is now UP 2012 Nov 17 12:07:56 SITE1-AGG1 %$ VDC-1 %$ %VDC_MGR-2-VDC_ONLINE: vdc 1 has come online 2012 Nov 17 12:07:58 SITE1-AGG1 %$ VDC-1 %$ %PLATFORM-2-PS_OK: Power supply 1 ok (Serial number DTM131000A4) 2012 Nov 17 12:07:58 SITE1-AGG1 %$ VDC-1 %$ %PLATFORM-2-PS_FANOK: Fan in Power supply 1 ok 2012 Nov 17 12:07:58 SITE1-AGG1 %$ VDC-1 %$ %PLATFORM-2-PS_OK: Power supply 2 ok (Serial number DTM140700HS) 2012 Nov 17 12:07:58 SITE1-AGG1 %$ VDC-1 %$ %PLATFORM-2-PS_FANOK: Fan in Power supply 2 ok 2012 Nov 17 12:07:58 SITE1-AGG1 %$ VDC-1 %$ %PLATFORM-2-PS_DETECT: Power supply 3 detected but shutdown (Serial number DTM1413004P) 2012 Nov 17 12:07:59 SITE1-AGG1 %$ VDC-1 %$ %PLATFORM-2-XBAR_DETECT: Xbar 1 detected (Serial number JAF1308ABCS) 2012 Nov 17 12:08:01 SITE1-AGG1 %$ VDC-1 %$ %PLATFORM-2-XBAR_DETECT: Xbar 2 detected (Serial number JAB120600NX) 2012 Nov 17 12:08:02 SITE1-AGG1 %$ VDC-1 %$ %PLATFORM-2-XBAR_DETECT: Xbar 3 detected (Serial number JAF1508AJHN) 2012 Nov 17 12:08:04 SITE1-AGG1 %$ VDC-1 %$ %PLATFORM-2-MOD_DETECT: Module 1 detected (Serial number JAB121602HP) Module-Type 10/100/1000 Mbps Ethernet Module Model N7K-M148GT-11 2012 Nov 17 12:08:04 SITE1-AGG1 %$ VDC-1 %$ %PLATFORM-2-MOD_PWRUP: Module 1 powered up (Serial number JAB121602HP) 2012 Nov 17 12:08:11 SITE1-AGG1 %$ VDC-1 %$ %PLATFORM-2-MOD_DETECT: Module 3 detected (Serial number JAF1441BSED) Module-Type 10 Gbps Ethernet Module Model N7K-M132XP-12 2012 Nov 17 12:08:11 SITE1-AGG1 %$ VDC-1 %$ %PLATFORM-2-MOD_DETECT: Module 4 detected (Serial number JAF1542ABML) Module-Type 1/10 Gbps Ethernet Module Model N7K-F132XP-15 2012 Nov 17 12:08:12 SITE1-AGG1 %$ VDC-1 %$ %PLATFORM-2-MOD_PWRUP: Module 3 powered up (Serial number JAF1441BSED) 2012 Nov 17 12:08:12 SITE1-AGG1 %$ VDC-1 %$ %PLATFORM-2-MOD_PWRUP: Module 4 powered up (Serial number JAF1542ABML) 2012 Nov 17 12:08:15 SITE1-AGG1 %$ VDC-1 %$ %PLATFORM-2-MOD_DETECT: Module 10 detected (Serial number JAF1521BNMK) Module-Type 10 Gbps Ethernet XL Module Model N7K-M132XP-12L 2012 Nov 17 12:08:15 SITE1-AGG1 %$ VDC-1 %$ %PLATFORM-2-MOD_PWRUP: Module 10 powered up (Serial number JAF1521BNMK) 2012 Nov 17 12:08:30 SITE1-AGG1 %$ VDC-1 %$ %CMPPROXY-STANDBY-2-LOG_CMP_UP: Connectivity Management processor(on module 6) is now UP 2012 Nov 17 12:08:33 SITE1-AGG1 %$ VDC-1 %$ %PLATFORM-2-FANMOD_FAN_OK: Fan module 1 (Fan1(sys_fan1) fan) ok 2012 Nov 17 12:08:33 SITE1-AGG1 %$ VDC-1 %$ %PLATFORM-2-FANMOD_FAN_OK: Fan module 2 (Fan2(sys_fan2) fan) ok 2012 Nov 17 12:08:33 SITE1-AGG1 %$ VDC-1 %$ %PLATFORM-2-FANMOD_FAN_OK: Fan module 3 (Fan3(fab_fan1) fan) ok 2012 Nov 17 12:08:33 SITE1-AGG1 %$ VDC-1 %$ %PLATFORM-2-FANMOD_FAN_OK: Fan module 4 (Fan4(fab_fan2) fan) ok 2012 Nov 17 12:11:40 SITE1-AGG1 %$ VDC-1 %$ %VDC_MGR-2-VDC_ONLINE: vdc 2 has come online 2012 Nov 17 12:12:31 SITE1-AGG1 %$ VDC-1 %$ %VDC_MGR-2-VDC_ONLINE: vdc 3 has come online 2012 Nov 17 12:13:21 SITE1-AGG1 %$ VDC-1 %$ %VDC_MGR-2-VDC_ONLINE: vdc 4 has come online 2012 Nov 17 13:10:33 SITE1-AGG1 %$ VDC-1 %$ %PLATFORM-2-MOD_TEMPMINALRM: Xbar-1 reported minor temperature alarm. Sensor=2 Temperature=43 MinThreshold=42 2012 Nov 17 19:56:35 SITE1-AGG1 %$ VDC-1 %$ %PLATFORM-2-MOD_TEMPOK: Xbar-1 recovered from minor temperature alarm. Sensor=2 Temperature=41 MinThreshold=42
扫描NVRAM系统日志可能提供有关导致交换机/Supervisor重新加载/重置的特定故障的一些详细信息。
1.3模块例外日志
模块异常日志是每个模块上所有错误和异常情况的自动日志。有些例外是灾难性的,有些会部分影响模块中的某些端口,有些则用于警告。每个日志条目都有记录异常、异常级别、错误代码、受影响端口、时间戳的特定设备。异常日志存储在Supervisor的NVRAM中,并且可以使用此CLI命令显示。提供了示例输出。
SITE1-AGG1# show module internal exceptionlog ********* Exception info for module 1 ******** exception information --- exception instance 1 ---- Module Slot Number: 1 Device Id : 10 Device Name : eobc Device Errorcode : 0xc0005043 Device ID : 00 (0x00) Device Instance : 05 (0x05) Dev Type (HW/SW) : 00 (0x00) ErrNum (devInfo) : 67 (0x43) System Errorcode : 0x4042004d EOBC link failure Error Type : Warning PhyPortLayer : Ethernet Port(s) Affected : none DSAP : 0 (0x0) UUID : 0 (0x0) Time : Mon Nov 5 20:39:38 2012 (Ticks: 5098948A jiffies) exception information --- exception instance 2 ---- Module Slot Number: 1 Device Id : 10 Device Name : eobc Device Errorcode : 0xc0005047 Device ID : 00 (0x00) Device Instance : 05 (0x05) Dev Type (HW/SW) : 00 (0x00) ErrNum (devInfo) : 71 (0x47) System Errorcode : 0x4042004e EOBC heartbeat failure Error Type : Warning PhyPortLayer : Ethernet Port(s) Affected : none DSAP : 0 (0x0) UUID : 0 (0x0) Time : Mon Nov 5 20:39:37 2012 (Ticks: 50989489 jiffies)
例外日志提供排除错误和异常情况的关键信息。此处列出了一些设备ID。
#define DEV_LINECARD_CTRL 1 #define DEV_SAHARA_FPGA 2 #define DEV_RIVIERA_ASIC 3 #define DEV_LUXOR_ASIC 4 #define DEV_FRONTIER_U_ASIC 5 #define DEV_FRONTIER_D_ASIC 6 #define DEV_ALADDIN_ASIC 7 #define DEV_SSA_ASIC 8 #define DEV_MIRAGE_ASIC 9 #define DEV_EOBC_MAC 10 #define DEV_SUPERVISOR_CTRL 11 #define DEV_BELLAGIO_ASIC 12 #define DEV_SIBYTE 13 #define DEV_FLAMINGO 14 #define DEV_FATW_CTRL 15 #define DEV_MGMT_MAC 16 #define DEV_MOD_RDN_CTRL 17 #define DEV_MOD_ENV 18 #define DEV_GG_FPGA 19 #define DEV_BALLY_MAIN_BOARD 20 #define DEV_BALLY_DAUGHTER_CARD 21 #define DEV_LOCAL_SSO_ASIC 22 #define DEV_REMOTE_SSO_ASIC 23 #define DEV_ID_UD_FIX_FPGA 24 #define DEV_ID_PM_FPGA 25 // PM - Power Mngmnt #define DEV_ID_SUP_XBUS2 26 #define DEV_MARRIOTT_FPGA 27 #define DEV_REUSE_ME 28 #define DEV_GBIC 29 #define DEV_XGFC_FPGA 30 #define DEV_GNN_FPGA 31 #define DEV_SIBYTE_MEM_EPLD 32 #define DEV_BATTERY 33 #define DEV_IDE_DISK 45 #define DEV_XCVR 46 #define DEV_LINECARD 48 #define DEV_TEMP_SENSOR 49 #define DEV_HIFN_COMP 50 #define DEV_X2 51
在多层数据交换机(MDS)机箱中,管理引擎模块的安装方式与线卡模块略有不同。当系统中存在两个管理引擎并且系统已通电时,其中一个管理引擎将变为活动状态,另一个将变为备用状态。主用Supervisor启动和备用Supervisor启动不同,此处将讨论。
如果系统中没有活动管理引擎,启动的管理引擎将默认为活动管理引擎。名为“系统管理器”的流程负责按顺序在管理引擎上加载所有软件组件。平台管理器是在管理引擎上运行的首批软件组件之一。此组件将加载所有内核驱动程序,并与系统管理器进行握手。成功后,系统管理器将根据进程之间的内部依赖关系继续执行并启动其余进程。
从模块管理器的角度来看,Supervisor就像另一个线卡模块一样,有细微的差异。当平台管理器向模块管理器指示Supervisor处于UP状态时,模块管理器不会等待注册。相反,它会通知所有软件组件Supervisor已启动(也称为Sup插入序列)。 所有组件都将配置管理引擎。如果任何组件返回故障,将重新启动Supervisor。
如果系统中有活动Supervisor,则启动的Supervisor将默认为备用Supervisor状态。备用Supervisor需要镜像主用Supervisor的状态。这通过主用管理器上的“系统管理器”实现,即启动主用管理引擎状态到备用管理引擎的gsync(全局同步)。一旦备用管理引擎上的所有组件与主用管理引擎的组件同步,模块管理器就会收到备用管理引擎已启动的通知。
现在,模块管理器将继续操作,并通知主用管理引擎上的所有软件组件配置备用管理引擎(也称为备用管理引擎插入序列)。 在备用Sup插入序列期间,任何组件的任何错误都将导致备用Supervisor重新启动。
MDS在运行时维护大量调试信息。但是,每当主管重新启动时,大部分调试信息都会丢失。但是,所有关键信息都存储在非易失性RAM中,该RAM可用于重建故障。当主用Supervisor重新启动时,在重新启动之前,无法获取存储在其nvram中的信息。一旦Supervisor重新启动,以下命令可用于转储持久日志:
Switch# show logging nvram
Switch# show system reset-reason
Switch# show module internal exception-log
示例 1:主用Sup重新启动(由于Supervisor进程崩溃)
在本示例中,Supervisor进程崩溃(服务“xbar”),导致活动管理引擎重新启动。当Supervisor再次恢复运行时,存储在reset-reason中的信息会为Supervisor的重新启动提供明确的指示。
switch# show system reset-reason ----- reset reason for module 6 ----- 1) At 94009 usecs after Tue Sep 27 18:52:13 2005 Reason: Reset triggered due to HA policy of Reset Service: Service "xbar" Version: 2.1(2)
如果系统中有备用管理引擎,则备用管理引擎现在将成为主用管理引擎。在备用管理引擎上显示系统日志信息也会提供相同的信息(虽然不像“show system reset-reason”那样显式)。
Switch# show logging 2005 Sep 27 18:58:05 172.20.150.204 %SYSMGR-3-SERVICE_CRASHED: Service "xbar" (PID 1225) hasn't caught signal 9 (no core). 2005 Sep 27 18:58:06 172.20.150.204 %SYSMGR-3-SERVICE_CRASHED: Service "xbar" (PID 2349) hasn't caught signal 9 (no core). 2005 Sep 27 18:58:06 172.20.150.204 %SYSMGR-3-SERVICE_CRASHED: Service "xbar" (PID 2352) hasn't caught signal 9 (no core).
示例 2:活动Sup重新启动(由于运行时诊断失败)
在本例中,插槽6中的Supervisor处于活动状态,并且Supervisor上的仲裁器报告致命错误。当任何硬件设备报告致命错误时,包含该设备的模块将重新启动。在这种情况下,活动Supervisor将重新启动。如果有备用管理引擎,备用管理引擎将接管。备用管理引擎和异常日志上的系统日志消息将包含用于识别错误来源的信息。
Switch# show logging 2005 Sep 28 14:17:47 172.20.150.204 %XBAR-5-XBAR_STATUS_REPORT: Module 6 reported status for component 12 code 0x60a02. 2005 Sep 28 14:17:59 172.20.150.204 %PORT-5-IF_UP: Interface mgmt0 on slot 5 is up 2005 Sep 28 14:18:00 172.20.150.204 %CALLHOME-2-EVENT: SUP_FAILURE switch# show module internal exceptionlog module 6 ********* Exception info for module 6 ******** exception information --- exception instance 1 ---- device id: 12 device errorcode: 0x80000020 system time: (1127917068 ticks) Wed Sep 28 14:17:48 2005 error type: FATAL error Number Ports went bad: 1,2,3,4,5,6 exception information --- exception instance 2 ---- device id: 12 device errorcode: 0x00060a02 system time: (1127917067 ticks) Wed Sep 28 14:17:47 2005 error type: Warning Number Ports went bad: 1,2,3,4,5,6
此外,当重新启动的SUP再次联机时,“show system reset-reason”也将包含相关信息。在这种情况下,模块6(即活动sup)由Sap 48重新启动,错误代码为0x80000020。拥有此sap的进程可通过命令“show system internal mts sup sap 48 description”获取,该命令表示该进程是xbar-manager。
switch(standby)# show system reset-reason ----- reset reason for module 6 ----- 1) At 552751 usecs after Wed Sep 28 14:17:48 2005 Reason: Reset Requested due to Fatal Module Error Service: lcfail:80000020 sap:48 node:060 Version: 2.1(2)
示例 3:备用管理引擎无法联机
在本例中,活动管理引擎已启动并正在运行,备用管理引擎已插入系统。但show module不表示模块已启动。
switch# show module Mod Ports Module-Type Model Status --- ----- -------------------------------- ------------------ ------------ 5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active * 8 8 IP Storage Services Module powered-dn Mod Sw Hw World-Wide-Name(s) (WWN) --- ----------- ------ -------------------------------------------------- 5 2.1(2) 1.1 -- Mod MAC-Address(es) Serial-Num --- -------------------------------------- ---------- 5 00-0b-be-f7-4d-1c to 00-0b-be-f7-4d-20 JAB070307XG
但是,如果您登录备用管理引擎的控制台,则表明它是备用的。
runlog>telnet sw4-ts 2004 Trying 172.22.22.55... Connected to sw4-ts.cisco.com (172.22.22.55). Escape character is '^]'. MDS Switch login: admin Password: Cisco Storage Area Networking Operating System (SAN-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2005, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained herein are owned by other third parties and are used and distributed under license. Some parts of this software are covered under the GNU Public License. A copy of the license is available at http://www.gnu.org/licenses/gpl.html. switch(standby)#
如前所述,当备用管理引擎插入系统时,主用管理引擎的所有组件的配置和状态都复制到备用(gsync)。 在此过程完成之前,活动Supervisor不会认为存在备用Supervisor。要验证此过程是否完成,您可以在活动Supervisor上发出以下命令。命令的输出表明正在进行同步(可能从未完成)。
switch# show system redundancy status Redundancy mode --------------- administrative: HA operational: None This supervisor (sup-1) ----------------------- Redundancy state: Active Supervisor state: Active Internal state: Active with HA standby Other supervisor (sup-2) ------------------------ Redundancy state: Standby Supervisor state: HA standby Internal state: HA synchronization in progress
发生这种情况的最可能原因是,如果备用上的一个软件组件无法将其状态与活动Supervisor同步。要验证哪些进程未同步,您可以在活动Supervisor上发出此命令,并且输出表明许多软件组件尚未完成gsync。
switch# show system internal sysmgr gsyncstats Name Gsync done Gsync time(sec) ---------------- ---------- ------------- aaa 1 0 ExceptionLog 1 0 platform 1 1 radius 1 0 securityd 1 0 SystemHealth 1 0 tacacs 0 N/A acl 1 0 ascii-cfg 1 1 bios_daemon 0 N/A bootvar 1 0 callhome 1 0 capability 1 0 cdp 1 0 cfs 1 0 cimserver 1 0 cimxmlserver 0 N/A confcheck 1 0 core-dmon 1 0 core-client 0 N/A device-alias 1 0 dpvm 0 N/A dstats 1 0 epld_upgrade 0 N/A epp 1 1
此外,查看备用管理引擎,我们发现xbar软件组件已重新启动23次。这似乎是备用设备未启动的最可能原因。
switch(standby)# show system internal sysmgr service all Name UUID PID SAP state Start count ---------------- ---------- ------ ----- ----- ----------- aaa 0x000000B5 1458 111 s0009 1 ExceptionLog 0x00000050 [NA] [NA] s0002 None platform 0x00000018 1064 39 s0009 1 radius 0x000000B7 1457 113 s0009 1 securityd 0x0000002A 1456 55 s0009 1 vsan 0x00000029 1436 15 s0009 1 vshd 0x00000028 1408 37 s0009 1 wwn 0x00000030 1435 114 s0009 1 xbar 0x00000017 [NA] [NA] s0017 23 xbar_client 0x00000049 1434 917 s0009 1
示例 3:备用管理引擎处于通电状态
在本示例中,备用sup插入插槽6。在active-sup上发出show module命令,显示备用sup处于通电状态。
switch# show module Mod Ports Module-Type Model Status --- ----- -------------------------------- ------------------ ------------ 5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active * 6 0 Supervisor/Fabric-1 powered-up 8 8 IP Storage Services Module powered-dn Mod Sw Hw World-Wide-Name(s) (WWN) --- ----------- ------ -------------------------------------------------- 5 2.1(2) 1.1 -- Mod MAC-Address(es) Serial-Num --- -------------------------------------- ---------- 5 00-0b-be-f7-4d-1c to 00-0b-be-f7-4d-20 JAB070307XG
在本示例中,show logging不提供任何有价值的信息,也不提供show module internal exception-log。但是,由于给定模块的所有状态转换都存储在模块管理器中,因此我们可以查看模块管理器的状态转换来找出问题所在。内部状态转换包括:
Switch# show module internal event-history module 5 64) FSM:<ID(1): Slot 6, node 0x0601> Transition at 563504 usecs after Wed Sep 28 14:44:53 2005 Previous state: [LCM_ST_LC_NOT_PRESENT] Triggered event: [LCM_EV_PFM_MODULE_SUP_INSERTED] Next state: [LCM_ST_SUPERVISOR_INSERTED] 65) FSM:<ID(1): Slot 6, node 0x0601> Transition at 563944 usecs after Wed Sep 28 14:44:53 2005 Previous state: [LCM_ST_SUPERVISOR_INSERTED] Triggered event: [LCM_EV_START_SUP_INSERTED_SEQUENCE] Next state: [LCM_ST_CHECK_INSERT_SEQUENCE] 66) Event:ESQ_START length:32, at 564045 usecs after Wed Sep 28 14:44:53 2005 Instance:1, Seq Id:0x2710, Ret:success Seq Type:SERIAL 67) Event:ESQ_REQ length:32, at 564422 usecs after Wed Sep 28 14:44:53 2005 Instance:1, Seq Id:0x1, Ret:success [E_MTS_TX] Dst:MTS_SAP_MIGUTILS_DAEMON(949), Opc:MTS_OPC_LC_INSERTED(1081) 68) Event:ESQ_RSP length:32, at 566174 usecs after Wed Sep 28 14:44:53 2005 Instance:1, Seq Id:0x1, Ret:success [E_MTS_RX] Src:MTS_SAP_MIGUTILS_DAEMON(949), Opc:MTS_OPC_LC_INSERTED(1081) 69) Event:ESQ_REQ length:32, at 566346 usecs after Wed Sep 28 14:44:53 2005 Instance:1, Seq Id:0x2, Ret:success [E_MTS_TX] Dst:MTS_SAP_NTP(72), Opc:MTS_OPC_LC_INSERTED(1081) 70) Event:ESQ_RSP length:32, at 566635 usecs after Wed Sep 28 14:44:53 2005 Instance:1, Seq Id:0x2, Ret:success [E_MTS_RX] Src:MTS_SAP_NTP(72), Opc:MTS_OPC_LC_INSERTED(1081) 71) Event:ESQ_REQ length:32, at 566772 usecs after Wed Sep 28 14:44:53 2005 Instance:1, Seq Id:0x3, Ret:success [E_MTS_TX] Dst:MTS_SAP_XBAR_MANAGER(48), Opc:MTS_OPC_LC_INSERTED(1081) 73) Event:ESQ_RSP length:32, at 586418 usecs after Wed Sep 28 14:44:53 2005 Instance:1, Seq Id:0x3, Ret:(null) [E_MTS_RX] Src:MTS_SAP_XBAR_MANAGER(48), Opc:MTS_OPC_LC_INSERTED(1081) 74) FSM:<ID(1): Slot 6, node 0x0601> Transition at 586436 usecs after Wed Sep 28 14:44:53 2005 Previous state: [LCM_ST_CHECK_INSERT_SEQUENCE] Triggered event: [LCM_EV_LC_INSERTED_SEQ_FAILED] Next state: [LCM_ST_CHECK_REMOVAL_SEQUENCE] 75) Event:ESQ_START length:32, at 586611 usecs after Wed Sep 28 14:44:53 2005 Instance:1, Seq Id:0x2710, Ret:success Seq Type:SERIAL 76) Event:ESQ_REQ length:32, at 593649 usecs after Wed Sep 28 14:44:53 2005 Instance:1, Seq Id:0x1, Ret:success [E_MTS_TX] Dst:MTS_SAP_MIGUTILS_DAEMON(949), Opc:MTS_OPC_LC_REMOVED(1082) 77) Event:ESQ_RSP length:32, at 594854 usecs after Wed Sep 28 14:44:53 2005 Instance:1, Seq Id:0x1, Ret:success [E_MTS_RX] Src:MTS_SAP_MIGUTILS_DAEMON(949), Opc:MTS_OPC_LC_REMOVED(1082) 90) FSM:<ID(1): Slot 6, node 0x0601> Transition at 604447 usecs after Wed Sep 28 14:44:53 2005 Previous state: [LCM_ST_CHECK_REMOVAL_SEQUENCE] Triggered event: [LCM_EV_ALL_LC_REMOVED_RESP_RECEIVED] Next state: [LCM_ST_LC_FAILURE] 91) FSM:<ID(1): Slot 6, node 0x0601> Transition at 604501 usecs after Wed Sep 28 14:44:53 2005 Previous state: [LCM_ST_LC_FAILURE] Triggered event: [LCM_EV_LC_INSERTED_SEQ_FAILED] Next state: [LCM_ST_LC_FAILURE] 92) FSM:<ID(1): Slot 6, node 0x0601> Transition at 604518 usecs after Wed Sep 28 14:44:53 2005 Previous state: [LCM_ST_LC_FAILURE] Triggered event: [LCM_EV_SUPERVISOR_FAILURE] Next state: [LCM_ST_LC_NOT_PRESENT] Curr state: [LCM_ST_LC_NOT_PRESENT] switch#
查看索引92上的日志,表示管理引擎处于故障状态,触发事件为LCM_EV_LC_INSERTED_SEQ_FAILED。(插入序列失败)。 升级日志以找出插入序列失败的原因,请参阅在MTS_SAP_XBAR_MANAGER(索引73和索引74)的响应后插入序列失败。 这表示插入备用管理引擎时xbar配置有问题。通过查看故障组件(本例中为xbar组件)的内部日志可以完成更多调试。