排除Tomahawk和Lightspeed卡上的交换矩阵数据路径故障
目录
简介
本文档介绍在思科聚合服务路由器(ASR)9000系列运行期间出现的交换矩阵数据路径故障消息。
背景信息
该消息以以下格式显示：
- 路由器控制台上会显示警报，如下所示。
- 这意味着这些消息的环回路径在某处被破坏。
RP/0/RP0/CPU0:Oct 28 12:46:58.459 IST: pfm_node_rp[349]: %PLATFORM-DIAGS-3-PUNT_FABRIC_DATA_PATH_FAILED :
Set|online_diag_rsp[24790]|System Punt/Fabric/data Path Test(0x2000004)|failure threshold is 3,
(slot, NP) failed: (0/9/CPU0, 1) (0/9/CPU0, 3)
前面提到的0/9/CPU0上的NP1和NP3出现问题。
本文档面向想要了解错误消息以及发现问题时必须采取的操作的任何人。
基于Tomahawk的线路卡(LC)可用作服务边缘优化（增强型QoS）或数据包传输优化（基本QoS）LC。
- SE — 服务边缘优化
- TR — 优化数据包传输
4端口和8端口100千兆以太网LC以两种形式提供，支持LAN/WAN/OTN统一PHY CPAK端口或仅LAN PHY CPAK端口。
这些LC基于Tomahawk:
- A9K-8X100G-LB-SE
- A9K-8X100G-LB-TR
- A9K-8X100GE-SE
- A9K-8X100GE-TR
- A9K-4X100GE-SE
- A9K-4X100GE-TR
- A9K-400G-DWDM-TR
- A9K-MOD400-SE
- A9K-MOD400-TR
- A9K-MOD200-SE
- A9K-MOD200-TR
- A9K-24X10GE-1G-SE
- A9K-24X10GE-1G-TR
- A9K-48X10GE-1G-SE
- A9K-48X10GE-1G-TR
- A99-12X100GE
- A99-8X100GE-SE
- A99-8X100GE-TR
注意：以A99-X开头的基于Tomahawk的LC部件号与Cisco ASR 9904、ASR 9906、ASR 9910、ASR 9912和ASR 9922机箱兼容。它们与Cisco ASR 9006和ASR 9010路由器不兼容。
基于光速的LC可用作服务边缘优化（增强型QoS）或数据包传输优化（基本QoS）LC。与基于Tomahawk的LC不同，并非每个LC型号都可用于 — SE和 — TR类型。
- SE — 服务边缘优化
- TR — 优化数据包传输
这些LC基于Lightspeed:
- A9K-16X100GE-TR
- A99-16X100GE-X-SE
- A99-32X100GE-TR
基于Lightspeed-Plus(LSP)的LC可用作服务边缘优化（增强型QoS）或数据包传输优化（基本QoS）LC。
这些LC基于LSP:
- A9K-4HG-FLEX-TR
- A9K-4HG-FLEX-SE
- A99-4HG-FLEX-TR
- A99-4HG-FLEX-SE
- A9K-8HG-FLEX-TR
- A9K-8HG-FLEX-SE
- A9K-20HG-FLEX-TR
- A9K-20HG-FLEX-SE
- A99-32X100GE-X-TR
- A99-32X100GE-X-SE
- A99-10X400GE-X-TR
- A99-10X400GE-X-SE
Punt交换矩阵诊断数据包路径
- 在路由处理器卡CPU上运行的诊断应用程序会定期注入发往每个网络处理器(NP)的诊断数据包。
- 诊断数据包在NP内环回，并重新注入源于该数据包的路由处理器卡CPU。
- 通过路由处理器卡上的诊断应用程序对每个NP执行唯一数据包的定期运行状况检查，可针对路由器运行期间数据路径中的任何功能错误发出警报。
- 必须注意，主用路由处理器和备用路由处理器上的诊断应用程序会定期为每个NP注入一个数据包，并维护每个NP的成功或失败计数。
- 每分钟将诊断数据包发送到NP(发送到每个虚拟队列接口(VQI)四次（总共四分钟/VQI），并在该NP的所有VQI上运行)。为了对此进行简要介绍，下面是一个示例：
假设该LC有四个NP，在线诊断必须执行所有NP（以了解它们是否正常 — 交换矩阵路径）。现在，每个NP可以各有20个VQI(0-19、20-39、40-59、60-79)。
在第一分钟内，联机诊断向每个NP发送一个数据包。
1 min : against VQI 0, 20, 40, 60 (to all 4 NPs)
2 min:""""""""
3 min: """""""
4 min : """""
5th min : against VQI 1, 21, 41, 61..
6 min : """"""""
所有VQI完成后，循环中重复此过程。
- 当达到丢弃的诊断数据包的阈值时，应用程序会在平台故障管理器(PFM)中引发警报。
RP/0/RP1/CPU0:AG2-2#show pfm location 0/RP1/CPU0
node: node0_RP0_CPU0
---------------------
CURRENT TIME: Apr 7 01:04:04 2022PFM TOTAL: 1 EMERGENCY/ALERT(E/A): 0 CRITICAL(CR): 0 ERROR(ER): 1
-------------------------------------------------------------------------------------------------
Raised Time |S#|Fault Name |Sev|Proc_ID|Dev/Path Name |Handle
--------------------+--+-----------------------------------+---+-------+--------------+----------
Apr 7 00:54:52 2022|0 |PUNT_FABRIC_DATA_PATH_FAILED |ER |10042 >>ID |System Punt/Fa|0x2000004
要收集有关PFM警报的所有信息，请捕获以下命令输出：
show pfm location all
show pfm trace location all
如果要查看有关特定进程所引发警报的详细信息，可以使用以下命令：
show pfm process name <process_name> location <location> >>> location where the PFM alarm is observed
高级LC架构
战斧LC
8x100G架构
战斧 — 8x100G LC
12 x 100G架构
战斧12*100G LC
光速LC
A9K-20HG-FLEX-SE/TR
A9K-20HG-FLEX-SE/TR
A99-32x100GE-X-SE/TR
A99-32x100GE-X-SE/TR
A9K-8HG-FLEX-SE/TR
LC架构
LC架构
虚拟输出队列和仲裁器
每个路由交换机处理器/交换机处理器(RSP/RP)有两个矩阵芯片，均由一个公共仲裁器控制（双RSP/RP意味着每个机箱具有弹性仲裁器）。只有活动RSP/RP上的仲裁器控制所有四个交换矩阵芯片（假设为双RSP）。但是，两个仲裁程序都会收到交换矩阵访问请求，以便了解整个系统在任何给定时间的状态，以便RSP/RP之间的故障转移可以立即进行。仲裁器之间没有keepalive，但RSP/RP具有复杂可编程逻辑器件(CPLD)ASIC（类似于FPGA），其功能之一是通过低级keepalive跟踪另一个RSP/RP状态，并确定活动仲裁器是什么。
每个交换矩阵互联ASIC都有一组VQI，这是代表系统中100G实体的一组队列（适用于Tomahawk）。每个100G实体（单个出口NP上的1x100G端口用入口NP中的单个100G VQI表示）具有多个优先级类。
每个VQI都有一组四个虚拟输出队列(VOQ)，用于不同的数据包优先级，其中三个用于ASR 9000转发架构。这些优先级对应优先级1和2，默认在入口QoS策略中。有两个严格优先级队列和一个正常队列（第四个队列用于组播，不用于单播转发）。
通常，默认队列在来自出口NP VQI的背压期间首先开始丢弃数据包。只有当出口网络处理单元(NPU)超载（提供比电路可处理的更多Bps或PPS）时，它才开始向入口LC/NP施加反压力。这表示在该入口LC上的交换矩阵接口ASIC(FIA)上出现VQI流停滞。
示例：
RP/0/RP0/CPU0:AG3_1#show controllers np ports all location 0/0/CPU0 >>> LC0 is installed in slot 2
Node: 0/0/CPU0:
----------------------------------------------------------------
NP Bridge Fia Ports
-- ------ --- ---------------------------------------------------
0 -- 0 TenGigE0/0/0/0/0 - TenGigE0/0/0/0/9, TenGigE0/0/0/1/0 - TenGigE0/0/0/1/9
1 -- 1 TenGigE0/0/0/2/0 - TenGigE0/0/0/2/9, HundredGigE0/0/0/3
2 -- 2 HundredGigE0/0/0/4 - HundredGigE0/0/0/5 >>>Below is the VQI assignment
3 -- 3 HundredGigE0/0/0/6 - HundredGigE0/0/0/7
RP/0/RP0/CPU0:AG3_1#sh controller fabric vqi assignment slot 2
slot = 2
fia_inst = 2 >>>FIA 2
VQI = 40 SPEED_100G
VQI = 41 SPEED_100G
VQI = 42 SPEED_100G
VQI = 43 SPEED_100G
VQI = 44 SPEED_100G
VQI = 45 SPEED_100G
VQI = 46 SPEED_100G
VQI = 47 SPEED_100G
VQI = 56 SPEED_100G
VQI = 57 SPEED_100G
VQI = 58 SPEED_100G
VQI = 59 SPEED_100G
VQI = 60 SPEED_100G
VQI = 61 SPEED_100G
VQI = 62 SPEED_100G
VQI = 63 SPEED_100G
当入口LC决定要将特定数据包发送到特定出口NPU时，入口LC上的修改(MDF)阶段将数据包封装到交换矩阵目标报头中。当FIA查看该“地址”时，它会检查VOQ以查找特定出口NPU/目标/LC，并查看是否有足够的可用带宽。当它准备将其出队到该LC时，入口FIA请求交换矩阵（仲裁器）对该目标LC授予授权。仲裁算法是QOS感知的，它确保P1类数据包优先于P2类等等。仲裁器将许可请求从入口FIA中继到出口FIA。
入口FIA可以将发往同一出口LC的多个数据包分组到所谓的超帧中。这意味着通过交换矩阵链路传输的不是本地帧/数据包，而是超帧。必须注意这一点，因为在恒定100pps的测试中，CLI可以显示仅报告50pps的交换矩阵计数器。这并不是丢包，只是意味着每个超帧中有两个数据包通过交换矩阵传输。超帧包括排序信息和目标FIA支持重新排序（数据包可在多个交换矩阵链路上“喷射”）。只有单播数据包会被放入超帧中，不会是组播数据包。
一旦出口LC收到数据包，授权就会返回到仲裁器。仲裁器每个VOQ的令牌数量有限。当仲裁器允许入口FIA向特定VOQ发送（超级）帧时，仅当出口FIA将帧传递到出口NP时，该令牌才返回到池。如果出口NP向出口FIA发出反压信号，则令牌仍然占用。这就是仲裁器最终在入口FIA中耗尽该VOQ的令牌的方式。发生这种情况时，入口FIA开始丢弃传入的数据包。反压的触发因素是出口NP中接收帧描述符(RFD)缓冲区的利用率级别。当NP微码处理数据包时，RFD缓冲区会保存数据包。处理数据包的功能越多，数据包在RFD缓冲区中的停留时间就越长。
- 入口FIA向所有机箱仲裁程序发出交换矩阵请求。
- 主动仲裁器会检查免费访问授权令牌，并在出现拥塞时处理其QoS算法。
- 从RSP上的本地仲裁者到活动仲裁者的信用机制。
- 主动仲裁器将交换矩阵授权令牌发送到入口FIA。
- 入口FIA在交换矩阵链路上均衡帧负载（超级）。
- 出口FIA将交换矩阵令牌返回到中央仲裁器。
最好提及RSP上从本地仲裁者到活动仲裁者的信用机制。另添加一节以涵盖仲裁器故障的可能情况（无需提及错误代码，但要了解仲裁器ASIC错误），以便在出现任何仲裁器问题且由于本地或中心仲裁器导致队列堆积而无法获得授权时进行查看。
虚拟输出队列概述
虚拟输出队列
发往不同出口NP的数据包会被放入不同的VOQ集中。一个NP上的拥塞不会阻止发往不同NP的数据包。
交换矩阵仲裁器图
交换矩阵仲裁器
交换矩阵互联
ASR9006和ASR9010交换矩阵互联
ASR9006和ASR9010交换矩阵互联
ASR9922交换矩阵互联
ASR9912同样支持10个LC和单个交换矩阵互联芯片。
ASR9922交换矩阵互联
ASR9922和ASR9912背板
ASR9922和ASR9912背板
在线诊断概述
- 联机诊断工具在LC和RP CPU上运行。
- 测试转发路径的诊断测试包括：
- 在主用和备用RP CPU上运行的PuntFabricDataPath测试，将诊断数据包发送到系统中的每个主用NP。活动RP发送。
- PuntFabricDataPath诊断数据包以单播形式发送，而备用数据包以组播形式发送。将响应数据包发送回始发RP CPU。
- 测试转发路径的诊断测试包括：
- LC内的NP环回测试。
- NPLoopback测试在每个LC CPU上运行，将诊断数据包发送到每个NP。将响应数据包发送回LC CPU。
对问题进行分类
此处的步骤提供了一些有关如何缩小与点路径故障相关问题的提示。 它们不需要以完全相同的顺序被遵循。
开始分类所需的信息
- 查找受影响的NP和LC:
show logging | inc “PUNT_FABRIC_DATA_PATH”
RP/0/RP1/CPU0:Oct 28 12:46:58.459 IST: pfm_node_rp[349]: %PLATFORM-DIAGS-3-PUNT_FABRIC_DATA_PATH_FAILED :
Set|online_diag_rsp[24790]|System Punt/Fabric/data Path Test(0x2000004)|failure threshold is 3, (slot, NP)
failed: (0/9/CPU0, 1) (0/9/CPU0, 3)
前面提到的0/9/CPU0上的NP1和NP3出现问题。
- 要查找机箱插槽，请输入
run nslot all命令。 - PFM警报
RP/0/RP1/CPU0:AG2-2#show pfm location 0/RP1/CPU0
node: node0_RP1_CPU0
---------------------
CURRENT TIME: Mar 25 12:11:29 2022
PFM TOTAL: 1 EMERGENCY/ALERT(E/A): 0 CRITICAL(CR): 0 ERROR(ER): 1
-------------------------------------------------------------------------------------------------
Raised Time |S#|Fault Name |Sev|Proc_ID|Dev/Path Name |Handle
--------------------+--+-----------------------------------+---+-------+--------------+----------
Mar 25 12:03:30 2022|1 |PUNT_FABRIC_DATA_PATH_FAILED |ER |8947 |System Punt/Fa|0x2000004
RP/0/RP1/CPU0:AG2-2#sh pfm process 8947 location 0/rp1/CPU0
node: node0_RP1_CPU0
---------------------
CURRENT TIME: Mar 25 12:12:36 2022
PFM TOTAL: 1 EMERGENCY/ALERT(E/A): 0 CRITICAL(CR): 0 ERROR(ER): 1
PER PROCESS TOTAL: 0 EM: 0 CR: 0 ER: 0
Device/Path[1 ]:Fabric loopbac [0x2000003 ] State:RDY Tot: 0
Device/Path[2 ]:System Punt/Fa [0x2000004 ] State:RDY Tot: 1
1 Fault Id: 432
Sev: ER
Fault Name: PUNT_FABRIC_DATA_PATH_FAILED
Raised Timestamp: Mar 25 12:03:30 2022
Clear Timestamp: Mar 25 12:07:32 2022
Changed Timestamp: Mar 25 12:07:32 2022
Resync Mismatch: FALSE
MSG: failure threshold is 3, (slot, NP) failed: (0/9/CPU0, 1) (0/9/CPU0, 3)
诊断数据包流程图
- DIAG消息数据包在RP和LC之间的路径（诊断数据包间隔为一分钟）。
RP上的数据包路径：
online_diags <===> SPP <===> Fabric <===> NP
LC上的数据包路径：
online_diags <===> SPP <===> Punt-switch <====> NP
- LC内的NP环回测试
每分钟都会将每个NP的DIAGS数据包从LC CPU注入到Punt交换机，并且所有数据包在NP处环回。它们根本不会连接到交换矩阵。转折点或半点是每个NP的微代码。
- 诊断发送路径：LC：联机诊断>注入> LC-NP >（循环）
- 诊断返回路径：LC-NP > Punt > online diagnostics: LC
诊断测试
RP/0/RP0/CPU0:AG2-2(admin)#show diagnostic content location <> >>> (in cXR)
RP/0/RP0/CPU0:AG2-2#show diagnostic content location <> >>> (in eXR)
A9K-8X100GE-L-SE 0/0/CPU0:
Diagnostics test suite attributes:
M/C/* - Minimal bootup level test / Complete bootup level test / NA
B/O/* - Basic ondemand test / not Ondemand test / NA
P/V/* - Per port test / Per device test / NA
D/N/* - Disruptive test / Non-disruptive test / NA
S/* - Only applicable to standby unit / NA
X/* - Not a health monitoring test / NA
F/* - Fixed monitoring interval test / NA
E/* - Always enabled monitoring test / NA
A/I - Monitoring is active / Monitoring is inactive
n/a - Not applicable
Test Interval Thre- Timeout
ID Test Name Attributes (day hh:mm:ss.ms shold ms )
==== ================================== ============ ================= ===== =====
1) CPUCtrlScratchRegister ----------> *B*N****A 000 00:01:00.000 3 n/a
2) DBCtrlScratchRegister -----------> *B*N****A 000 00:01:00.000 3 n/a
3) PortCtrlScratchRegister ---------> *B*N****A 000 00:01:00.000 3 n/a
4) PHYScratchRegister --------------> *B*N****A 000 00:01:00.000 3 n/a
5) NPULoopback ---------------------> *B*N****A 000 00:01:00.000 3 n/a
RP/0/RP0/CPU0:AG2-2#show diagnostic result location 0/0/CPU0
A9K-8X100GE-L-SE 0/0/CPU0:
Overall diagnostic result: PASS
Diagnostic level at card bootup: bypass
Test results: (. = Pass, F = Fail, U = Untested)
1 ) CPUCtrlScratchRegister ----------> .
2 ) DBCtrlScratchRegister -----------> .
3 ) PortCtrlScratchRegister ---------> .
4 ) PHYScratchRegister --------------> .
5 ) NPULoopback ---------------------> .
- 您可以手动详细测试此参数“inject diags packets”，如本示例中所述：
admin diag start location 0/x/cpu0 test NPULoopback (cXR)
RP/0/RP0/CPU0:AG3_1#diagnostic start location 0/0/CPU0 test NPULoopback >>> eXR
Fri May 13 06:53:00.902 EDT
RP/0/RP0/CPU0:AG3_1#show diagnostic res location 0/0/CPU0 test 5 detail >>> Here there are
multiple test 1-5 (check previous examples)
Test results: (. = Pass, F = Fail, U = Untested)
___________________________________________________________________________
5 ) NPULoopback ---------------------> .
Error code ------------------> 0 (DIAG_SUCCESS)
Total run count -------------> 67319
Last test execution time ----> Fri May 13 06:53:01 2022
First test failure time -----> n/a
Last test failure time ------> n/a
Last test pass time ---------> Fri May 13 06:53:01 2022
Total failure count ---------> 0
Consecutive failure count ---> 0
___________________________________________________________________________
- 检查NP是否正在接收/发送DIAG消息：
RP/0/RSP1/CPU0:AG2-2#show controllers np counters location | inc DIAG| LC_CPU
108 PARSE_RSP_INJ_DIAGS_CNT 25195 0 >>> total DIAG packets injected by Active+Stdby RP
904 PUNT_DIAGS_RSP_ACT 12584 0 >>> Loopbacks to Active RP
906 PUNT_DIAGS_RSP_STBY 12611 0 >>> Loopbacks to Stdby R
122 PARSE_LC_INJ_DIAGS_CNT 2618 0 >>> total DIAG packets injected by LC
790 DIAGS 12618 0 >>> total DIAG packets replied back to LC
16 MDF_TX_LC_CPU 3998218312 937 >>> a packet punted to LC CPU PARSE_RSP_INJ_DIAGS_CNT should match (PUNT_DIAGS_RSP_ACT + PUNT_DIAGS_RSP_STDBY)
PARSE_LC_INJ_DIAGS_CNT should match DIAGS
PARSE_XX_INJ_DIAGS_CNT should increment periodically.
- 检查软件数据包路径(SPP)是否正在发送/接收DIAG消息：
show spp sid stats location | inc DIAG 2. DIAG 35430
2. DIAG 35430
这些DIAG计数器已接收并发送。 它们始终可以在LC上匹配和递增。
- debug punt-inject l2-packets diag np 0 location 0/9/CPU0
示例日志：SPP正在发送和接收序列为no 0x4e packets的诊断数据包。
LC/0/1/CPU0:Jun 6 04:14:05.581 : spp[89]: Sent DIAG packet. NP:0 Slot:0 Seq:0x4e
LC/0/1/CPU0:Jun 6 04:14:05.584 : spp[89]: Rcvd DIAG packet. NP:0 Slot:0 Seq:0x4e
- 检查数据包路径中是否有丢包：
show drops all location
show drops all ongoing location
- 检查在线诊断调试（cXR中）：
在线诊断有助于多次检查发送/接收或丢失数据包的时间戳。 此类时间戳可以与SPP捕获比较以进行数据包关联。
admin debug diagnostic engineer location
admin debug diagnostic error location
注：输入 admin undebug all 命令，以禁用这些调试。
调试的输出示例：
RP/0/RSP0/CPU0:Mar 25 05:43:43.384 EST: online_diag_rsp[349]: Slot 1 has 4 NPs >>> Sending DIAG
messages to NPs on slot 1
RP/0/RSP0/CPU0:Mar 25 05:43:43.384 EST: online_diag_rsp[349]: PuntFabricDataPath: sending
a pak (seq 25), destination physical slot 1 (card type 0x3d02aa), NP 0, sfp=0xc6
RP/0/RSP0/CPU0:Mar 25 05:43:43.384 EST: online_diag_rsp[349]: PuntFabricDataPath: sending
a pak (seq 25), destination physical slot 1 (card type 0x3d02aa), NP 1, sfp=0xde
RP/0/RSP0/CPU0:Mar 25 05:43:43.384 EST: online_diag_rsp[349]: PuntFabricDataPath: sending
a pak (seq 25), destination physical slot 1 (card type 0x3d02aa), NP 2, sfp=0xf6
RP/0/RSP0/CPU0:Mar 25 05:43:43.384 EST: online_diag_rsp[349]: PuntFabricDataPath: sending
a pak (seq 25), destination physical slot 1 (card type 0x3d02aa), NP 3, sfp=0x10e
RP/0/RSP0/CPU0:Mar 25 05:43:43.888 EST: online_diag_rsp[349]: PuntFabricDataPath:
Time took to receive 22 pkts: 503922888 nsec, timeout value: 500000000 nsec
RP/0/RSP0/CPU0:Mar 25 05:43:43.888 EST: online_diag_rsp[349]: PuntFabricDataPath:
Received 22 packets, expected 24 => Some replies missed
RP/0/RSP0/CPU0:Mar 25 05:43:43.888 EST: online_diag_rsp[349]: PuntFabricDataPath:
Got a packet from physical slot 1, np 0
RP/0/RSP0/CPU0:Mar 25 05:43:43.888 EST: online_diag_rsp[349]: Successfully verified
a packet, seq. no.: 25
RP/0/RSP0/CPU0:Mar 25 05:43:43.888 EST: online_diag_rsp[349]: PuntFabricDataPath:
Got a packet from physical slot 1, np 2 <= Replies from NP1 and NP3 missing
RP/0/RSP0/CPU0:Mar 25 05:43:43.888 EST: online_diag_rsp[349]: Successfully verified
a packet, seq. no.: 25
RP/0/RSP0/CPU0:Mar 25 05:43:43.888 EST: online_diag_rsp[349]: PuntFabricDataPath:
Got a packet from physical slot 3, np 0
- 诊断跟踪：
RP/0/RP1/CPU0:AG2-2#show diagnostic trace location 0/rp1/CPU0
Fri Mar 25 12:16:40.866 IST
1765 wrapping entries (3136 possible, 2048 allocated, 0 filtered, 3503120 total)
Mar 16 02:40:21.641 diags/online/gold_error 0/RP1/CPU0 t7356 Failed to get ack: got 0 responses,
expected 1
Mar 16 02:40:36.490 diags/online/message 0/RP1/CPU0 t8947 My nodeid 0x120, rack# is 0, slot# 1,
board type = 0x100327
Mar 16 02:40:36.948 diags/online/message 0/RP1/CPU0 t8947 dev cnt=25, path cnt=3, shm loc for
dev alarms@0x7fd4f0bec000, path alarms@0x7fd4f0bec01c, path alarm data@0x7fd4f0bec028
Mar 16 02:40:37.022 diags/online/message 0/RP1/CPU0 t8947 Last rpfo time: 1647378637
Mar 24 06:03:27.479 diags/online/error 0/RP1/CPU0 2105# t9057 PuntFabricDataPath test error:
physical slot 11(LC# 9): expected np mask: 0x0000000f, actual: 0x0000000b, failed: 0x00000004
Mar 24 06:03:27.479 diags/online/error 0/RP1/CPU0 634# t9057 PuntFabricDataPath test failure detected,
detail in the form of (0-based) (slot, NP: count): (LC9,2: 13)
交换矩阵分类
- 交换矩阵运行状况（提供链路状态、统计信息、丢弃和警报的摘要）：
show controllers fabric health location <>
- 主干运行状况：
show controllers fabric health spine all
- 板载故障记录(OBFL)（在重新加载后，也可使用此功能）：
admin
sysadmin-vm:0_RP0# show logging onboard fabric location 0/0
- 检查入口LC FIA上的交换矩阵计数器：
show controllers fabric fia errors ingress location <>
show controllers fabric fia stats location
- 入口LC交叉开关（不适用于Trident和SIP-700）：
show controllers fabric crossbar statistics instance [0-1] location <>
- 出口LC交叉开关（不适用于Trident和SIP-700）：
show controllers fabric crossbar statistics instance [0-1] location <>
- 出口LC FIA:
show controllers fabric fia errors egress location <>
show controllers fabric fia stats location
- 主干统计信息：
show controllers fabric crossbar statistics instance [0-1] spine [0-6]
- 检查交换矩阵丢弃：
- 入口LC FIA:
show controllers fabric fia drops ingress location <>
- 出口LC FIA:
show controllers fabric fia drops egress location <>
- ASIC错误：
- LSP:
show controllers fabric crossbar asic-errors instance 0 location<>
show asic-errors fia <> all location <>
-
- 战斧：
show asic-errors fia <> all location <>
RP/0/RP0/CPU0:AG3_1#show controllers np fabric-counters all np0 location 0/0/CPU0
Node: 0/0/CPU0:
----------------------------------------------------------------
Egress fabric-to-bridge interface 2 counters for NP 0
INTERLAKEN_CNT_TX_BYTES 0x000073fc 23b6d99b
INTERLAKEN_CNT_TX_FRM_GOOD 0x000000ae a79d6612
INTERLAKEN_CNT_TX_FRM_BAD 0x00000000 00000000 >>> this is 0 which is good,
need to check if it is incremented
-------------------------------------------------------------
Egress fabric-to-bridge interface 3 counters for NP 0
INTERLAKEN_CNT_TX_BYTES 0x0004abdd fe02068d
INTERLAKEN_CNT_TX_FRM_GOOD 0x000005b8 089aac95
INTERLAKEN_CNT_TX_FRM_BAD 0x00000000 00000000
-------------------------------------------------------------
Node: 0/0/CPU0:
----------------------------------------------------------------
Ingress fabric-to-bridge interface 2 counters for NP 0
INTERLAKEN_CNT_RX_BYTES 0x0004aeb5 a4b9dbbe
INTERLAKEN_CNT_RX_FRM_GOOD 0x0000058e b7b91c15
INTERLAKEN_CNT_RX_FRM_BAD 0x00000000 00000000
INTERLAKEN_CNT_RX_BURST_CRC32_ERROR 0x00000000 00000000
INTERLAKEN_CNT_RX_BURST_CRC24_ERROR 0x00000000 00000000
INTERLAKEN_CNT_RX_BURST_SIZE_ERROR 0x00000000 00000000
-------------------------------------------------------------
Ingress fabric-to-bridge interface 3 counters for NP 0
INTERLAKEN_CNT_RX_BYTES 0x000094ce b8783f95
INTERLAKEN_CNT_RX_FRM_GOOD 0x000000f5 33cf9ed7
INTERLAKEN_CNT_RX_FRM_BAD 0x00000000 00000000
INTERLAKEN_CNT_RX_BURST_CRC32_ERROR 0x00000000 00000000
INTERLAKEN_CNT_RX_BURST_CRC24_ERROR 0x00000000 00000000
INTERLAKEN_CNT_RX_BURST_SIZE_ERROR 0x00000000 00000000
- 要验证FIA的链路状态，请执行以下操作：
show controllers fabric fia link-status location
RP/0/RP0/CPU0:AG3_1#show controllers fabric fia link-status location 0/0/CPU0
********** FIA-0 **********
Category: link-0
spaui link-0 Up >>> FIA to NP link
spaui link-1 Up >>> FIA to NP link
arb link-0 Up >>> Arbitor link
xbar link-0 Up >>> FIA to XBAR link
xbar link-1 Up >>> FIA to XBAR link
xbar link-2 Up >>> FIA to XBAR link
- 要验证XBAR的链路状态，请执行以下操作：
RP/0/RP0/CPU0:AG3_1#show controllers fabric crossbar link-status instance 0 lo 0/0/CPU0
Mon May 2 04:05:06.161 EDT
PORT Remote Slot Remote Inst Logical ID Status
======================================================
00 0/0/CPU0 01 2 Up
01 0/FC3 01 0 Up
02 0/FC3 00 0 Up
03 0/FC4 01 0 Up
04 0/FC2 01 0 Up
05 0/FC4 00 0 Up
06 0/FC2 00 0 Up
07 0/FC1 01 0 Up
10 0/FC1 00 0 Up
14 0/FC0 01 0 Up
15 0/FC0 00 0 Up
16 0/0/CPU0 02 0 Up
18 0/0/CPU0 02 2 Up
19 0/0/CPU0 02 1 Up
20 0/0/CPU0 03 2 Up
21 0/0/CPU0 03 1 Up
22 0/0/CPU0 03 0 Up
23 0/0/CPU0 00 2 Up
24 0/0/CPU0 00 1 Up
25 0/0/CPU0 00 0 Up
26 0/0/CPU0 01 0 Up
27 0/0/CPU0 01 1 Up
如果在LSP卡中观察到以下日志：
LC/0/3/CPU0:Jul 5 13:05:53.365 IST: fab_xbar[172]: %PLATFORM-CIH-5-ASIC_ERROR_THRESHOLD :
sfe[1]: An interface-err error has occurred causing packet drop transient.
ibbReg17.ibbExceptionHier.ibbReg17.ibbExceptionLeaf0.intIpcFnc0UcDataErr Threshold has been exceeded
此处的17*2有助于使用 show controllers fabric crossbar link-status instance 1 lo 0/3/CPU0 指令：
日志收集：
show platform
show inventory
show tech fabric
show tech np
show tech ethernet interface
show logging
show pfm location all
show pfm trace location <location id>
show controllers pm vqi location all
show hw-module fpd location all (cxr) / admin show hw-module fpd (exr)
show controllers fti trace <process-name> location <Card location>
admin show tech obfl
Cxr:
From Admin:
show logging onboard common location <>
show logging onboard error location <>
Exr:
From sysadmin/calvados:
show logging onboard fabric location <>
- 如果FIA中存在ASIC错误：
对于LS:
show controllers asic LS-FIA instance <instance> block <block_name> register-name <register_name> location <>
对于LSP:
show controllers asic LSP-FIA instance <instance> block <block_name> register-name <register_name> location <>
如果报告的错误如下所示：
LC/0/9/CPU0:Mar 1 05:12:25.474 IST: fialc[137]: %PLATFORM-CIH-5-ASIC_ERROR_THRESHOLD :
fia[3]: A link-err error has occurred causing performance loss persistent.
fnc2serdesReg1.fnc2serdesExceptionHier.fnc2serdesReg1.fnc2serdesExceptionLeaf0.
iNTprbsErrTxphyrdydropped6 Threshold has been exceeded
- 实例是FIA ASIC的实例编号。这里，block_name为“fnc2serdesReg1”，register_name为“fnc2serdesExceptionLeaf0”。
- 如果LC/RSP XBAR上出现ASIC错误：
show controllers asic SKB-XBAR instance <instance> block-name <block_name> register-name <register_name> location <>
如果报告的错误如下所示：
LC/0/7/CPU0:Mar 4 06:42:01.241 IST: fab_xbar[213]: %PLATFORM-CIH-5-ASIC_ERROR_THRESHOLD :
sfe[0]: An interface-err error has occurred causing packet drop transient.
ibbReg11.ibbExceptionHier.ibbReg11.ibbExceptionLeaf0.intIpcFnc1UcDataErr Threshold has been exceeded
- 实例是SFE/XBAR ASIC的编号。此处，"0" block_name为"ibbReg11",register_name为"ibbExceptionLeaf0"。
- 如果在FC XBAR上报告ASIC错误：
show controllers asic FC2-SKB-XBAR instance <instance> block-name <block_name> register-name <register_name> location
如果报告的错误如下所示：
RP/0/RP0/CPU0:Mar 4 06:41:14.398 IST: fab_xbar_sp3[156]: %PLATFORM-CIH-3-ASIC_ERROR_SPECIAL_HANDLE_THRESH :
fc3xbar[1]: A link-err error has occurred causing packet drop transient.
cflReg17.cflExceptionHier.cflReg17.cflExceptionLeaf4.intCflPal1RxAlignErrPktRcvd Threshold has been exceeded
那么ASIC是“FC3-SKB-XBAR”实例是SFE/XBAR ASIC的编号。此处是“1”，均来自“fc3xbar[1]” block_name为“cflReg17”，register_name为“cflExceptionLeaf4”。
示例：
RP/0/RSP0/CPU0: AG2-10#sh logging | i ASIC
RP/0/RSP0/CPU0:May 11 20:48:57.658 IST: fab_xbar[184]: %PLATFORM-CIH-5-ASIC_ERROR_THRESHOLD :
sfe[0]: An interface-err error has occurred causing packet drop transient.
ibbReg13.ibbExceptionHier.ibbReg13.ibbExceptionLeaf0.intIpcFnc0UcDataErr Threshold has been exceeded
RP/0/RSP0/CPU0: AG2-10#sh controllers fabric crossbar link-status instance 0 location 0/rsp0/CPU0
PORT Remote Slot Remote Inst Logical ID Status
======================================================
04 0/0/CPU0 00 1 Up
06 0/0/CPU0 00 0 Up
08 0/7/CPU0 00 1 Up
10 0/7/CPU0 00 0 Up
24 0/2/CPU0 00 0 Up
26 0/2/CPU0 00 1 Up
>>> ibbReg13 >> 13*2 = 26 SO IT IS POINTING TO LC2 – IN THIS CASE YOU CAN DO OIR TO RECOVER THE ASIC ERROR
40 0/RSP0/CPU0 00 0 Up
RP/0/RSP0/CPU0: AG2-10#show controllers asic SKB-XBAR instance 0 block-name ibbReg13 register-name ibbExceptionLeaf0 location 0/RSP0/CPU0
address name value
0x00050d080 SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_int1Stat 0x00000000 (4 bytes)
address name value
0x00050d084 SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_int1StatRw1s 0x00000000 (4 bytes)
address name value
0x00050d088 SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_int1Enable 0xfffffffb (4 bytes)
address name value
0x00050d08c SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_int1First 0x00000000 (4 bytes)
address name value
0x00050d090 SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_int2Stat 0x00000c50 (4 bytes)
address name value
0x00050d094 SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_int2StatRw1s 0x00000c50 (4 bytes)
address name value
0x00050d098 SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_int2Enable 0x00000000 (4 bytes)
address name value
0x00050d09c SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_int2First 0x00000000 (4 bytes)
address name value
0x00050d0a0 SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_haltEnable 0x00000000 (4 bytes)
address name value
0x00050d0a4 SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_fault 0x00000000 (4 bytes)
address name value
0x00050d0a8 SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_intMulti 0x00000840 (4 bytes)
address name value
0x00050d0ac SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_leaf 0x00000000 (4 bytes)
RP/0/RSP0/CPU0:AG2-10#
仲裁程序故障分类
要检查链路状态，请执行以下操作：
RP/0/RSP0/CPU0:AG2-10#sho controllers fabric arbiter link-status location 0/1/$
Port Remote Slot Remote Elem Remote Inst Status
=======================================================
00 0/1/CPU0 FIA 0 Up
01 0/1/CPU0 FIA 1 Up
24 0/RSP0/CPU0 ARB 0 Up
25 0/RSP1/CPU0 ARB 0 Up
要检查VQI可用性，请执行以下操作：
RP/0/RP0/CPU0:AG3_1#sh controllers fabric vqi assignment all
Current mode: Highbandwidth mode - 2K VQIs
Node Number of VQIs
----------------------------
0/0/CPU0 80
0/1/CPU0 40
0/2/CPU0 48
0/3/CPU0 80
0/5/CPU0 80
0/7/CPU0 80
0/12/CPU0 64
RP*/RSP* 8
----------------------------
In Use = 480
Available = 1568
检查分配给VQI的速度：
RP/0/RP0/CPU0:AG3_1#sh controller fabric vqi assignment slot 7
Thu May 12 07:58:59.897 EDT
slot = 7
fia_inst = 0
VQI = 400 SPEED_100G
VQI = 401 SPEED_100G
VQI = 402 SPEED_100G
VQI = 403 SPEED_100G
VQI = 404 SPEED_100G
VQI = 405 SPEED_100G
VQI = 406 SPEED_100G
slot = 7
fia_inst = 1
VQI = 416 SPEED_40G
VQI = 417 SPEED_40G
VQI = 418 SPEED_40G
VQI = 419 SPEED_40G
VQI = 420 SPEED_100G
如果观察到FIA上的任何尾部丢弃，请检查以下步骤：
检查VQI中的队列深度：
RP/0/RP0/CPU0:AG3_1#show controllers fabric fia q-depth location 0/0/CPU0
Thu May 12 08:00:42.186 EDT
********** FIA-0 **********
Category: q_stats_a-0
Voq ddr pri Cellcnt Slot_FIA_NP
28 0 2 2 LC0_1_1
********** FIA-0 **********
Category: q_stats_b-0
Voq ddr pri Cellcnt Slot_FIA_NP
********** FIA-1 **********
Category: q_stats_a-1
Voq ddr pri Cellcnt Slot_FIA_NP
7 0 2 12342 LC0_0_0
>>> Here Packet count is high so we need to check for LC0 FIA0 NP0 (egress) is there any congestion
or any other issue in LC0 FIA0 or NP0
Here Pri = 2 is the default queue (BE) , Pri = 0 is P1 (Voice, real time) queue, Pri = 1 is P2
97 0 2 23 LC1_0_0
RP/0/RP0/CPU0:AG3_1#show controllers fabric vqi assignment slot 02
slot = 2
fia_inst = 0
VQI = 0 SPEED_10G
VQI = 1 SPEED_10G
VQI = 2 SPEED_10G
VQI = 3 SPEED_10G
VQI = 4 SPEED_10G
VQI = 5 SPEED_10G
VQI = 6 SPEED_10G
VQI = 7 SPEED_10G
VQI的端口映射详细信息：
RP/0/RP0/CPU0:AG3_1#show controllers pm vqi location 0/0/CPU0
Platform-manager VQI Assignment Information
Interface Name | ifh Value | VQI | NP#
--------------------------------------------------
TenGigE0_0_0_0_1 | 0x4000680 | 1 | 0
TenGigE0_0_0_0_2 | 0x40006c0 | 2 | 0
TenGigE0_0_0_0_3 | 0x4000700 | 3 | 0
TenGigE0_0_0_0_4 | 0x4000740 | 4 | 0
TenGigE0_0_0_0_5 | 0x4000780 | 5 | 0
TenGigE0_0_0_0_6 | 0x40007c0 | 6 | 0
TenGigE0_0_0_0_7 | 0x4000800 | 7 | 0
RP/0/RP0/CPU0:AG3_1#show controllers pm interface tenGigE 0/0/0/0/7
Ifname(1): TenGigE0_0_0_0_7, ifh: 0x4000800 :
iftype 0x1e
egress_uidb_index 0x12, 0x0, 0x0, 0x0
ingress_uidb_index 0x12, 0x0, 0x0, 0x0
port_num 0x0
subslot_num 0x0
ifsubinst 0x0
ifsubinst port 0x7
phy_port_num 0x7
channel_id 0x0
channel_map 0x0
lag_id 0x7e
virtual_port_id 0xa
switch_fabric_port 7 >>> VQI matching for the ports
in_tm_qid_fid0 0x38001e
in_tm_qid_fid1 0x0
in_qos_drop_base 0xa69400
out_tm_qid_fid0 0x1fe002
out_tm_qid_fid1 0xffffffff
np_port 0xd3
日志收集：
Show tech fabric
Show tech np
show controllers pm trace ?
async Platform manager async trace
creation Platform manager interface creation/deletion trace
error Platform manager error trace
information Platform manager information trace
init Platform manager init trace
other Platform manager common trace
stats Platform manager stats trace
NP故障分类
NP加载验证：
RP/0/RP0/CPU0:AG3_1#show controller np load all location 0/0/CPU0
Node: 0/0/CPU0:
----------------------------------------------------------------
Load Packet Rate
NP0: 2% utilization 3095766 pps
NP1: 3% utilization 5335675 pps
NP2: 0% utilization 498 pps
NP3: 0% utilization 1117 pps
端口映射：
RP/0/RP0/CPU0:AG3_1#show controllers np ports all location 0/0/CPU0
Node: 0/0/CPU0:
----------------------------------------------------------------
NP Bridge Fia Ports
-- ------ --- ---------------------------------------------------
0 -- 0 TenGigE0/0/0/0/0 - TenGigE0/0/0/0/9, TenGigE0/0/0/1/0 - TenGigE0/0/0/1/9
1 -- 1 TenGigE0/0/0/2/0 - TenGigE0/0/0/2/9, HundredGigE0/0/0/3
2 -- 2 HundredGigE0/0/0/4 - HundredGigE0/0/0/5
3 -- 3 HundredGigE0/0/0/6 - HundredGigE0/0/0/7
战斧
请注意，这是管理模式：
sysadmin-vm:0_RP0# show controller switch statistics location 0/LC0/LC-SW
Thu May 12 12:32:37.160 UTC+00:00
Rack Card Switch Rack Serial Number
--------------------------------------
0 LC0 LC-SW
Tx Rx
Phys State Drops/ Drops/
Port State Changes Tx Packets Rx Packets Errors Errors Connects To
----------------------------------------------------------------------------
0 Up 2 3950184361 3977756349 0 0 NP0
1 Up 2 0 0 0 0 NP0
8 Up 1 1319787462 209249871 0 0 LC CPU N0 P0
9 Up 1 3374323096 1819796660 0 0 LC CPU N0 P1
16 Up 2 2245174606 1089972811 0 0 NP1
17 Up 2 0 0 0 0 NP1
18 Up 2 65977 16543963 0 0 NP2
19 Up 2 0 0 0 0 NP2
32 Up 2 128588820 3904804720 0 0 NP3
33 Up 2 0 0 0 0 NP3
show asic-error np <> all loc <> >>> Ignore the macwrap errors as they are seen for every
interface flaps/ Execute 3-4 times to verify the drops increment
show controller np fast-drop <> loc <> >>> Execute 3-4 times to verify the drops increment
RP/0/RP0/CPU0:AG3_1#show controller np fast-drop np0 location 0/0/CPU0
Thu May 12 10:13:22.981 EDT
Node: 0/0/CPU0:
----------------------------------------------------------------
All fast drop counters for NP 0:
TenGigE0/0/0/1/0-TenGigE0/0/0/1/9:[Priority1] 0
TenGigE0/0/0/1/0-TenGigE0/0/0/1/9:[Priority2] 0
TenGigE0/0/0/1/0-TenGigE0/0/0/1/9:[Priority3] 0
TenGigE0/0/0/0/0-TenGigE0/0/0/0/9:[Priority1] 0
TenGigE0/0/0/0/0-TenGigE0/0/0/0/9:[Priority2] 0
TenGigE0/0/0/0/0-TenGigE0/0/0/0/9:[Priority3] 0
show controllers np punt-path-counters all HOST-IF-0 np<> location <>
[Check for IF_CNT_RX_FRM & IF_CNT_TX_FRM] >>> To check if diagnostic packets make it to the LC NP
Host CPU network port
光速
show asic-error np <> all loc <> >>> Ignore the macwrap errors as they are seen for every interface flap
RP/0/RP0/CPU0:AG3_1#sho asic-errors np 0 all location 0/5/CPU0
************************************************************
* 0_5_CPU0 *
************************************************************
************************************************************
* Single Bit Errors *
************************************************************
************************************************************
* Multiple Bit Errors *
************************************************************
************************************************************
* Parity Errors *
************************************************************
************************************************************
* Generic Errors *
************************************************************
ASR, ASR9K Lightspeed 20*100GE SE LC, 0/5/CPU0, npu[0]
Name : mphmacwrapReg1.mphmacwrapExceptionLeaf4.mphWrapIrqUmacIpInt82
Leaf ID : 0x2023e082
Error count : 1
Last clearing : Thu Apr 7 11:41:47 2022
Last N errors : 1
--------------------------------------------------------------
First N errors.
@Time, Error-Data
------------------------------------------
show controller np fast-drop <> loc <> >>> Execute 3-4 times to verify the drops increment
RP/0/RP0/CPU0:AG3_1#show controller np fast-drop np0 location 0/5/CPU0
Thu May 12 10:13:28.321 EDT
Node: 0/5/CPU0:
----------------------------------------------------------------
All fast drop counters for NP 0:
HundredGigE0_5_0_0[Crit] 0
HundredGigE0_5_0_0[HP] 0
HundredGigE0_5_0_0[LP2] 0
HundredGigE0_5_0_0[LP1] 0
HundredGigE0_5_0_0[Crit+HP_OOR] 0
HundredGigE0_5_0_0[LP2+LP1_OOR] 0
HundredGigE0_5_0_1[Crit] 0
HundredGigE0_5_0_1[HP] 0
HundredGigE0_5_0_1[LP2] 0
HundredGigE0_5_0_1[LP1] 0
HundredGigE0_5_0_1[Crit+HP_OOR] 0
请注意，这是管理模式：
sysadmin-vm:0_RP0# show controller switch statistics location 0/LC5/LC-SW >>> Execute 3-4
times to verify the errors increment
Rack Card Switch Rack Serial Number
--------------------------------------
0 LC5 LC-SW
Tx Rx
Phys State Drops/ Drops/
Port State Changes Tx Packets Rx Packets Errors Errors Connects To
-----------------------------------------------------------------------------
0 Up 4 1456694749 329318054 0 4 CPU -- EOBC
1 Up 2 21 23 0 0 CPU -- flexE
2 Up 4 1063966999 87683758 0 0 CPU -- PUNT
3 Up 4 885103800 3021484524 0 0 CPU -- BFD
4 Up 3 329319167 1456700372 0 0 RP0
5 Up 3 0 0 0 0 RP1
6 Up 1 11887785 2256 0 0 IPU 0
7 Up 1 0 1086 0 0 IPU 1
9 Up 4 74028034 3025657779 0 0 NP0
10 Up 4 5 0 0 0 NP0
11 Down 1 0 0 0 0 PHY0 -- flexE
12 Up 4 264928 264929 0 0 NP1
13 Up 2 5 0 0 0 NP1
14 Down 1 0 0 0 0 PHY1 -- flexE
15 Up 4 1516538834 1159586563 0 0 NP2
日志收集：
show tech np
show tech fabric
show asic-errors fia trace all location <>
- 在eXR中，收集np_datalog:
RP/0/RP0/CPU0:AG3_1#run chvrf 0 ssh lc0_xr
LC : [one time capture]
show_np -e <> -d npdatalog [<> should be the affected NP]
Path where NP datalogs is saved : /misc/scratch/np/NPdatalog_0_0_CPU0_np0_prm__20220512-105332.txt.gz
LC : 5 to 10 times
show_np -e <> -d pipeline [<> should be the affected NP]
- 对于LSP上的NP Init故障：
RP/0/RP0/CPU0:AG2-2#show controllers np ports all location 0/6/CPU0
Node: 0/6/CPU0:
----------------------------------------------------------------
NP Bridge Fia Ports
-- ------ --- ---------------------------------------------------
0 -- 0 HundredGigE0/6/0/0 - HundredGigE0/6/0/31 --
1 -- 1 HundredGigE0/6/0/4 - HundredGigE0/6/0/7
NP2 is down. >>>>>>>>>. NP Down/Init Failure
3 -- 3 HundredGigE0/6/0/12 - HundredGigE0/6/0/154 --
4 -- 4 HundredGigE0/6/0/16 - HundredGigE0/6/0/19
这些日志观察：
LC/0/6/CPU0:Mar 23 02:53:56.175 IST: npu_server_lsp[138]: %PLATFORM-LDA-3-INIT_FAIL :
Failed to initialize lda_bb_np_reset_process 13795 inst 0x2 LC INIT: Failed in NP HAL
Reset np (0x00000001 - Operation not permitted) : npu_server_lsp : (PID=4597) :
-Traceback= 7fea2d5cd9f6 7fea2d7d5816 7fea21465efa 7fea21465fc2 7fea42ad0bed 55a9dbd66031
7fea45e1c855 7fea45e1cc2b 7fea2624d526 7fea3571b96a 7fea4d6e4831 55a9dbd691e9
LC/0/6/CPU0:Mar 23 02:53:56.185 IST: npu_server_lsp[138]: %PLATFORM-NP-4-INIT_DEBUG_MSG :
LDA NP2 Reset failed!! Check for a downlevel IPU version.
日志收集：
show tech-support ethernet interfaces
show tech-support ethernet controllers
show tech-support np
show tech-support fpd
admin show tech-support ctrace (in eXR)
show tech fabric
show asic-errors fia trace all location <>
show logging
gather (in eXR)
RP/0/RP0/CPU0:AG3_1#admin
sysadmin-vm:0_RP0#
[sysadmin-vm:0_RP0:~]$bash -l
[sysadmin-vm:0_RP0:~]$ gather
File will be generated and will get saved in rp0_xr:/misc/disk1
Tomahawk、LSQ和LSP的一般日志收集
show platform
show inventory
show tech fabric
show tech np
show tech ethernet interface
show logging
show pfm location all
show pfm trace location <location id>
sh pfm process <> location <>
show controllers pm vqi location all
show hw-module fpd location all (cxr) / admin show hw-module fpd (exr)
show controllers fti trace <process-name> location <card location>
Cxr:
From admin:
show logging onboard common location <>
show logging onboard error location <>
Exr:
From sysadmin/calvados:
show logging onboard fabric location <>"
常见错误签名和建议
| 分类 |
Error |
观察结果 |
建议 |
| NP初始化失败 |
LC/0/0/CPU0:Sep 29 00:41:13.171 IST: pfm_node_lc[304]: %PLATFORM-NP-1-NP_INIT_FAIL_NO_RESET: Set|prm_server_ty[168018]|0x1008006|Persistent NP initialization failure， line card reload not required. |
由于硬件奇偶校验/TCAM错误，NP可能会进入NP初始化持续错误，从而迫使NP关闭。 |
LC通过CLI重新加载以进行恢复。 |
| 如果LC在第一次重新加载后是暂时的，则问题可以恢复。 |
完成同一错误的重复实例的退货许可(RMA)。 |
||
| RMA（如果看到重复实例）。（捕获故障板的照片，以检查SR的针脚是否损坏/弯曲）。 |
|||
| 由于错误字段处理，新卡可能会面临相同的问题。 |
|||
| 映射到NP的接口保持关闭/无影响。 |
|||
| ASIC严重故障 — 双位ECC错误 |
LC/0/8/CPU0:May 29 18:29:09.836 IST: pfm_node_lc[301]: %FABRIC-FIA-0-ASIC_FATAL_FAULT : Set|fialc[159811]|0x108a000|交换矩阵接口asic ASIC0遇到致命故障0x1 - DDR DOUBLE ECC错误 |
这是FIA上的双位ECC错误是硬错误。 |
FIA上出现HW错误。 |
| 该错误可能重新出现，因此建议使用admin shut of LC。 |
RMA卡。 |
||
| 映射到NP/FIA的接口保持关闭/无影响。 |
|||
| 在FIA提出fib_mgr进程块的一个案例中，可以看到此问题。 |
|||
| SERDES错误 |
·RP/0/RSP1/CPU0:Apr 17 12:22:10.690 IST:pfm_node_rp[378]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK0:Set|fab_xbar[209006]|0x101702f|XBAR_1_Slot_1 |
LC交换矩阵或RSP交换矩阵上的交换矩阵错误 |
LC通过CLI重新加载，以便针对重复错误恢复瞬时/CRC错误。 |
| DATA_NB_SERDES_1_FAIL_0 |
LC/0/3/CPU0:Apr 10 18:55:03.213 IST: pfm_node_lc[304]: %FABRIC-FIA-1-DATA_NB_SERDES_1_FAIL_0 : Set|fialc[168004]|0x103d001|FIA 1上的数据NB服务器链路1故障 RP/0/RSP0/CPU0:Apr 10 18:55:13.043 IST: FABMGR[227]: %PLATFORM-FABMGR-2-FABRIC_INTERNAL_FAULT: 0/3/CPU0（插槽3）遇到交换矩阵故障。接口将关闭。 |
接口重新训练机制，以自动恢复交换矩阵上的SERDES错误。 如果出现硬件问题，该错误可能会再次在LC或RSP上出现。 接口处于up/frequent errors on SERDES impacted the traffic（SERDES上的接口保持/频繁错误影响流量）。 |
OIR后重复实例的RMA。 |
| ASIC INIT错误 |
·LC/0/6/CPU0:Jul 17 00:01:40.738 2019:pfm_node_lc[301]: %FABRIC-FIA-1-ASIC_INIT_ERROR ：在FIA实例0上检测到Set|fialc[168003]|0x108a000|ASIC INIT错误 |
LC上任何FIA的FIA实例关闭事件，系统日志中出现ASIC INIT错误。 |
LC通过CLI重新加载以排除任何临时问题。 |
| FIA ASIC严重错误 (TS_NI_INTR_LCL_TIMER_EXPIRED) |
LC/0/19/CPU0:Mar 8 04:52:29.020 IST: pfm_node_lc[301]: %FABRIC-FIA-0-FATAL_INTERRUPT_ERROR : Set|fialc[172098]|0x108a003|FIA FATAL error interrupt on FIA 3: TS_NI_INTR_LCL_TIMER_EXPIRED |
对于新卡，可以发现设备在发货/安装时处理不当，导致主板物理损坏。 少数电路板没有出现任何物理损坏，但在EFA期间观察到焊料裂缝。这表明包装过紧，并可能随着时间的推移而发生故障。 接口处于up/frequent errors on SERDES impacted the traffic（SERDES上的接口保持/频繁错误影响流量）。 |
如果问题仍然存在，请继续执行RMA/R&R。 |
| NP快速重置(Tomahawk) |
LC/0/4/CPU0:Jul 6 04:06:49.259 IST: prm_server_ty[318]: %PLATFORM-NP-3-ECC : prm_ser_check：已完成NP快速重置以成功从NP 1上的软错误中恢复。无需进一步纠正操作。 |
NP检测到软奇偶校验问题，并尝试通过尝试NP快速重置来修复该问题。 |
首次出现时没有RMA。 |
| NP奇偶校验LC重新加载 |
LC/0/6/CPU0:Jan 27 20:38:08.011 IST: prm_server_to[315]: %PLATFORM-NP-0-LC_RELOAD: NP3在一小时内进行了三次快速重置，启动NPdatalog收集和自动LC重新启动 |
通常，在三次恢复尝试后，LC自行重新加载，以解决Tomahawk卡通常看到的NP上的奇偶校验问题。 LC通过重新加载LC并修复报告的NP中的不可恢复的软奇偶校验问题来采取自动恢复操作。 映射到NP的接口会因重置/无影响而关闭。 |
RMA，表示同一错误的重复实例。 |
| LC_NP_LOOPBACK_FAILED |
LC/0/1/CPU0:Jul 26 17:29:06.146 IST: pfm_node_lc[304]: %PLATFORM-DIAGS-0-LC_NP_LOOPBACK_FAILED_TX_PATH : Set|online_diag_lc[168022]|线卡NPU环回测试(0x2000006)|链路故障掩码为0x1。 |
LC NP环回诊断测试在某个NP上失败。 |
LC通过CLI重新加载以排除任何临时问题。 |
| 在PFM中将警报设置为“LC_NP_LOOPBACK_FAILED_XX_PATH”。 |
RMA，表示同一错误的重复实例。 |
||
| 映射到NP的接口会因重置/无影响而关闭。 |
|||
| FABRIC-FIA-1-SUSTAINED_CRC_ERR |
LC/0/5/CPU0:Mar 6 05:47:34.748 IST: pfm_node_lc[303]: %FABRIC-FIA-1-SUSTAINED_CRC_ERR : Set|fialc[168004]|0x103d000|交换矩阵接口ASIC-0存在持续CRC错误 |
由于FIA持续CRC错误，Fia关闭。 |
LC通过CLI重新加载以排除任何临时问题。 |
| 在FIA关闭事件中，FIA上的接口也会关闭。 |
RMA，表示同一错误的重复实例。 |
||
| 接口保持开启/无影响。 |
|||
| FAB ARB XIF1错误 |
·LC/0/6/CPU0:Jan 25 19:31:22.787 IST: pfm_node_lc[302]: %PLATFORM-FABARBITER-1-RX_LINK_ERR : Clear|fab_arb[163918]|0x1001001|LIT_XIF1_K_CHAR_ERR LC/0/6/CPU0:Jan 25 19:31:22.787 IST: pfm_node_lc[302]: %PLATFORM-FABARBITER-1-SYNC_ERR : Clear|fab_arb[163918]|0x1001001|LIT_XIF1_LOSS_SYNC LC/0/6/CPU0:Jan 25 19:33:23.010 IST: pfm_node_lc[302]: %PLATFORM-FABARBITER-1-RX_LINK_ERR : Set|fab_arb[163918]|0x1001001|LIT_XIF1_DISP_ERR |
LC & Fabric arbiter sync & rx_link错误的PUNT错误。 接口保持开启/无影响。 |
OIR卡以排除任何瞬时问题。 RMA，表示同一错误的重复实例。 |
| FPOE_read_write错误 |
xbar错误跟踪(show tech fabric) |
Cisco Bug ID CSCvv45788 |
软件缺陷 |
| FIA_XBAR系列 |
#show controller fabric fia link-status location 0/9/CPU0 |
OIR卡以排除任何瞬时问题。RMA，表示同一错误的重复实例。 |
|
| NP DIAG ICFD快速重置 |
NP-DIAG on NP0, ICFD(STS-1), NP可以是0-4 NP3在一小时内进行了3次快速重置，启动NPdatalog收集和LC自动重启 |
触发NP的FAST重置 如果在一小时内有3个NP快速重置，则LC会重新加载。 |
如果LC重新加载多次，则RMA。 |
| PRM运行状况监控无法获取数据包NP快速重置 |
NP-DIAG运行状况监控故障 NP3在一小时内进行了3次快速重置，启动NPdatalog收集和LC自动重启 |
触发NP的FAST重置 如果在一小时内有3个NP快速重置，则LC会重新加载。 |
如果LC重新加载多次，则RMA。 |
| PRM运行状况监控获取损坏的数据包 — NP快速重置 |
NP-DIAG运行状况监控损坏 NP3在一小时内进行了3次快速重置，启动NPdatalog收集和LC自动重启 |
触发NP的FAST重置 如果在一小时内有3个NP快速重置，则LC会重新加载。 |
如果LC重新加载多次，则RMA。 |
| 最频繁的非活动故障 |
NP上的NP-DIAG故障 Upper inactivity上的Ucode中断 — NP是否快速重置 |
触发NP的FAST重置 如果在一小时内有3个NP快速重置，则LC会重新加载。 |
如果LC重新加载多次，则RMA。 |
| LSP NP初始化失败 |
LC/0/6/CPU0:Mar 23 02:53:56.175 IST: npu_server_lsp[138]: %PLATFORM-LDA-3-INIT_FAIL ：无法初始化lda_bb_np_reset_process 13795 inst 0x2 LC INIT：在NP HAL重置np(0x00000001 — 不允许操作)中失败：npu_server_lsp :(PID=4597): -Traceback9fea2d f6 7fea2d7d5816 7fea21465efa 7fea21465fc2 7fea42ad0bed 55a9dbd66031 7fea45e1c855 7fea45e1cc2b 7fea2624d526 7fea3571b96a 7fea4d6e4831 55a9dbd691e9 |
需要收集以下信息：
文件生成并保存在rp0_xr:/misc/disk1中
|
LC通过CLI重新加载以排除任何临时问题。 |
| Tomahawk NP初始化失败(DDR培训失败) |
+++ show prm server trace error location 0/7/CPU0 [14:36:59.520 IST Sat Jan 29 222] ++++ 97个包装条目（可能有2112个，已分配320个，已过滤0个，共97个） 1月29日00:22:10.135 prm_server/error 0/7/CPU0 t10 prm_np_Channel_PowerUp : 0x80001d46为通道3阶段4供电时出错 1月29日00:22:10.136 prm_server/error 0/7/CPU0 t10 np_thread_channel_bringup : 0xa57c0200电源阶段4在通道3上失败 1月29日00:22:10.136 prm_server/error 0/7/CPU0 t10 np_thread_channel_bringup NP3启动失败，请重试。重试次数1 1月29日00:22:35.125 prm_server/error 0/7/CPU0 t10 prm_np_Channel_PowerUp : 0x80001d46为通道3阶段4供电时出错 1月29日00:22:35.125 prm_server/error 0/7/CPU0 t10 np_thread_channel_bringup : 0xa57c0200电源阶段4在通道3上失败 1月29日00:22:35.125 prm_server/error 0/7/CPU0 t10 np_thread_channel_bringup NP3启动失败，请重试。重试次数2 1月29日00:22:59.075 prm_server/error 0/7/CPU0 t10 prm_np_Channel_PowerUp : 0x80001d46为通道3阶段4供电时出错 1月29日00:22:59.075 prm_server/error 0/7/CPU0 t10 np_thread_channel_bringup : 0xa57c0200电源阶段4在通道3上失败 1月29日00:22:59.075 prm_server/error 0/7/CPU0 t10 np_thread_channel_bringup 3次尝试后，NP3未能初始化。 1月29日00:23:00.087 prm_server/error 0/7/CPU0 t10 prm_send_pfm_msg：持续NP初始化失败，不需要线路卡重新加载。 签入NP驱动程序日志 <NP#3>DDR培训失败（状态0x1） <NP#3>ddr3TipRunAlg：调整失败0 <NP#3>ddrTipRunAlgo操作码：ddr3TipRunAlg失败（错误0x1） <NP#3>***错误：未知0x1 |
节点：node0_7_CPU0 2029年1月00:22:58|8 |NP_INIT_FAIL_NO_RESET |E/A|5356 |网络进程|0x1008000 |
LC通过CLI重新加载以排除任何临时问题。如果问题仍观察RMA，则LC Cisco Bug ID CSCwa85165 |
| LSP NP初始化失败(HbmReadParticleError) |
LC/0/13/CPU0:Jan 10 13:34:59.106 IST: npu_server_lsp[278]: %PLATFORM-NP-4-SHUTDOWN_START : NP4: EMRHIMREG.ch1Psch0HbmReadParticleError detected， NP shutdown in progress LC/0/13/CPU0:Jan 10 13:34:59.106 IST: pfm_node_lc[330]: %PLATFORM-NP-0-UNRECOVERABLE_ERROR : Set|npu_server_lsp[4632]|0x10a5004|在NP4上检测到不可恢复的错误 |
+++ show controllers np interrupts all location 0/13/CPU0 [16:02:16.712 IST Mon Jan 10 2022] ++++ 节点：0/13/CPU0: ---------------------------------------------------------------- NPU中断名称Id Cnt — ------------------------------------------------------------------------------------------ -------------- ------- <snip> 4 hbmdpReg0.hbmdpExceptionLeaf0.hbmdpIntNwlHbmdpRdDataUncorrectableErrCh1Psch1 0x201dc013 1 4 hbmdpReg0.hbmdpExceptionLeaf0.hbmdpIntNwlHbmdpRdDataUncorrectableErrCh1Psch0 0x201dc012 1 |
验证是否已安装Cisco Bug ID CSCvt59803。 LC通过CLI重新加载有助于恢复。 |
| 仲裁链路断开与备用 |
交换矩阵管理器： |
OIR卡以排除任何瞬时问题。RMA，表示同一错误的重复实例。 |
|
| Serdes错误 |
show serdes trace location 0/X/CPU0 | 在“HTL_ERR_DEVICE_NOT_CONNECTED”)中，您会看到以下错误： |
Cisco Bug ID CSCvz75552 |
软件缺陷 |
已知缺陷
| Cisco Bug ID | 组件 | 标题 |
| Cisco Bug ID CSCvy00012 | asr9k-diags-online | 数据包内存耗尽（由online_diag_rsp完成） |
| Cisco Bug ID CSCvw57721 | asr9k-servicepack | 包含Lightspeed NP和仲裁服务器更新固件的Umbrella SMU |
| Cisco Bug ID CSCvz75552 | asr9k-vic-ls | Phy固件挂起并导致A9K-20HG-FLEX上无法识别光纤 |
| Cisco Bug ID CSCvz76691 | asr9k-servicepack | 针对Tomahawk线卡具有改进链路状态中断处理的伞SMU |
| Cisco Bug ID CSCvz84139 | asr9k-ls-fabric | 当路由器升级到742时fab_si崩溃 |
| Cisco Bug ID CSCwa81006 | asr9k-pfm | ASR9K/eXR在某些情况下无法提交故障管理器数据路径端口关闭 |
| Cisco Bug ID CSCvz16840 | asr9k-fia | CLI重新加载LC时BLB会话抖动，因为转发路径由于6.5.2中添加的更改而提前关闭 |
| Cisco Bug ID CSCwb64255 | asr9k-fab-xbar | Starscreel(9912)和Megatron(9922)机箱中SKB的新SI设置 |
| Cisco Bug ID CSCwa09794 | asr9k-fab-xbar | 对SKB-SM15的RO机箱进行微调后获得新的SI |
| Cisco Bug ID CSCvv45788 | asr9k-fab-xbar | fab_xbar和mgid-programmer进程同时访问硬件 |
| Cisco Bug ID CSCwd22196 | asr9k-prm | Tomahawk LC上ILKN链路之间的RFD缓冲区耗尽 |
| Cisco Bug ID CSCwb66960 | asr9k-fab-infra | ASR9k punt交换矩阵故障隔离 |
| Cisco Bug ID CSCwa79758 | asr9k-fab-xbar | 对另一个具有XBAR链路故障的LSP LC执行OIR后，LSP LC上的组播丢失 |
| Cisco Bug ID CSCvw88284 | asr9k-lda-ls | RSP5 BW在9910/9906机箱上默认为200G，而不是600G。 |
| Cisco Bug ID CSCvm82379 | asr9k-fab-arb | fab-arb在接受sh tech fabric时崩溃 |
| Cisco Bug ID CSCvh00349 | asr9k-fia | ASR9k交换矩阵可以处理备用时发送的ucast数据包 |
| Cisco Bug ID CSCvk44688 | asr9k-fia | FPGA多次出错，无法恢复 |
| Cisco Bug ID CSCvy31670 | asr9k-ls-fia | LSP：删除FC0启用交换矩阵速率限制器，FC4不启用 |
| Cisco Bug ID CSCvt59803 | asr9k-ls-npdriver | LSP:PLATFORM-NP-4-SHUTDOWN IMRHIMREG.ch1Psch1HbmReadParticleError |
行为 fault-manager datapath port shutdown/toggle 命令
- 此
fault-manager datapath port shutdown命令有助于在活动RP/RSP上关闭设置了Punt Datapath Failure警报的各个FIA/NP的端口，并且直到您重新加载LC后，接口才会自动打开。此CLI命令在7. x.x版本中无法按预期工作。(CLI命令fault-manager datapath port shutdown不按照7.x.x中的设计工作 — 已在7.7.2中修复。 - 此
fault-manager datapath port toggleCLI命令运行正常。当Punt Datapath Failure警报清除后，它会打开端口。 - 如果冗余路径上存在适当的链路级冗余和BW可用性，这有助于防止服务中断。
测试 — 验证前面提到的命令操作。
在NP0 LC7上引发PUNT错误生成：
RP/0/RP0/CPU0:ASR-9922-A#monitor np counter PUNT_DIAGS_RSP_ACT np0 count 20 location 0/7/CPU0
Wed Jul 7 14:15:17.489 UTC
Usage of NP monitor is recommended for cisco internal use only.
Please use instead 'show controllers np capture' for troubleshooting packet drops in NP
and 'monitor np interface' for per (sub)interface counter monitoring
Warning: Every packet captured will be dropped! If you use the 'count'
option to capture multiple protocol packets, this could disrupt
protocol sessions (eg, OSPF session flap). So if capturing protocol
packets, capture only 1 at a time.
Warning: A mandatory NP reset will be done after monitor to clean up.
This will cause ~150ms traffic outage. Links will stay Up.
Proceed y/n [y] > y
Monitor PUNT_DIAGS_RSP_ACT on NP0 ... (Ctrl-C to quit)
Wed Jul 7 14:17:08 2021 -- NP0 packet
From Fabric: 127 byte packet
0000: 00 09 00 00 b4 22 00 00 ff ff ff ff 00 00 ff ff ....4"..........
0010: 00 ff 00 ff f0 f0 f0 f0 cc cc cc cc aa aa aa aa ....ppppLLLL****
0020: 55 55 55 55 00 00 00 00 01 00 00 00 00 00 00 00 UUUU............
0030: 00 00 00 00 ff ff ff ff 00 00 ff ff 00 ff 00 ff ................
0040: f0 f0 f0 f0 cc cc cc cc aa aa aa aa 55 55 55 55 ppppLLLL****UUUU
0050: 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................
0060: ff ff ff ff 00 00 ff ff 00 ff 00 ff f0 f0 f0 f0 ............pppp
0070: cc cc cc cc aa aa aa aa 55 55 55 55 00 00 00 LLLL****UUUU...
(count 1 of 20)
Wed Jul 7 14:18:09 2021 -- NP0 packet
From Fabric: 256 byte packet
0000: 00 09 00 00 b5 22 00 00 ff ff ff ff 00 00 ff ff ....5"..........
0010: 00 ff 00 ff f0 f0 f0 f0 cc cc cc cc aa aa aa aa ....ppppLLLL****
0020: 55 55 55 55 00 00 00 00 01 00 00 00 00 00 00 00 UUUU............
0030: 00 00 00 00 ff ff ff ff 00 00 ff ff 00 ff 00 ff ................
0040: f0 f0 f0 f0 cc cc cc cc aa aa aa aa 55 55 55 55 ppppLLLL****UUUU
0050: 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................
0060: ff ff ff ff 00 00 ff ff 00 ff 00 ff f0 f0 f0 f0 ............pppp
0070: cc cc cc cc aa aa aa aa 55 55 55 55 00 00 00 00 LLLL****UUUU....
0080: 01 00 00 00 00 00 00 00 00 00 00 00 ff ff ff ff ................
0090: 00 00 ff ff 00 ff 00 ff f0 f0 f0 f0 cc cc cc cc ........ppppLLLL
00a0: aa aa aa aa 55 55 55 55 00 00 00 00 01 00 00 00 ****UUUU........
00b0: 00 00 00 00 00 00 00 00 ff ff ff ff 00 00 ff ff ................
00c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
(count 2 of 20)
Wed Jul 7 14:19:09 2021 -- NP0 packet
Actual packet size 515 bytes truncated size 384:
From Fabric: 384 byte packet
0000: 00 09 00 00 b6 22 00 00 ff ff ff ff 00 00 ff ff ....6"..........
0010: 00 ff 00 ff f0 f0 f0 f0 cc cc cc cc aa aa aa aa ....ppppLLLL****
0020: 55 55 55 55 00 00 00 00 01 00 00 00 00 00 00 00 UUUU............
0030: 00 00 00 00 ff ff ff ff 00 00 ff ff 00 ff 00 ff ................
0040: f0 f0 f0 f0 cc cc cc cc aa aa aa aa 55 55 55 55 ppppLLLL****UUUU
0050: 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................
0060: ff ff ff ff 00 00 ff ff 00 ff 00 ff f0 f0 f0 f0 ............pppp
0070: cc cc cc cc aa aa aa aa 55 55 55 55 00 00 00 00 LLLL****UUUU....
0080: 01 00 00 00 00 00 00 00 00 00 00 00 ff ff ff ff ................
0090: 00 00 ff ff 00 ff 00 ff f0 f0 f0 f0 cc cc cc cc ........ppppLLLL
00a0: aa aa aa aa 55 55 55 55 00 00 00 00 01 00 00 00 ****UUUU........
00b0: 00 00 00 00 00 00 00 00 ff ff ff ff 00 00 ff ff ................
00c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
RP/0/RP0/CPU0:ASR-9922-A#sh pfm location 0/RP0/CPU0
Wed Jul 7 14:19:17.174 UTC
node: node0_RP0_CPU0
---------------------
CURRENT TIME: Jul 7 14:19:17 2021
PFM TOTAL: 2 EMERGENCY/ALERT(E/A): 1 CRITICAL(CR): 0 ERROR(ER): 1
-------------------------------------------------------------------------------------------------
Raised Time |S#|Fault Name |Sev|Proc_ID|Dev/Path Name |Handle
--------------------+--+-----------------------------------+---+-------+--------------+----------
Jul 1 10:13:45 2021|0 |SPINE_UNAVAILABLE |E/A|5082 |Fabric Manager|0x1034000
Jul 7 14:19:09 2021|0 |PUNT_FABRIC_DATA_PATH_FAILED |ER |9429 |System Punt/Fa|0x2000004
RP/0/RP0/CPU0:ASR-9922-A#sh pfm process 9429 location 0/Rp0/CPU0
Wed Jul 7 14:19:37.128 UTC
node: node0_RP0_CPU0
---------------------
CURRENT TIME: Jul 7 14:19:37 2021
PFM TOTAL: 2 EMERGENCY/ALERT(E/A): 1 CRITICAL(CR): 0 ERROR(ER): 1
PER PROCESS TOTAL: 0 EM: 0 CR: 0 ER: 0
Device/Path[1 ]:Fabric loopbac [0x2000003 ] State:RDY Tot: 0
Device/Path[2 ]:System Punt/Fa [0x2000004 ] State:RDY Tot: 1
1 Fault Id: 432
Sev: ER
Fault Name: PUNT_FABRIC_DATA_PATH_FAILED
Raised Timestamp: Jul 7 14:19:09 2021
Clear Timestamp: N/A
Changed Timestamp: N/A
Resync Mismatch: FALSE
MSG: failure threshold is 3, (slot, NP) failed: (0/7/CPU0, 0)
Device/Path[3 ]:Crossbar Switc [0x108c000 ] State:RDY Tot: 0
Device/Path[4 ]:Crossbar Switc [0x108c001 ] State:RDY Tot: 0
Device/Path[5 ]:Crossbar Switc [0x108c002 ] State:RDY Tot: 0
Device/Path[6 ]:Crossbar Switc [0x108c003 ] State:RDY Tot: 0
Device/Path[7 ]:Crossbar Switc [0x108c004 ] State:RDY Tot: 0
Device/Path[8 ]:Crossbar Switc [0x108c005 ] State:RDY Tot: 0
Device/Path[9 ]:Crossbar Switc [0x108c006 ] State:RDY Tot: 0
Device/Path[10]:Crossbar Switc [0x108c007 ] State:RDY Tot: 0
Device/Path[11]:Crossbar Switc [0x108c008 ] State:RDY Tot: 0
Device/Path[12]:Crossbar Switc [0x108c009 ] State:RDY Tot: 0
Device/Path[13]:Crossbar Switc [0x108c00a ] State:RDY Tot: 0
Device/Path[14]:Crossbar Switc [0x108c00b ] State:RDY Tot: 0
Device/Path[15]:Crossbar Switc [0x108c00c ] State:RDY Tot: 0
Device/Path[16]:Crossbar Switc [0x108c00d ] State:RDY Tot: 0
Device/Path[17]:Crossbar Switc [0x108c00e ] State:RDY Tot: 0
Device/Path[18]:Fabric Interfa [0x108b000 ] State:RDY Tot: 0
Device/Path[19]:Fabric Arbiter [0x1086000 ] State:RDY Tot: 0
Device/Path[20]:CPU Controller [0x108d000 ] State:RDY Tot: 0
Device/Path[21]:Device Control [0x109a000 ] State:RDY Tot: 0
Device/Path[22]:ClkCtrl Contro [0x109b000 ] State:RDY Tot: 0
Device/Path[23]:NVRAM [0x10ba000 ] State:RDY Tot: 0
Device/Path[24]:Hooper switch [0x1097000 ] State:RDY Tot: 0
Device/Path[25]:Hooper switch [0x1097001 ] State:RDY Tot: 0
Device/Path[26]:Hooper switch [0x1097002 ] State:RDY Tot: 0
Device/Path[27]:Hooper switch [0x1097003 ] State:RDY Tot: 0
在此例中，端口未关闭：
RP/0/RP0/CPU0:ASR-9922-A#sh ipv4 int brief location 0/7/CPU0
Wed Jul 7 14:21:29.693 UTC
Interface IP-Address Status Protocol Vrf-Name
TenGigE0/7/0/0 unassigned Down Down default
TenGigE0/7/0/1 unassigned Down Down default
TenGigE0/7/0/2 unassigned Down Down default
TenGigE0/7/0/3 unassigned Down Down default
TenGigE0/7/0/4 unassigned Down Down default
TenGigE0/7/0/5 unassigned Down Down default
TenGigE0/7/0/6 unassigned Down Down default
TenGigE0/7/0/7 unassigned Shutdown Down default
TenGigE0/7/0/8 unassigned Shutdown Down default
TenGigE0/7/0/9 unassigned Shutdown Down default
TenGigE0/7/0/10 unassigned Down Down default
TenGigE0/7/0/11 unassigned Down Down default
TenGigE0/7/0/12 unassigned Down Down default
TenGigE0/7/0/13 unassigned Shutdown Down default
TenGigE0/7/0/14 unassigned Shutdown Down default
TenGigE0/7/0/15 unassigned Shutdown Down default
TenGigE0/7/0/16 unassigned Shutdown Down default
TenGigE0/7/0/17 unassigned Shutdown Down default
TenGigE0/7/0/18 unassigned Down Down default
TenGigE0/7/0/19 unassigned Up Up default >>>>>>> Port is UP
RP/0/RP0/CPU0:ASR-9922-A#sh logging last 200 | in 0/7/0
Wed Jul 7 14:22:35.715 UTC
RP/0/RP0/CPU0:ASR-9922-A#
测试案例1.2:
NP/端口行为 fault-manager datapath port toggle 指令：
RP/0/RP0/CPU0:ASR-9922-A#sh run formal | in data
Wed Jul 7 14:52:11.714 UTC
Building configuration...
fault-manager datapath port toggle
RP/0/RP0/CPU0:ASR-9922-A#
PFM中无警报：
RP/0/RP0/CPU0:ASR-9922-A#sh pfm location 0/Rp0/CPU0
Wed Jul 7 14:55:13.410 UTC
node: node0_RP0_CPU0
---------------------
CURRENT TIME: Jul 7 14:55:13 2021
PFM TOTAL: 1 EMERGENCY/ALERT(E/A): 1 CRITICAL(CR): 0 ERROR(ER): 0
-------------------------------------------------------------------------------------------------
Raised Time |S#|Fault Name |Sev|Proc_ID|Dev/Path Name |Handle
--------------------+--+-----------------------------------+---+-------+--------------+----------
Jul 1 10:13:45 2021|0 |SPINE_UNAVAILABLE |E/A|5082 |Fabric Manager|0x1034000
RP/0/RP0/CPU0:ASR-9922-A#
在NP0 LC7中生成PUNT错误：
RP/0/RP0/CPU0:ASR-9922-A#monitor np counter PUNT_DIAGS_RSP_ACT np0 count 20 location 0/7/CPU0
Wed Jul 7 14:51:18.596 UTC
Usage of NP monitor is recommended for cisco internal use only.
Please use instead 'show controllers np capture' for troubleshooting packet drops in NP
and 'monitor np interface' for per (sub)interface counter monitoring
Warning: Every packet captured will be dropped! If you use the 'count'
option to capture multiple protocol packets, this could disrupt
protocol sessions (eg, OSPF session flap). So if capturing protocol
packets, capture only 1 at a time.
Warning: A mandatory NP reset will be done after monitor to clean up.
This will cause ~150ms traffic outage. Links will stay Up.
Proceed y/n [y] > y
Monitor PUNT_DIAGS_RSP_ACT on NP0 ... (Ctrl-C to quit)
Wed Jul 7 14:53:21 2021 -- NP0 packet
From Fabric: 127 byte packet
0000: 00 09 00 00 d8 22 00 00 ff ff ff ff 00 00 ff ff ....X"..........
0010: 00 ff 00 ff f0 f0 f0 f0 cc cc cc cc aa aa aa aa ....ppppLLLL****
0020: 55 55 55 55 00 00 00 00 01 00 00 00 00 00 00 00 UUUU............
0030: 00 00 00 00 ff ff ff ff 00 00 ff ff 00 ff 00 ff ................
0040: f0 f0 f0 f0 cc cc cc cc aa aa aa aa 55 55 55 55 ppppLLLL****UUUU
0050: 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................
0060: ff ff ff ff 00 00 ff ff 00 ff 00 ff f0 f0 f0 f0 ............pppp
0070: cc cc cc cc aa aa aa aa 55 55 55 55 00 00 00 LLLL****UUUU...
(count 1 of 20)
Wed Jul 7 14:54:22 2021 -- NP0 packet
From Fabric: 256 byte packet
0000: 00 09 00 00 d9 22 00 00 ff ff ff ff 00 00 ff ff ....Y"..........
0010: 00 ff 00 ff f0 f0 f0 f0 cc cc cc cc aa aa aa aa ....ppppLLLL****
0020: 55 55 55 55 00 00 00 00 01 00 00 00 00 00 00 00 UUUU............
0030: 00 00 00 00 ff ff ff ff 00 00 ff ff 00 ff 00 ff ................
0040: f0 f0 f0 f0 cc cc cc cc aa aa aa aa 55 55 55 55 ppppLLLL****UUUU
0050: 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................
0060: ff ff ff ff 00 00 ff ff 00 ff 00 ff f0 f0 f0 f0 ............pppp
0070: cc cc cc cc aa aa aa aa 55 55 55 55 00 00 00 00 LLLL****UUUU....
0080: 01 00 00 00 00 00 00 00 00 00 00 00 ff ff ff ff ................
0090: 00 00 ff ff 00 ff 00 ff f0 f0 f0 f0 cc cc cc cc ........ppppLLLL
00a0: aa aa aa aa 55 55 55 55 00 00 00 00 01 00 00 00 ****UUUU........
00b0: 00 00 00 00 00 00 00 00 ff ff ff ff 00 00 ff ff ................
00c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
(count 2 of 20)
RP/0/RP0/CPU0:ASR-9922-A#sh pfm location 0/Rp0/CPU0
Wed Jul 7 14:56:24.459 UTC
node: node0_RP0_CPU0
---------------------
CURRENT TIME: Jul 7 14:56:24 2021
PFM TOTAL: 2 EMERGENCY/ALERT(E/A): 1 CRITICAL(CR): 0 ERROR(ER): 1
-------------------------------------------------------------------------------------------------
Raised Time |S#|Fault Name |Sev|Proc_ID|Dev/Path Name |Handle
--------------------+--+-----------------------------------+---+-------+--------------+----------
Jul 1 10:13:45 2021|0 |SPINE_UNAVAILABLE |E/A|5082 |Fabric Manager|0x1034000
Jul 7 14:55:23 2021|0 |PUNT_FABRIC_DATA_PATH_FAILED |ER |9429 |System Punt/Fa|0x2000004
RP/0/RP0/CPU0:ASR-9922-A#sh pfm process 9429 location 0/RP0/CPU0
Wed Jul 7 14:56:39.961 UTC
node: node0_RP0_CPU0
---------------------
CURRENT TIME: Jul 7 14:56:40 2021
PFM TOTAL: 2 EMERGENCY/ALERT(E/A): 1 CRITICAL(CR): 0 ERROR(ER): 1
PER PROCESS TOTAL: 0 EM: 0 CR: 0 ER: 0
Device/Path[1 ]:Fabric loopbac [0x2000003 ] State:RDY Tot: 0
Device/Path[2 ]:System Punt/Fa [0x2000004 ] State:RDY Tot: 1
1 Fault Id: 432
Sev: ER
Fault Name: PUNT_FABRIC_DATA_PATH_FAILED
Raised Timestamp: Jul 7 14:55:23 2021
Clear Timestamp: N/A
Changed Timestamp: N/A
Resync Mismatch: FALSE
MSG: failure threshold is 3, (slot, NP) failed: (0/7/CPU0, 0)
Device/Path[3 ]:Crossbar Switc [0x108c000 ] State:RDY Tot: 0
Device/Path[4 ]:Crossbar Switc [0x108c001 ] State:RDY Tot: 0
Device/Path[5 ]:Crossbar Switc [0x108c002 ] State:RDY Tot: 0
Device/Path[6 ]:Crossbar Switc [0x108c003 ] State:RDY Tot: 0
Device/Path[7 ]:Crossbar Switc [0x108c004 ] State:RDY Tot: 0
Device/Path[8 ]:Crossbar Switc [0x108c005 ] State:RDY Tot: 0
Device/Path[9 ]:Crossbar Switc [0x108c006 ] State:RDY Tot: 0
Device/Path[10]:Crossbar Switc [0x108c007 ] State:RDY Tot: 0
Device/Path[11]:Crossbar Switc [0x108c008 ] State:RDY Tot: 0
Device/Path[12]:Crossbar Switc [0x108c009 ] State:RDY Tot: 0
Device/Path[13]:Crossbar Switc [0x108c00a ] State:RDY Tot: 0
Device/Path[14]:Crossbar Switc [0x108c00b ] State:RDY Tot: 0
Device/Path[15]:Crossbar Switc [0x108c00c ] State:RDY Tot: 0
Device/Path[16]:Crossbar Switc [0x108c00d ] State:RDY Tot: 0
Device/Path[17]:Crossbar Switc [0x108c00e ] State:RDY Tot: 0
Device/Path[18]:Fabric Interfa [0x108b000 ] State:RDY Tot: 0
Device/Path[19]:Fabric Arbiter [0x1086000 ] State:RDY Tot: 0
Device/Path[20]:CPU Controller [0x108d000 ] State:RDY Tot: 0
Device/Path[21]:Device Control [0x109a000 ] State:RDY Tot: 0
Device/Path[22]:ClkCtrl Contro [0x109b000 ] State:RDY Tot: 0
Device/Path[23]:NVRAM [0x10ba000 ] State:RDY Tot: 0
Device/Path[24]:Hooper switch [0x1097000 ] State:RDY Tot: 0
Device/Path[25]:Hooper switch [0x1097001 ] State:RDY Tot: 0
Device/Path[26]:Hooper switch [0x1097002 ] State:RDY Tot: 0
Device/Path[27]:Hooper switch [0x1097003 ] State:RDY Tot: 0
接口TenGigE0/7/0/19关闭NP0:
RP/0/RP0/CPU0:ASR-9922-A#show logging last 200 | in 0/7/0
Wed Jul 7 14:58:42.959 UTC
LC/0/7/CPU0:Jul 7 14:55:23.798 UTC: ifmgr[270]: %PKT_INFRA-LINK-3-UPDOWN :
Interface TenGigE0/7/0/19, changed state to Down
LC/0/7/CPU0:Jul 7 14:55:23.798 UTC: ifmgr[270]: %PKT_INFRA-LINEPROTO-5-UPDOWN :
Line protocol on Interface TenGigE0/7/0/19, changed state to Down
RP/0/RP0/CPU0:Jul 7 14:55:23.802 UTC: BM-DISTRIB[1290]: %L2-BM-6-ACTIVE :
TenGigE0/7/0/19 is no longer Active as part of Bundle-Ether854 (Link is down)
LC/0/7/CPU0:Jul 7 14:55:25.854 UTC: vic_0_0[379]: %PLATFORM-VIC-4-RFI :
Interface TenGigE0/7/0/19, Detected Remote Fault
LC/0/7/CPU0:Jul 7 14:55:26.936 UTC: lda_server[114]: %PKT_INFRA-FM-2-FAULT_CRITICAL :
ALARM_CRITICAL :OPTICS RX POWER LANE-0 LOW ALARM :CLEAR : Te0/7/0/0:
停止生成PUNT错误：
RP/0/RP0/CPU0:ASR-9922-A#sh ipv4 int brief location 0/7/CPU0
Wed Jul 7 14:59:16.322 UTC
Interface IP-Address Status Protocol Vrf-Name
TenGigE0/7/0/0 unassigned Down Down default
TenGigE0/7/0/1 unassigned Down Down default
TenGigE0/7/0/2 unassigned Down Down default
TenGigE0/7/0/3 unassigned Down Down default
TenGigE0/7/0/4 unassigned Down Down default
TenGigE0/7/0/5 unassigned Down Down default
TenGigE0/7/0/6 unassigned Down Down default
TenGigE0/7/0/7 unassigned Shutdown Down default
TenGigE0/7/0/8 unassigned Shutdown Down default
TenGigE0/7/0/9 unassigned Shutdown Down default
TenGigE0/7/0/10 unassigned Down Down default
TenGigE0/7/0/11 unassigned Down Down default
TenGigE0/7/0/12 unassigned Down Down default
TenGigE0/7/0/13 unassigned Shutdown Down default
TenGigE0/7/0/14 unassigned Shutdown Down default
TenGigE0/7/0/15 unassigned Shutdown Down default
TenGigE0/7/0/16 unassigned Shutdown Down default
TenGigE0/7/0/17 unassigned Shutdown Down default
TenGigE0/7/0/18 unassigned Down Down default
TenGigE0/7/0/19 unassigned Down Down default >>>>>>
警报已停止：
RP/0/RP0/CPU0:ASR-9922-A#sh pfm location 0/Rp0/CPU0
Wed Jul 7 15:01:44.478 UTC
node: node0_RP0_CPU0
---------------------
CURRENT TIME: Jul 7 15:01:44 2021
PFM TOTAL: 1 EMERGENCY/ALERT(E/A): 1 CRITICAL(CR): 0 ERROR(ER): 0
-------------------------------------------------------------------------------------------------
Raised Time |S#|Fault Name |Sev|Proc_ID|Dev/Path Name |Handle
--------------------+--+-----------------------------------+---+-------+--------------+----------
Jul 1 10:13:45 2021|0 |SPINE_UNAVAILABLE |E/A|5082 |Fabric Manager|0x1034000
RP/0/RP0/CPU0:ASR-9922-A#
接口打开：
RP/0/RP0/CPU0:ASR-9922-A#show logging | in 0/7/0/19
Wed Jul 7 15:06:11.532 UTC
LC/0/7/CPU0:Jul 7 14:55:23.798 UTC: ifmgr[270]: %PKT_INFRA-LINK-3-UPDOWN :
Interface TenGigE0/7/0/19, changed state to Down
LC/0/7/CPU0:Jul 7 14:55:23.798 UTC: ifmgr[270]: %PKT_INFRA-LINEPROTO-5-UPDOWN :
Line protocol on Interface TenGigE0/7/0/19, changed state to Down
RP/0/RP0/CPU0:Jul 7 14:55:23.802 UTC: BM-DISTRIB[1290]: %L2-BM-6-ACTIVE :
TenGigE0/7/0/19 is no longer Active as part of Bundle-Ether854 (Link is down)
LC/0/7/CPU0:Jul 7 14:55:25.854 UTC: vic_0_0[379]: %PLATFORM-VIC-4-RFI :
Interface TenGigE0/7/0/19, Detected Remote Fault
LC/0/7/CPU0:Jul 7 15:03:27.204 UTC: ifmgr[270]: %PKT_INFRA-LINK-3-UPDOWN :
Interface TenGigE0/7/0/19, changed state to Up
LC/0/7/CPU0:Jul 7 15:03:27.206 UTC: ifmgr[270]: %PKT_INFRA-LINEPROTO-5-UPDOWN :
Line protocol on Interface TenGigE0/7/0/19, changed state to Up
RP/0/RP0/CPU0:Jul 7 15:03:29.219 UTC: BM-DISTRIB[1290]: %L2-BM-6-ACTIVE :
TenGigE0/7/0/19 is Active as part of Bundle-Ether854
修订历史记录
| 版本 | 发布日期 | 备注 |
|---|---|---|
1.0 |
09-Jan-2023 |
初始版本 |
反馈