排除ASR 5500中的SERDES Lane故障
下載選項
已更新: 2022 年 8 月 18 日
文件 ID:218054
無偏見用語
本產品的文件集力求使用無偏見用語。針對本文件集的目的，無偏見係定義為未根據年齡、身心障礙、性別、種族身分、民族身分、性別傾向、社會經濟地位及交織性表示歧視的用語。由於本產品軟體使用者介面中硬式編碼的語言、根據 RFP 文件使用的語言，或引用第三方產品的語言，因此本文件中可能會出現例外狀況。深入瞭解思科如何使用包容性用語。
關於此翻譯
思科已使用電腦和人工技術翻譯本文件，讓全世界的使用者能夠以自己的語言理解支援內容。請注意，即使是最佳機器翻譯，也不如專業譯者翻譯的內容準確。Cisco Systems, Inc. 對這些翻譯的準確度概不負責，並建議一律查看原始英文文件（提供連結）。
簡介
本文檔介紹ASR 5500中序列化程式反序列化程式(SERDES)lane(link)的故障排除命令。
背景資訊
ASR 5500包含卡之間的SERDES鏈路，以便於交換矩陣和儲存卡(FSC)、資料處理卡(DPC)和管理輸入/輸出(MIO)卡之間的通訊和資料路徑。有時，這些SERDES鏈路可能會由於錯誤或硬體故障而中斷。
SERDES Lane故障排除命令
調查ASR 5500機箱SERDES通道的命令：
- 收集
show support details，請檢視「調試控制檯……」 輸出行的部分：
1397273780.205 card 5-cpu0: afio [5/0/7808] [ 80616.933] afio/afio_fe600_serdes.c:3297: #1: fe600=47=16/1, Fabric SERDES lane transitioned from up to down, serdes=29, devid=25=7/1
- 訪問機箱CLI的測試模式
cli test-commands password.
注意：使用此模式可能會導致嚴重的服務中斷
- 問題
show fabric health命令獲取交換結構的整體檢視。
提示：可從swift.com的 show fabric support details show support details的一部分
在示例中，DPC卡2和FSC卡14之間出現問題。
在輸出中，從插槽2中的源DPC向插槽14中的FSC報告故障：
Command: petra-b system-device-id 3
Command: show health
Petra-B 3=2/1
Fabric Status:
Status OK(+)------------------------+
Topology fault(T)------------------+|
Far side not expected(*)----------+||
Logically not connected(L)-------+|||
Physically not connected(P)-----+|||| NIF Status:
Rx Down(*)---------------------+||||| +----------NIF powered off(*)
Tx Down(*)--------------------+|||||| |+---------SERDES powered off(*)
Code Group(G)----------------+||||||| ||+--------Local side down(l)
Misalignment(M)-------------+|||||||| |||+-------Remote side down(r)
Cell Size(C)---------------+||||||||| ||||+------Rx activity(r)
Internally fixed(I)-------+|||||||||| |||||+-----Tx activity(t)
Not Accept Cells(A)------+||||||||||| ||||||+----Status OK(+)
|||||||||||| |||||||
SERDES Status: |||||||||||| |||||||
Status OK(+)-----------+ |||||||||||| |||||||
Rx power off(*)-------+| |||||||||||| |||||||
Tx power off(*)------+|| |||||||||||| |||||||
Sig not locked(S)---+||| |||||||||||| |||||||
Rx signal loss(*)--+|||| |||||||||||| |||||||
Modified Parms(m)-+||||| |||||||||||| |||||||
Admin down(D)----+|||||| |||||||||||| |||||||
||||||| |||||||||||| |||||||
Fabric lane-----+ ||||||| |||||||||||| |||||||
SERDES lane--+ | ||||||| |||||||||||| ||||||| Config
Source Dev SL FL vvvvvvv vvvvvvvvvvvv vvvvvvv Rate Topology CRC Errs Remote Dev SL FL Last Change
------- --- -- -- ------- ------------ ------- ------------ -------- -------- ------- --- -- -- ----------------------
3= 2/1 FAP 47 15 + A M L 6250.00 Mbps - - 43=14/1 FE 82 82 FAULT_DETECTED ***
在另一個方向的相同鏈路的輸出中，從插槽14中的FSC卡到插槽2中的DPC卡，報告了相同的錯誤：
Command: fe600 system-device-id 43
Command: show health
FE600 43=14/1
Fabric Status:
Status OK(+)------------------------+
Topology fault(T)------------------+|
Far side not expected(*)----------+||
Logically not connected(L)-------+|||
Physically not connected(P)-----+|||| NIF Status:
Rx Down(*)---------------------+||||| +----------NIF powered off(*)
Tx Down(*)--------------------+|||||| |+---------SERDES powered off(*)
Code Group(G)----------------+||||||| ||+--------Local side down(l)
Misalignment(M)-------------+|||||||| |||+-------Remote side down(r)
Cell Size(C)---------------+||||||||| ||||+------Rx activity(r)
Internally fixed(I)-------+|||||||||| |||||+-----Tx activity(t)
Not Accept Cells(A)------+||||||||||| ||||||+----Status OK(+)
|||||||||||| |||||||
SERDES Status: |||||||||||| |||||||
Status OK(+)-----------+ |||||||||||| |||||||
Rx power off(*)-------+| |||||||||||| |||||||
Tx power off(*)------+|| |||||||||||| |||||||
Sig not locked(S)---+||| |||||||||||| |||||||
Rx signal loss(*)--+|||| |||||||||||| |||||||
Modified Parms(m)-+||||| |||||||||||| |||||||
Admin down(D)----+|||||| |||||||||||| |||||||
||||||| |||||||||||| |||||||
Fabric lane-----+ ||||||| |||||||||||| |||||||
SERDES lane--+ | ||||||| |||||||||||| ||||||| Config
Source Dev SL FL vvvvvvv vvvvvvvvvvvv vvvvvvv Rate Topology CRC Errs Remote Dev SL FL Last Change
------- --- -- -- ------- ------------ ------- ------------ -------- -------- ------- --- -- -- ----------------------
43=14/1 FE 82 82 + L T 6250.00 Mbps 3= 2/1 - 3= 2/1 FAP 47 15 FAULT_DETECTED ***
SERDES鏈路的另一類問題是鏈路的離線狀態。在示例中，插槽6中的DPC卡和17中的FSC卡之間的鏈路處於離線狀態：
23= 6/3 FAP 38 6 D 6250.00 Mbps 50=17/2 1557643 50=17/2 FE 65 65 OFFLINE ***
活動的SERDES鏈路總數和活動鏈路數顯示在 show fabric status 指令。在圖中所示的示例中，對兩條鏈路進行了計數，每條鏈路各佔一條。一條車道下車不是問題。交換矩陣容量大量過剩，並且單個通道不會影響吞吐量。唯一的問題是，如果鏈路由於錯誤而不斷上下，則可能丟失使用者和控制流量，因此如果鏈路斷開，情況會更好。
[local]ASR5500> show fabric status
Total number of FAPs: 24
Total number of FEs : 8
Total number of SERDES links: 1600
Total number of active SERDES links: 1598
附註：交換矩陣容量大量過剩，並且單個通道不會影響機箱的吞吐量。
- 通常，鏈路問題會自行清除。系統通過稱為Eyescan的自動步驟來解決問題。在
show serdes all-serdes history部分show fabric support details
附註：FE（交換矩陣元素）是FSC卡側。FAP（交換矩陣陣列處理器）是DPC和/或MIO卡端。
DPC卡有2個FAP，DPC2卡只有1個FAP;mio卡有4個FAP，FSC有2個FE。
命令輸出的格式為<card #>/<FAP/FE #>，例如，MIO 5將有5/1、5/2、5/3、5/4。
滿載DPC2機箱將有28個終端：8(8 DPC)+ 8(2 MIO * 4)+ 12(6 FCS * 2)
顯示了自動恢復後恢復的FE端的示例：
card=5, cpu=0, pid=7808, peer_mode=AFIO_IPC_PEER_MODE_DAEMON, sys_dev_id=47=16/1
Fabric Status:
Topology fault(T)----------------+
Far side not expected(*)--------+|
Logically not connected(L)-----+||
Physically not connected(P)---+|||
Rx Down(*)-------------------+||||
Tx Down(*)------------------+||||| NIF Status:
Code Group(G)--------------+|||||| +--------NIF powered off(*)
Misalignment(M)-----------+||||||| |+-------SERDES powered off(*)
Cell Size(C)-------------+|||||||| ||+------Local side down(l)
Internally fixed(I)-----+||||||||| |||+-----Remote side down(r)
Not Accept Cells(A)----+|||||||||| ||||
||||||||||| ||||
SERDES Status: ||||||||||| ||||
Rx power off(*)------+ ||||||||||| ||||
Tx power off(*)-----+| ||||||||||| ||||
Sig not locked(S)--+|| ||||||||||| ||||
Rx signal loss(*)-+||| ||||||||||| ||||
Admin Down(D)----+|||| ||||||||||| ||||
||||| ||||||||||| ||||
Fabric lane-----+ ||||| ||||||||||| ||||
SERDES lane--+ | ||||| ||||||||||| ||||
Record time Source Dev SL FL vvvvv vvvvvvvvvvv vvvv Remote Dev SL FL CRC Errs Last Change
------------------- ------- --- -- -- ----- ----------- ---- ------- --- -- -- -------- ----------------------
2014-05-18+12:38:17 47=16/1 FE 40 40 I 31= 8/1 FAP 43 11 1 CRC_ERROR
2014-05-18+12:39:27 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 ADMIN_DOWN
2014-05-18+12:39:28 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 EYESCAN_START
2014-05-18+13:14:41 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 EYESCAN_COMPLETE
2014-05-18+13:14:50 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 ADMIN_UP
恢復線路的另一端如下例所示：
card=5, cpu=0, pid=7808, peer_mode=AFIO_IPC_PEER_MODE_DAEMON, sys_dev_id=47=16/1
Fabric Status:
Topology fault(T)----------------+
Far side not expected(*)--------+|
Logically not connected(L)-----+||
Physically not connected(P)---+|||
Rx Down(*)-------------------+||||
Tx Down(*)------------------+||||| NIF Status:
Code Group(G)--------------+|||||| +--------NIF powered off(*)
Misalignment(M)-----------+||||||| |+-------SERDES powered off(*)
Cell Size(C)-------------+|||||||| ||+------Local side down(l)
Internally fixed(I)-----+||||||||| |||+-----Remote side down(r)
Not Accept Cells(A)----+|||||||||| ||||
||||||||||| ||||
SERDES Status: ||||||||||| ||||
Rx power off(*)------+ ||||||||||| ||||
Tx power off(*)-----+| ||||||||||| ||||
Sig not locked(S)--+|| ||||||||||| ||||
Rx signal loss(*)-+||| ||||||||||| ||||
Admin Down(D)----+|||| ||||||||||| ||||
||||| ||||||||||| ||||
Fabric lane-----+ ||||| ||||||||||| ||||
SERDES lane--+ | ||||| ||||||||||| ||||
Record time Source Dev SL FL vvvvv vvvvvvvvvvv vvvv Remote Dev SL FL CRC Errs Last Change
------------------- ------- --- -- -- ----- ----------- ---- ------- --- -- -- -------- ----------------------
2014-05-18+12:38:17 47=16/1 FE 40 40 I 31= 8/1 FAP 43 11 1 CRC_ERROR
2014-05-18+12:39:27 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 ADMIN_DOWN
2014-05-18+12:39:28 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 EYESCAN_START
2014-05-18+13:14:41 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 EYESCAN_COMPLETE
2014-05-18+13:14:50 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 ADMIN_UP
- 在StarOS版本16.1及更高版本中，系統能夠在機箱觀察到配置的出口隊列丟棄(EGQ)閾值時生成SNMP陷阱。此處顯示用於將閾值設定為每30秒週期50個EGQ丟棄的命令示例。
[local]asr5500# config
[local]asr5500(config)# fabric egress drop-threshold enable count 50 interval-secs 30
手動恢復
在Eyescan測試和重新程式設計之後，如果SERDES鏈路未恢復，則有必要進行手動恢復。很遺憾，對於軟體，我們無法確定SERDES鏈路的哪一端有故障。我們必須採取有條理的方法來解決這個問題。
注意：步驟1和2在RMA之前是強制性的
- 首先重新拔插一張卡。從卡中滑出並檢查卡背板中卡和機箱背板上的針腳是否損壞和彎曲。
- 如果觀察到針腳損壞和彎曲，則拍照並向Cisco TAC提出服務請求(SR)。
- 監控72小時。
- 如果問題仍然存在，請轉至步驟2。
- 如果清除，問題會得到解決。
- 重新拔插另一張卡。從卡中滑出並檢查卡背板中卡和機箱背板上的針腳是否損壞和彎曲。
- 如果觀察到針腳損壞和彎曲，則拍照並向Cisco TAC提出服務請求(SR)。
- 監控72小時。
- 如果問題仍然存在，請轉至步驟3。
- 如果清除，問題會得到解決。
- 使用Cisco TAC開啟SR，並連線重新拔插卡之前和之後收集的show support details以及損壞或彎曲針腳的圖片。
問題解決後， show fabric status 如下所示：
[local]ASR5500> show fabric status
Total number of FAPs: 24
Total number of FEs : 8
Total number of SERDES links: 1600
Total number of active SERDES links: 1600
SNMP陷阱 SERDESLanePermenentlyDown 現在已實施以指示何時由於Eyescan故障而永久關閉SERDES通道：
Sun Apr 17 00:05:00 2016 Internal trap notification 1303 (SERDESLanePermanentlyDown) SERDES lane is Down on local: slot 17 device 2 serdes lane index 14, Remote: slot 1 device 1 serdes lane index 40
[local]ASR5500> show fabric status
Total number of FAPs: 16
Total number of FEs : 12
Total number of SERDES links: 1456
Total number of active SERDES links: 1454
Total number of Fabric SERDES with errors: 0
Total number of NIF SERDES with errors : 0
[local]ASR5500> show fabric history
Command: arad system-device-id 1
Command: show serdes all-serdes history
Fabric Status:
+------Not Accept Cells(A)
SERDES Status: |+-----Cell Size(C)
Power off(*)-------------+ ||+----Misalignment(M)
Sig not locked(S)-------+| |||+---Code Group(G)
Admin down(D)----------+|| ||||+--Topology fault(T)
||| |||||
Logical Port---------+ ||| |||||
Fabric lane-----+ | ||| |||||
SERDES lane--+ | | ||| |||||
Record time Source Dev SL FL | vvv vvvvv Remote Dev SL FL CRC Errs Last Change
------------------- ------- --- --- --- --- --- ----- ------- --- --- --- -------- ----------------------
2016-04-16+23:53:05 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - FAULT_DETECTED
2016-04-16+23:53:14 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - ADMIN_DOWN
2016-04-16+23:57:02 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - ADMIN_UP
2016-04-16+23:57:02 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - FAULT_DETECTED
2016-04-16+23:57:11 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - ADMIN_DOWN
2016-04-17+00:00:59 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - ADMIN_UP
2016-04-17+00:00:59 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - FAULT_DETECTED
2016-04-17+00:01:08 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - ADMIN_DOWN
2016-04-17+00:05:00 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - ADMIN_UP
2016-04-17+00:05:00 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - FAULT_DETECTED
...
Command: fe600 system-device-id 42
Command: show serdes all-serdes history
NIF Status: Fabric Status:
Remote side down(r)-------+ +------------Not Accept Cells(A)
Local side down(l)-------+| |+-----------Internally fixed(*)
SERDES powered off(*)---+|| ||+----------Cell Size(C)
NIF powered off(*)-----+||| |||+---------Misalignment(M)
|||| ||||+--------Code Group(G)
SERDES Status: |||| |||||+-------Tx Down(*)
Rx power off(*)------+ |||| ||||||+------Rx Down(*)
Tx power off(*)-----+| |||| |||||||+-----Physically not connected(P)
Sig not locked(S)--+|| |||| ||||||||+----Logically not connected(L)
Rx signal loss(*)-+||| |||| |||||||||+---Far side not expected(*)
Admin down(D)----+|||| |||| ||||||||||+--Topology fault(T)
||||| |||| |||||||||||
Fabric lane-----+ ||||| |||| |||||||||||
SERDES lane--+ | ||||| |||| |||||||||||
Record time Source Dev SL FL vvvvv vvvv vvvvvvvvvvv Remote Dev SL FL CRC Errs Last Change
------------------- ------- --- -- -- ----- ---- ----------- ------- --- --- --- -------- ----------------------
2016-04-16+23:57:01 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - FAULT_DETECTED
2016-04-16+23:57:11 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - ADMIN_DOWN
2016-04-16+23:57:11 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - EYESCAN_START
2016-04-17+00:00:52 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - EYESCAN_FAILURE
2016-04-17+00:00:55 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - ADMIN_UP
2016-04-17+00:00:58 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - FAULT_DETECTED
2016-04-17+00:01:08 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - ADMIN_DOWN
2016-04-17+00:01:08 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - EYESCAN_START
2016-04-17+00:04:56 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - EYESCAN_FAILURE
2016-Apr-17+00:05:00.023 [snmp 22002 info] [5/0/7150 <afctrl:0> trap_api.c:17297] [software internal system syslog] Internal trap notification 1303 (SERDESLanePermanentlyDown) SERDES lane is Down on local: slot 17 device 2 serdes lane index 14, Remote: slot 1 device 1 serdes lane index 40
2016-Apr-17+00:05:00.023 [afctrl 186019 critical] [5/0/7150 <afctrl:0> l_msg_handler.c:1541] [hardware internal system syslog] Fabric device 17/2, serdes lane index 14, (remote fabric device 1/1, serdes lane index 40) is Administratively offline due to excessive calibration failures
2016-Apr-16+23:41:09.247 [system 1009 warning] [6/0/10430 <evlogd:1> evlgd_syslogd.c:162] [software internal system critical-info syslog] CPU[5/0]: afio: afio [5/0/9285] [ 426721.037] afio/afio_fe600_serdes.c:2827: #1: fe600=42=17/2, Fabric SERDES lane transitioned from up to down, serdes=14, devid=1=1/1, serdes=40
2016-Apr-16+23:41:09.247 [system 1009 warning] [5/0/7073 <evlogd:0> evlgd_syslogd.c:162] [software internal system critical-info syslog] CPU[5/0]: afio: afio [5/0/9285] [ 426721.037] afio/afio_fe600_serdes.c:2827: #1: fe600=42=17/2, Fabric SERDES lane transitioned from up to down, serdes=14, devid=1=1/1, serdes=40
相關資訊
修訂記錄
| 修訂 | 發佈日期 | 意見 |
|---|---|---|
1.0 |
19-Aug-2022 |
初始版本 |
意見