The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This document describes troubleshoot commands for the Serializer Deserializer (SERDES) lane(link) in the ASR 5500.
The ASR 5500 contains SERDES links between the cards in order to facilitate communication and data paths between the Fabric and Storage Card (FSC), Data Processing Card (DPC), and Management Input/Output (MIO) cards. At times, these SERDES links can go down due to errors or hardware failure.
Commands to investigate the SERDES Lanes of the ASR 5500 chassis:
show support details
, look in the "debug console..." portion for the lines of output:1397273780.205 card 5-cpu0: afio [5/0/7808] [ 80616.933] afio/afio_fe600_serdes.c:3297: #1: fe600=47=16/1, Fabric SERDES lane transitioned from up to down, serdes=29, devid=25=7/1
cli test-commands password <password>
.Caution: Use of this mode can cause significant service interruption
show fabric health
command for an overall picture of the switching fabric.Tip: The same information can be obtained from the show fabric support details
part of the show support details
In the example, there is an issue between DPC card 2 and FSC card 14.
In the output, the fault is reported from source DPC in slot 2 towards FSC in slot 14:
Command: petra-b system-device-id 3
Command: show health
Petra-B 3=2/1
Fabric Status:
Status OK(+)------------------------+
Topology fault(T)------------------+|
Far side not expected(*)----------+||
Logically not connected(L)-------+|||
Physically not connected(P)-----+|||| NIF Status:
Rx Down(*)---------------------+||||| +----------NIF powered off(*)
Tx Down(*)--------------------+|||||| |+---------SERDES powered off(*)
Code Group(G)----------------+||||||| ||+--------Local side down(l)
Misalignment(M)-------------+|||||||| |||+-------Remote side down(r)
Cell Size(C)---------------+||||||||| ||||+------Rx activity(r)
Internally fixed(I)-------+|||||||||| |||||+-----Tx activity(t)
Not Accept Cells(A)------+||||||||||| ||||||+----Status OK(+)
|||||||||||| |||||||
SERDES Status: |||||||||||| |||||||
Status OK(+)-----------+ |||||||||||| |||||||
Rx power off(*)-------+| |||||||||||| |||||||
Tx power off(*)------+|| |||||||||||| |||||||
Sig not locked(S)---+||| |||||||||||| |||||||
Rx signal loss(*)--+|||| |||||||||||| |||||||
Modified Parms(m)-+||||| |||||||||||| |||||||
Admin down(D)----+|||||| |||||||||||| |||||||
||||||| |||||||||||| |||||||
Fabric lane-----+ ||||||| |||||||||||| |||||||
SERDES lane--+ | ||||||| |||||||||||| ||||||| Config
Source Dev SL FL vvvvvvv vvvvvvvvvvvv vvvvvvv Rate Topology CRC Errs Remote Dev SL FL Last Change
------- --- -- -- ------- ------------ ------- ------------ -------- -------- ------- --- -- -- ----------------------
3= 2/1 FAP 47 15 + A M L 6250.00 Mbps - - 43=14/1 FE 82 82 FAULT_DETECTED ***
In the output for the same link in the other direction from FSC card in slot 14 to DPC card in slot 2 the same error is reported:
Command: fe600 system-device-id 43
Command: show health
FE600 43=14/1
Fabric Status:
Status OK(+)------------------------+
Topology fault(T)------------------+|
Far side not expected(*)----------+||
Logically not connected(L)-------+|||
Physically not connected(P)-----+|||| NIF Status:
Rx Down(*)---------------------+||||| +----------NIF powered off(*)
Tx Down(*)--------------------+|||||| |+---------SERDES powered off(*)
Code Group(G)----------------+||||||| ||+--------Local side down(l)
Misalignment(M)-------------+|||||||| |||+-------Remote side down(r)
Cell Size(C)---------------+||||||||| ||||+------Rx activity(r)
Internally fixed(I)-------+|||||||||| |||||+-----Tx activity(t)
Not Accept Cells(A)------+||||||||||| ||||||+----Status OK(+)
|||||||||||| |||||||
SERDES Status: |||||||||||| |||||||
Status OK(+)-----------+ |||||||||||| |||||||
Rx power off(*)-------+| |||||||||||| |||||||
Tx power off(*)------+|| |||||||||||| |||||||
Sig not locked(S)---+||| |||||||||||| |||||||
Rx signal loss(*)--+|||| |||||||||||| |||||||
Modified Parms(m)-+||||| |||||||||||| |||||||
Admin down(D)----+|||||| |||||||||||| |||||||
||||||| |||||||||||| |||||||
Fabric lane-----+ ||||||| |||||||||||| |||||||
SERDES lane--+ | ||||||| |||||||||||| ||||||| Config
Source Dev SL FL vvvvvvv vvvvvvvvvvvv vvvvvvv Rate Topology CRC Errs Remote Dev SL FL Last Change
------- --- -- -- ------- ------------ ------- ------------ -------- -------- ------- --- -- -- ----------------------
43=14/1 FE 82 82 + L T 6250.00 Mbps 3= 2/1 - 3= 2/1 FAP 47 15 FAULT_DETECTED ***
Another type of issue with the SERDES link is the offline state of the link. In the example, the link between DPC card in slot 6 and FSC card in 17 is offline:
23= 6/3 FAP 38 6 D 6250.00 Mbps 50=17/2 1557643 50=17/2 FE 65 65 OFFLINE ***
The total number of active SERDES links and the number of active links is shown in the output of the show fabric status
command. In the shown example, two links were counted down, one for each side of the link. One lane down is not a problem. There is a lot of excess fabric capacity, and a single lane does not impact the throughput. The only issue is if the links continually go up and down because of errors, in which case it is possible that user and control traffic gets dropped, and that is why it is better if the link is taken down.
[local]ASR5500> show fabric status
Total number of FAPs: 24
Total number of FEs : 8
Total number of SERDES links: 1600
Total number of active SERDES links: 1598
Note: There is a lot of excess fabric capacity, and a single lane does not impact the throughput of the chassis.
show serdes all-serdes history
sections of the show fabric support details
Note: The FE (Fabric Element) is the FSC card side. The FAP (Fabric Array Processor) is the DPC and/or MIO card side.
The DPC cards have 2 FAPs, the DPC2 cards have only 1 FAP; the MIO cards have 4 FAPs, and FSCs have 2 FEs.
The format in the output of commands is <card #>/<FAP/FE #>, for example, MIO 5 would have 5/1, 5/2, 5/3, 5/4.
A fully loaded DPC2 chassis would have 28 endpoints: 8 (8 DPC) + 8 (2 MIO * 4) + 12 ( 6 FCS * 2)
The example of the FE side that was restored after automatic recovery is shown:
card=5, cpu=0, pid=7808, peer_mode=AFIO_IPC_PEER_MODE_DAEMON, sys_dev_id=47=16/1
Fabric Status:
Topology fault(T)----------------+
Far side not expected(*)--------+|
Logically not connected(L)-----+||
Physically not connected(P)---+|||
Rx Down(*)-------------------+||||
Tx Down(*)------------------+||||| NIF Status:
Code Group(G)--------------+|||||| +--------NIF powered off(*)
Misalignment(M)-----------+||||||| |+-------SERDES powered off(*)
Cell Size(C)-------------+|||||||| ||+------Local side down(l)
Internally fixed(I)-----+||||||||| |||+-----Remote side down(r)
Not Accept Cells(A)----+|||||||||| ||||
||||||||||| ||||
SERDES Status: ||||||||||| ||||
Rx power off(*)------+ ||||||||||| ||||
Tx power off(*)-----+| ||||||||||| ||||
Sig not locked(S)--+|| ||||||||||| ||||
Rx signal loss(*)-+||| ||||||||||| ||||
Admin Down(D)----+|||| ||||||||||| ||||
||||| ||||||||||| ||||
Fabric lane-----+ ||||| ||||||||||| ||||
SERDES lane--+ | ||||| ||||||||||| ||||
Record time Source Dev SL FL vvvvv vvvvvvvvvvv vvvv Remote Dev SL FL CRC Errs Last Change
------------------- ------- --- -- -- ----- ----------- ---- ------- --- -- -- -------- ----------------------
2014-05-18+12:38:17 47=16/1 FE 40 40 I 31= 8/1 FAP 43 11 1 CRC_ERROR
2014-05-18+12:39:27 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 ADMIN_DOWN
2014-05-18+12:39:28 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 EYESCAN_START
2014-05-18+13:14:41 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 EYESCAN_COMPLETE
2014-05-18+13:14:50 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 ADMIN_UP
Recovery of another side of the line is shown in the example:
card=5, cpu=0, pid=7808, peer_mode=AFIO_IPC_PEER_MODE_DAEMON, sys_dev_id=47=16/1
Fabric Status:
Topology fault(T)----------------+
Far side not expected(*)--------+|
Logically not connected(L)-----+||
Physically not connected(P)---+|||
Rx Down(*)-------------------+||||
Tx Down(*)------------------+||||| NIF Status:
Code Group(G)--------------+|||||| +--------NIF powered off(*)
Misalignment(M)-----------+||||||| |+-------SERDES powered off(*)
Cell Size(C)-------------+|||||||| ||+------Local side down(l)
Internally fixed(I)-----+||||||||| |||+-----Remote side down(r)
Not Accept Cells(A)----+|||||||||| ||||
||||||||||| ||||
SERDES Status: ||||||||||| ||||
Rx power off(*)------+ ||||||||||| ||||
Tx power off(*)-----+| ||||||||||| ||||
Sig not locked(S)--+|| ||||||||||| ||||
Rx signal loss(*)-+||| ||||||||||| ||||
Admin Down(D)----+|||| ||||||||||| ||||
||||| ||||||||||| ||||
Fabric lane-----+ ||||| ||||||||||| ||||
SERDES lane--+ | ||||| ||||||||||| ||||
Record time Source Dev SL FL vvvvv vvvvvvvvvvv vvvv Remote Dev SL FL CRC Errs Last Change
------------------- ------- --- -- -- ----- ----------- ---- ------- --- -- -- -------- ----------------------
2014-05-18+12:38:17 47=16/1 FE 40 40 I 31= 8/1 FAP 43 11 1 CRC_ERROR
2014-05-18+12:39:27 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 ADMIN_DOWN
2014-05-18+12:39:28 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 EYESCAN_START
2014-05-18+13:14:41 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 EYESCAN_COMPLETE
2014-05-18+13:14:50 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 ADMIN_UP
[local]asr5500# config
[local]asr5500(config)# fabric egress drop-threshold enable count 50 interval-secs 30
When a SERDES link has not been restored after the Eyescan testing and reprogramming, then manual recovery is necessary. Unfortunately with software, we are not able to determine which side of the SERDES link is at fault. We have to take a methodical approach to fix the issue.
Caution: Steps 1 and 2 are mandatory before RMA
When the issue is resolved, the show fabric status
looks like this:
[local]ASR5500> show fabric status
Total number of FAPs: 24
Total number of FEs : 8
Total number of SERDES links: 1600
Total number of active SERDES links: 1600
An SNMP trap SERDESLanePermenentlyDown
has now been implemented to indicate when a SERDES lane has gone down permanently due to Eyescan failures:
Sun Apr 17 00:05:00 2016 Internal trap notification 1303 (SERDESLanePermanentlyDown) SERDES lane is Down on local: slot 17 device 2 serdes lane index 14, Remote: slot 1 device 1 serdes lane index 40 [local]ASR5500> show fabric status Total number of FAPs: 16 Total number of FEs : 12 Total number of SERDES links: 1456 Total number of active SERDES links: 1454 Total number of Fabric SERDES with errors: 0 Total number of NIF SERDES with errors : 0 [local]ASR5500> show fabric history Command: arad system-device-id 1 Command: show serdes all-serdes history Fabric Status: +------Not Accept Cells(A) SERDES Status: |+-----Cell Size(C) Power off(*)-------------+ ||+----Misalignment(M) Sig not locked(S)-------+| |||+---Code Group(G) Admin down(D)----------+|| ||||+--Topology fault(T) ||| ||||| Logical Port---------+ ||| ||||| Fabric lane-----+ | ||| ||||| SERDES lane--+ | | ||| ||||| Record time Source Dev SL FL | vvv vvvvv Remote Dev SL FL CRC Errs Last Change ------------------- ------- --- --- --- --- --- ----- ------- --- --- --- -------- ---------------------- 2016-04-16+23:53:05 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - FAULT_DETECTED 2016-04-16+23:53:14 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - ADMIN_DOWN 2016-04-16+23:57:02 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - ADMIN_UP 2016-04-16+23:57:02 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - FAULT_DETECTED 2016-04-16+23:57:11 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - ADMIN_DOWN 2016-04-17+00:00:59 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - ADMIN_UP 2016-04-17+00:00:59 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - FAULT_DETECTED 2016-04-17+00:01:08 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - ADMIN_DOWN 2016-04-17+00:05:00 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - ADMIN_UP 2016-04-17+00:05:00 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - FAULT_DETECTED ... Command: fe600 system-device-id 42 Command: show serdes all-serdes history NIF Status: Fabric Status: Remote side down(r)-------+ +------------Not Accept Cells(A) Local side down(l)-------+| |+-----------Internally fixed(*) SERDES powered off(*)---+|| ||+----------Cell Size(C) NIF powered off(*)-----+||| |||+---------Misalignment(M) |||| ||||+--------Code Group(G) SERDES Status: |||| |||||+-------Tx Down(*) Rx power off(*)------+ |||| ||||||+------Rx Down(*) Tx power off(*)-----+| |||| |||||||+-----Physically not connected(P) Sig not locked(S)--+|| |||| ||||||||+----Logically not connected(L) Rx signal loss(*)-+||| |||| |||||||||+---Far side not expected(*) Admin down(D)----+|||| |||| ||||||||||+--Topology fault(T) ||||| |||| ||||||||||| Fabric lane-----+ ||||| |||| ||||||||||| SERDES lane--+ | ||||| |||| ||||||||||| Record time Source Dev SL FL vvvvv vvvv vvvvvvvvvvv Remote Dev SL FL CRC Errs Last Change ------------------- ------- --- -- -- ----- ---- ----------- ------- --- --- --- -------- ---------------------- 2016-04-16+23:57:01 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - FAULT_DETECTED 2016-04-16+23:57:11 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - ADMIN_DOWN 2016-04-16+23:57:11 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - EYESCAN_START 2016-04-17+00:00:52 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - EYESCAN_FAILURE 2016-04-17+00:00:55 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - ADMIN_UP 2016-04-17+00:00:58 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - FAULT_DETECTED 2016-04-17+00:01:08 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - ADMIN_DOWN 2016-04-17+00:01:08 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - EYESCAN_START 2016-04-17+00:04:56 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - EYESCAN_FAILURE 2016-Apr-17+00:05:00.023 [snmp 22002 info] [5/0/7150 <afctrl:0> trap_api.c:17297] [software internal system syslog] Internal trap notification 1303 (SERDESLanePermanentlyDown) SERDES lane is Down on local: slot 17 device 2 serdes lane index 14, Remote: slot 1 device 1 serdes lane index 40 2016-Apr-17+00:05:00.023 [afctrl 186019 critical] [5/0/7150 <afctrl:0> l_msg_handler.c:1541] [hardware internal system syslog] Fabric device 17/2, serdes lane index 14, (remote fabric device 1/1, serdes lane index 40) is Administratively offline due to excessive calibration failures 2016-Apr-16+23:41:09.247 [system 1009 warning] [6/0/10430 <evlogd:1> evlgd_syslogd.c:162] [software internal system critical-info syslog] CPU[5/0]: afio: afio [5/0/9285] [ 426721.037] afio/afio_fe600_serdes.c:2827: #1: fe600=42=17/2, Fabric SERDES lane transitioned from up to down, serdes=14, devid=1=1/1, serdes=40 2016-Apr-16+23:41:09.247 [system 1009 warning] [5/0/7073 <evlogd:0> evlgd_syslogd.c:162] [software internal system critical-info syslog] CPU[5/0]: afio: afio [5/0/9285] [ 426721.037] afio/afio_fe600_serdes.c:2827: #1: fe600=42=17/2, Fabric SERDES lane transitioned from up to down, serdes=14, devid=1=1/1, serdes=40
Revision | Publish Date | Comments |
---|---|---|
1.0 |
19-Aug-2022 |
Initial Release |