Introduction
This document describes how to troubleshoot Fabric and storage Card (FSC) card when "ThreshFabricEGQDiscards" SNMP trap.
Prerequisites
Requirements
Cisco recommends that you have knowledge of these topics:
Components Used
This document is not restricted to specific software and hardware versions.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Problem
Error "ThreshFabricEGQDiscards" is noticed when Single Event Upset (SEU) on the fabric edge (FE) chipset on one FSC card in an ASR5500 chassis. Due to this bit flip in FE tables, the FE chip starts to corrupt packets (cells) in the fabric causing Egress Queue Discards which leads to heartbeat failures between the cards.
You can see example of this problem using the Command line interface (CLI) command show snmp trap history verbose.
Sat Jan 02 03:59:30 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 9 device 2 threshold 50 measured value 2430 interval 30
Sat Jan 02 03:59:30 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 9 device 1 threshold 50 measured value 2096 interval 30
Sat Jan 02 03:59:40 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 5 device 4 threshold 50 measured value 481 interval 30
Sat Jan 02 03:59:40 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 4 device 2 threshold 50 measured value 3761 interval 30
Sat Jan 02 03:59:40 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 4 device 1 threshold 50 measured value 3660 interval 30
Sat Jan 02 03:59:40 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 5 device 2 threshold 50 measured value 173 interval 30
Sat Jan 02 03:59:40 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 5 device 1 threshold 50 measured value 133 interval 30
Sat Jan 02 03:59:42 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 8 device 2 threshold 50 measured value 2977 interval 30
Sat Jan 02 03:59:42 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 8 device 1 threshold 50 measured value 4310 interval 30
Sat Jan 02 03:59:44 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 3 device 1 threshold 50 measured value 4499 interval 30
Sat Jan 02 03:59:44 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 3 device 2 threshold 50 measured value 4091 interval 30
Sat Jan 02 03:59:45 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 10 device 1 threshold 50 measured value 2796 interval 30
Sat Jan 02 03:59:45 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 10 device 2 threshold 50 measured value 5418 interval 30
Sat Jan 02 03:59:47 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 1 device 2 threshold 50 measured value 4747 interval 30
Sat Jan 02 03:59:47 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 1 device 1 threshold 50 measured value 5243 interval 30
Sat Jan 02 03:59:49 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 7 device 2 threshold 50 measured value 4644 interval 30
Sat Jan 02 03:59:49 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 7 device 1 threshold 50 measured value 5017 interval 30
This line is seen under multiple cards cpu consoles:
Note: command debug console card is hidden/test command. This command is also collected every time for all cards on ASR5500 when show support details command is run on StarOs node.
******** debug console card 1 cpu 0 tail 10000 only *******
Saturday January 02 05:45:38 EST 2016
[...]
2016-Jan-02+03:59:47.479 card 1-cpu0: afio [1/0/2701] [2862193.674] afio/afio_petrab_egress.c:121: #1: petrab=1=1/1, PetraB EGQ Egress drop threshold exceeded, drop count=5243, interval=30 secs, threshold=50
Troubleshoot
Check if egress drops are incrementing.
Note: If fabric errors are incrementing and you are running StarOs node on version release 19.0 or higher then proceed to the Solution section in this article.
Note: If fabric errors are incrementing and you are running StarOs node version bellow version release.19.0 please raise Service Reuqest towards TAC.
Step 1. Enter test mode, here is documentation how to enable it on StarOs node.
cli test-commands [encrypted] password password
Step 2. Check fabric health.
show fabric health | grep -i -E "^Petra-B|EGQ"
Example of output when problem is not present:
[local]#show fabric health | grep -i -E "^Petra-B|EGQ"
Petra-B 1=1/1
Petra-B 2=1/2
Petra-B 3=2/1
Petra-B 4=2/2
Petra-B 5=3/1
Petra-B 6=3/2
[...]
Example of output where you see increese in EGQ Discard Packets:
[local]#show fabric health | grep -i -E "^Petra-B|EGQ"
Petra-B 1=1/1
EGQ.RqpDiscardPacketCounter 1143278
EGQ.EhpDiscardPacketCounter 1143278
EGQ.PqpDiscardUnicastPacketCounter 1143278
Petra-B 2=1/2
EGQ.RqpDiscardPacketCounter 1068491
EGQ.EhpDiscardPacketCounter 1068491
EGQ.PqpDiscardUnicastPacketCounter 1068491
[local]#show fabric health | grep -i -E "^Petra-B|EGQ"
Petra-B 1=1/1
EGQ.RqpDiscardPacketCounter 1346022 <<<
EGQ.EhpDiscardPacketCounter 1346022 <<<
EGQ.PqpDiscardUnicastPacketCounter 1346022 <<<
Petra-B 2=1/2
EGQ.RqpDiscardPacketCounter 1271360 <<<
EGQ.EhpDiscardPacketCounter 1271360 <<<
EGQ.PqpDiscardUnicastPacketCounter 1271360 <<<
Solution
Automatic Recovery Mechanism
Type of behavior change:
New CLI command to enable FSC auto recovery/reset procedure upon detecting excessive fabric egress discards
Release introduced:
19.0
Old behavior:
Manual recovery process to reset FSCs.
New behavior:
New CLI configuration commands, please check documentation:
fabric fsc auto-recovery enable max-attempts <X> to enable this feature.
max-attempts is the number of times it resets each FSCs. By default, max-attempts is unlimited.
fabric fsc auto-recovery disable to disable this feature.
show afctrl fsc-auto-recovery displays details about FSC auto recovery, including devices yet to be reset, reset count, max attempts, Egress Drop Threshold state, and FSC auto recovery history.
Caution: Impact on customer: FSC FE devices are reset and any packets in flight are lost.
Note: All values except the history are replicated when the MIO fails over.
Feedback