Field Notice: FN - 64117 - A9K-8X100G and A9K-4X100G Line Cards Might Experience a Power Sequencer Failure in Certain ASR-9922 Configurations - Software Upgrade Recommended

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.0	29-Apr-16	Initial Release
10.0	13-Oct-17	Migration to new field notice system

Products Affected

Affected Product ID	Comments
A99-8X100GE-CM
A99-8X100GE-CM=
A99-8X100GE-SE
A99-8X100GE-SE=
A99-8X100GE-TR
A99-8X100GE-TR=
A9K-4PT-CM-SEL2P
A9K-4PT-CM-SEL2P=
A9K-4PT-CM-SEL3P
A9K-4PT-CM-SEL3P=
A9K-4PT-CM-TRL2P
A9K-4PT-CM-TRL2P=
A9K-4PT-CM-TRL3P
A9K-4PT-CM-TRL3P=
A9K-4X100-SE-2P-A=
A9K-4X100GE-SE
A9K-4X100GE-SE-A=
A9K-4X100GE-TR
A9K-8HG-200G-SE=
A9K-8HG-200G-TR=
A9K-8HG-2P-SE-OTN=
A9K-8HG-2P-TR-OTN=
A9K-8HG-400G-SE=
A9K-8HG-400G-TR=
A9K-8PT-CM-SEL2P
A9K-8PT-CM-SEL2P=
A9K-8PT-CM-SEL3P
A9K-8PT-CM-SEL3P=
A9K-8PT-CM-TRL2P
A9K-8PT-CM-TRL2P=
A9K-8PT-CM-TRL3P
A9K-8X100G-LB-SE
A9K-8X100G-LB-SE=
A9K-8X100G-LB-TR
A9K-8X100G-LB-TR=
A9K-8X100GE-CM
A9K-8X100GE-CM=
A9K-8X100GE-L-SE
A9K-8X100GE-L-SE=
A9K-8X100GE-SE
A9K-8X100GE-SE=
A9K-8X100GE-TR
A9K-8X100GE-TR=

Defect Information

Defect ID	Headline
CSCuy53875	Increase default FAN speed on asr9922 to improve cooling on A9K-8X100G -4X100G Line Cards

Problem Description

The Tomahawk line cards might experience a cooling issue when installed in an ASR-9922 chassis, which results in a line card failure/shutdown with this syslog error message:

RP/0/RP0/CPU0:Feb 25 00:44:57.277 GMT: canb-server[152]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/0/CPU0 , Power Sequencer Failure (0x09000000)

This syslog error message is a high level error message which can be triggered for different reasons. If this error message is triggered and these three conditions are met you might encounter this problem:

1. The chassis is an ASR-9922.

2. An A9K and A99 8X100GE and/or A9K-4X100GE line card is deployed in the ASR-9922 chassis.

3. The chassis runs with a fan speed of approximately 3600 RPM.

Example of a Syslog Message Where This Issue Is Not Hit

If the syslog error message listed in the Problem Description section is seen after a router reload, a line card reset, or Software Maintenance Upgrade (SMU) and the A9K A99 8X100GE and/or A9K-4X100GE line cards have booted up normally, the line cards are most likely "not" affected.

Background

Fans in the ASR-9922 router that run at default speed might not provide sufficient airflow to the Tomahawk (A9K and A99 8X100GE-xx and A9K-4X100GE-xx) line cards. As a result, the power brick on these line cards might operate at a temperature higher than recommended which could lead to a line card failure (shut down).

 

This issue is applicable only if all of these conditions are met:

 

1. The chassis is an ASR-9922.

2. An A9K-8X100GE and/or A9K-4X100GE line card is deployed in the ASR-9922 chassis.

3. The chassis runs with a fan speed of approximately 3600 RPM.

4. Line card failure/shutdown occurs with this syslog error message:

 

RP/0/RP0/CPU0:Feb 25 00:44:57.277 GMT: canb-server[152]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/0/CPU0 , Power Sequencer Failure (0x09000000)

Problem Symptom

The A9K and A99 8X100GE-xx and/or A9K-4X100GE-xx line cards might fail (shut down) with this syslog error message:

RP/0/RP0/CPU0:Feb 25 00:44:57.277 GMT: canb-server[152]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/0/CPU0 , Power Sequencer Failure (0x09000000)

Workaround/Solution

If you run an ASR-9922 with A9K or A99 8X100GE and/or A9K-4X100GE line card(s), the steps in this section should be performed.

If the A9K and A99 8X100GE or A9K-4X100GE line card fails/shuts down with the previously listed error message, return the failed line card on a Return Material Authorization (RMA) and install the SMU before you install the replacement line card.

In order to prevent potential failures, Cisco recommends to immediately install the appropriate SMU. If you are unable to immediately install the SMU, complete these steps in order to reconfigure the FAN speed via the Altitude setting:

Important Note:The altitude reconfiguration must be removed before you install the SMU.

Long Term Solution: Install the production SMU.

The Cisco IOS XR 6.0.1 CCO release will contain this fix, therefore the altitude reconfiguration must be removed before you upgrade to Release 6.0.1.

Production SMU Availability and Installation Steps

Software	SMU Link	Availability
5.3.1	asr9k-px-5.3.1.CSCuy53875.tar	Currently Available
5.3.2	asr9k-px-5.3.2.CSCuy53875.tar	Currently Available
5.3.3	asr9k-px-5.3.3.CSCuy53875.tar	Currently Available

1. Download the SMU to router.
2. Enter the admin show environment fans command in order to check the FAN speed. Make a note of the FAN speed.
3. Enter the admin show run | include altitude command in order to check the current altitude configuration.

Current altitude config:
RP/0/RSP0/CPU0:asr9k-2#admin show run | in altitude
Fri Apr 15 17:44:55.713 UTC
Building configuration...
environment altitude 2000 rack 0
RP/0/RSP0/CPU0:asr9k-2#

4. Remove the altitude configuration (admin config no environment altitude, that is, the value from the output in the previous step).

Remove altitude config:
RP/0/RSP0/CPU0:asr9k-2#admin config
Fri Apr 15 17:45:03.509 UTC
RP/0/RSP0/CPU0:asr9k-2(admin-config)#no environment altitude 2000 rack 0
RP/0/RSP0/CPU0:asr9k-2(admin-config)#
RP/0/RSP0/CPU0:asr9k-2(admin-config)#commit
Fri Apr 15 17:45:14.454 UTC
RP/0/RSP0/CPU0:asr9k-2(admin-config)#end
RP/0/RSP0/CPU0:asr9k-2#

Verify altitude config is removed:
RP/0/RSP0/CPU0:asr9k-2#admin show run | in altitude
Fri Apr 15 17:45:18.116 UTC
Building configuration...
RP/0/RSP0/CPU0:asr9k-2#

5. Activate the SMU.
6. If the SMU fails to activate, enter the admin config environment altitude <> rack <> command in order to revert the altitude configuration (altitude value can be found in the step 3 output).
7. Enter the admin show environment fans command in order to check the FAN speed after 5 to 10 minutes of SMU activation. Fan speed should be around 5400 if the ASR-9922 chassis has 8x100 or 4x100 line cards.

If you are unable to immediately install the SMU, complete these steps:

1. Immediately reconfigure the fan speed on the ASR-9922 router if the fan speed is less than 5400 RPM.
2. Enter the admin config environment altitude command in order to set the altitude value, based on the Ambient Temperature Table shown next, on all ASR-9922 routers which have a A9K-8X100GE or A9K-4X100GE line card.

Note: Altitude value is the only configurable value which has an effect on fan speed.

Important Note: The altitude reconfiguration must be removed before you install the SMU.

Ambient Temperature Table

Example: For Ambient Temp of 30 degrees configure RP2 Altitude value to 3000 and RP1 to 1500.
 

Ambient Temperature Average of Active RP inlet temp and standby RP inlet temp	Recommended Altitude values for RP2	Recommended Altitude values for RP1
20	4000	4000
21	4000	4000
22	4000	3800
23	4000	3500
24	4000	3200
25	4000	2900
26	4000	2600
27	3800	2300
28	3500	2100
29	3200	1800
30	3000	1500
31	2600	1200
32	2400	900
33	2100	600
34	1800	350
35	1500	100
36	1200	No Action
37	900	No Action
38	600	No Action
39	350	No Action
40	100	No Action

Reconfigure Fan Instructions

Apply this workaround only if the fan speed is less than 5400 RPM on the ASR-9922 router and A9K-8X100GE and/or A9K-4X100GE are installed.

1. Check the current altitude configuration.

RP/0/RP0/CPU0:ASR9922-B#admin show environment altitude
Altitude:
*0/**/*
112 meters above sea level <===== 112 meters
RP/0/RP0/CPU0:ASR9922-B#

2. Check the fan speed. This output shows that fan speed is around 3600 which is the lowest.

RP/0/RP0/CPU0:ASR9922-B#admin show environment fans
Fan speed (rpm) and run time (in hours) :

FAN0 FAN1 FAN2 FAN3 FAN4 FAN5 FAN6 FAN7 FAN8 FAN9 FAN10 FAN11
0/FT0/* (Speed)
3624 3609 3694 3568 3533 3567 3553 3403 3625 3559 3569 3407
0/FT0/* (Run Time)
4493 4493 4493 4493 4493 4493 4493 4493 4493 4493 4493 4493
0/FT1/* (Speed)
3719 3551 3551 3621 3563 3554 3603 3539 3536 3501 3555 3546
0/FT1/* (Run Time)
4495 4495 4495 4495 4495 4495 4495 4495 4495 4495 4495 4495
0/FT2/* (Speed)
3576 3593 3568 3543 3607 3562 3635 3568 3552 3540 3546 3486
0/FT2/* (Run Time)
4494 4494 4494 4494 4494 4494 4494 4494 4494 4494 4494 4494
0/FT3/* (Speed)
3639 3655 3565 3491 3630 3638 3561 3578 3567 3466 3568 3632
0/FT3/* (Run Time)
4484 4484 4484 4484 4484 4484 4484 4484 4484 4484 4484 4484
RP/0/RP0/CPU0:ASR9922-B#

3. Configure the altitude value based on the "Ambient Temp Table" above.

RP/0/RSP0/CPU0:asr9k-3#
RP/0/RSP0/CPU0:asr9k-3#admin config
RP/0/RSP0/CPU0:asr9k-3(admin-config)#environment altitude XXXX rack 0 (Note: XXXX is the value derived from the Ambient Temp Table above)
RP/0/RSP0/CPU0:asr9k-3(admin-config)#commit
RP/0/RSP0/CPU0:asr9k-3(admin-config)#end
RP/0/RSP0/CPU0:asr9k-3#

4. Confirm that the altitude configuration is modified.

RP/0/RP0/CPU0:ASR9922-B#admin show environment altitude
Mon Mar 7 11:52:57.053 UTC
Altitude:
*0/**/*
XXXX meters above sea level <==== Value changed to what was configured in Step 3.
RP/0/RP0/CPU0:ASR9922-B#

5. Check the fan speed after the altitude configuration change (wait for five minutes before you check the fan RPM).

RP/0/RP0/CPU0:ASR9922-B#admin show environment fans
Mon Mar 7 12:05:03.612 UTC
Fan speed (rpm) and run time (in hours) :
FAN0 FAN1 FAN2 FAN3 FAN4 FAN5 FAN6 FAN7 FAN8 FAN9 FAN10 FAN11
0/FT0/* (Speed)
5624 5675 5657 5563 5525 5559 5542 5679 5650 5562 5594 5644 <== Speed changed from 3600 to 5600
0/FT0/* (Run Time)
4493 4493 4493 4493 4493 4493 4493 4493 4493 4493 4493 4493
0/FT1/* (Speed)
5533 5534 5551 5616 5581 5530 5580 5530 5657 5681 5528 5533
0/FT1/* (Run Time)
4495 4495 4495 4495 4495 4495 4495 4495 4495 4495 4495 4495
0/FT2/* (Speed)
5565 5576 5550 5508 5599 5546 5638 5562 5545 5519 5543 5520
0/FT2/* (Run Time)
4494 4494 4494 4494 4494 4494 4494 4494 4494 4494 4494 4494
0/FT3/* (Speed)
5631 5634 5547 5679 5610 5640 5544 5500 5535 5656 5568 5620
0/FT3/* (Run Time)
4485 4485 4485 4485 4485 4485 4485 4485 4485 4485 4485 4485
RP/0/RP0/CPU0:ASR9922-B# 

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email
• By telephone

Receive Email Notification For New Field Notices

Cisco Notification Service—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.