Field Notice: FN - 72199 - Some Tetration 8RU and 39RU Security Appliances Might Fail Reboot or Full Upgrade to Tetration 3.5.1.17 And Earlier Due to Switch SSD Firmware Issue - Software Upgrade Recommended

Available Languages

Updated:June 29, 2021

Document ID:FN72199

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.0	29-Jun-21	Initial Release

Products Affected

Affected Product ID	Comments
TA-N9K-C9372PX-E
TA-N9K-C9372TX-E
TA-CL-G1-39-K9	TA-N9K-C9372TX-E and TA-N9K-C9372PX-E are the only affected PIDs within the Tetration cluster
TA-CL-G1-SFF8-K9	TA-N9K-C9372TX-E and TA-N9K-C9372PX-E are the only affected PIDs within the Tetration cluster

Defect Information

Defect ID	Headline
CSCvy45239	Bootflash in Nexus 9000 Switches in Some Tetration Appliances Becomes Unresponsive after 28K Hours

Problem Description

Due to a flaw in solid-state drive (SSD) firmware, the SSD that is internal to the switches that are contained within some first generation (M4-based) Tetration 8RU and 39RU Security Appliances can go into read-only state after approximately 3.2 years of cumulative operation. When this occurs, the switches will continue to pass traffic normally, but Tetration cluster reboots and full upgrades to Version 3.5.1.17 and earlier might fail.

Background

After 28,224 hours (approximately 3.2 years) of accumulated Power On Hours (POH), a memory buffer overrun condition occurs which triggers the firmware event in the SSD. This event causes the drive to go into a read-only state until it is power-cycled. No data loss will occur when the memory buffer overrun firmware event occurs and the switch will continue to pass traffic normally.

The affected SSDs are used in the switches in some M4-based Tetration 8RU and 39RU Security Appliances.

Problem Symptom

If the affected SSD goes into read-only mode, Tetration cannot save switch configuration updates. Tetration only attempts switch configuration updates during reboots and full upgrades. Therefore, some first generation (M4-based) Tetration 8RU and 39RU Security Appliances might fail reboots and full upgrades to Version 3.5.1.17 and earlier in the switch configuration stage.

Workaround/Solution

Refer to the How to Identify Affected Products section in order to determine if your Tetration cluster is affected. If it is affected, follow the instructions in either the Recommended Solution or Alternative Solution sections.

Recommended Solution - Patch Upgrade

The recommended solution to resolve the issue reported in this field notice is to patch upgrade your Tetration cluster to Version 3.4.1.35 or later or Version 3.5.1.23 or later. For instructions on how to perform a patch upgrade, refer to the Cisco Tetration Upgrade Guide.

Alternative Solution - Signed Script

If some limitation prevents you from patch upgrading your Tetration cluster to a fixed version or if your switches have already hit the read-only condition and you cannot patch upgrade directly to a fixed version, you can upgrade the firmware on the SSDs contained in the switches in your Tetration cluster with these steps:

1. Download the fixed SSD firmware and save it to a convenient location (Cisco.com login required).
2. Log into the Tetration User Interface.
3. In the left-hand menu, navigate to Maintenance > Explore. Note that you will need the Customer Support or Site Admin role assigned to your user account to access this page.
4. On the Maintenance Explorer page, use the drop-down list to change the request method from GET to POST.
5. In the first text box, enter orchestrator.service.consul for the snapshot host.
6. In the second text box, enter runsigned?log2file=true for the snapshot path.

7. In the Body field, paste this PGP-encrypted script exactly as it appears in this step.

Note that the BEGIN PGP MESSAGE and END PGP MESSAGE lines must be included. Copy and paste everything from the first hyphen in the BEGIN PGP MESSAGE line to the last hyphen in the END PGP MESSAGE line.

8. The completed fields should look like this:
9. Once all of the fields are completed properly, click Send at the top right of the page. Within a few seconds, you should see a response at the bottom of the page:
10. Leave the Maintenance Explorer page open in your browser, open a new browser tab, and go to this URL being careful to enter the FQDN for your cluster:

https://<INSERT CLUSTER FQDN>/switch_upgrade/ssdfwupload

You should see a basic interface like this:
11. Click Choose File to select the SSD firmware file you downloaded in step 1, then click Upload to upload the file. If the upload is successful, you should see a message that states "Upload complete." If you upload the wrong file or alter the file such that the MD5 hash changes, the upload will fail. Download the file again, if necessary, and try again.
12. Return to the Maintenance Explorer page in the previous browser tab and replace runsigned?log2file=true in the second text box with cat?args=/local/logs/tetration/snapshot/cmdlogs/snapshot_runsigned_log.txt. Click Send at the top right of the page. Within a few seconds, you should see a log of the progress of the script:
13. Scroll to the bottom of the page to see the end of the log. You should see information similar to this example:

Ensure the script is complete by looking for the log entry "END: N9K m500 SSD upgrade" at the end of the log. If this entry is not present, click Send at the top right of the page every 30 seconds until you see it. Finally, observe the results table, which contains an entry for each switch in your Tetration cluster. The fifth column, labeled "Impacted," should show "No" for each switch. This indicates that the SSDs contained in the switches in your Tetration cluster are no longer impacted by the issue reported in this field notice.

Troubleshoot

If you receive a 401 Unauthorized response in step 9, you might have attempted to run the script more than once or you might have pasted the PGP-encrypted script in step 7 incorrectly. If this happens, first try to proceed from step 10. If you can reach the basic upload interface in step 10, then the previous steps were successful and you can continue. If you cannot reach the basic upload interface in step 10, wait one hour and start again from step 2, and be careful to perform each step only once and to copy and paste the PGP-encrypted script as instructed.

How To Identify Affected Products

Complete these steps to verify whether the SSDs contained in the switches inside your Tetration cluster are affected by this issue:

Log into the Tetration User Interface.
In the left-hand menu, navigate to Maintenance > Explore. Note that you will need the Customer Support or Site Admin role assigned to your user account to access this page.
On the Maintenance Explorer page, use the drop-down list to change the request method from GET to POST.
In the first text box, enter orchestrator.service.consul for the snapshot host.
In the second text box, enter runsigned?log2file=true for the snapshot path.

In the Body field, paste the PGP-encrypted script exactly as it appears in this step. Note that the BEGIN PGP MESSAGE and END PGP MESSAGE lines must be included. Copy and paste everything from the first hyphen in the BEGIN PGP MESSAGE line to the last hyphen in the END PGP MESSAGE line.

The completed fields should look like this:
Once all of the fields are completed properly, click Send at the top right of the page. Within a few seconds, you should see a response at the bottom of the page:
Still on the Maintenance Explorer page, wait 30 seconds and then replace runsigned?log2file=true in the second text box with cat?args=/local/logs/tetration/snapshot/cmdlogs/snapshot_runsigned_log.txt. Click Send at the top right of the page. Within a few seconds, you should see the results of the script at the bottom of the page:

The first column of the results table contains the name of the switch. The second contains the model number of the SSD inside the switch. The third contains the firmware version of the SSD. The fourth contains the number of hours the SSD has been powered on. The last column indicates whether the SSD is impacted by the issue reported in this field notice. If an SSD is impacted, subtract the number in the fourth column from 28,224 to determine how many hours remain before the SSD will hit the condition reported in this field notice.

Troubleshoot

If you receive a 401 Unauthorized response in step 8, you might have attempted to run the script more than once or you might have pasted the PGP-encrypted script in step 6 incorrectly. If this happens, first try step 9 to see if the results are already present. If you see the results table, then steps 1 through 8 have already been executed successfully and you can interpret the results as outlined in step 9. If you do not see the results table, wait one hour and start again from step 1, and be careful to perform each step only once and to copy and paste the PGP-encrypted script as instructed.
If you see syntax errors in the results of the script in step 9, you might have run the script more than once. Wait one hour and start again with step 1, and make sure not to run any of the steps more than once.
If you only see the START line in the results of the script in step 9, the script might not have completed. Wait 30 seconds and perform step 9 again.

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

Receive Email Notification For New Field Notices

My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.