Field Notice: FN64329 - APIC SSD Degradation After High Percent Utilization of Solid State Drive - Hardware Upgrade Available

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Products Affected

Defect Information

Problem Description

The endurance of Application Policy Infrastructure Controller (APIC) Solid State Drives (SSDs) is worn out over the course of high usage. This leads to slow SSD writes, and the SSD can become read-only.

When the SSD drive is degraded, it can cause CPU spikes in APIC services.

This problem exists in these two SSD parts: APIC-SD120G0KS2-EV and/or APIC-SD120GBKS4-EV.

Cisco recommends that you replace these SSDs, regardless of percent utilized, with a new Enterprise level SSD - Part Number UCS-SD200G12S3-EP. This part, UCS-SD200G12S3-EP, is included in the upgrade form contained in this field notice.

Background

SSD endurance that exceeds 90% indicates that the system is in a problem state.

Use the CLI commands in this section in order to verify the occurrence of the problem: (Note root level is required to execute this command)

• Enter the MegaCli command in order to find the SSD device, as shown here:

  MegaCli -LdpdInfo -a0 |grep 'Target\|Device Id\|Inquiry' 
  Virtual Drive: 0 (Target Id: 0)
  Device Id: 4
  Inquiry Data: 9XG4F2FAST91000640NS CC03
  Virtual Drive: 1 (Target Id: 1)
  Device Id: 5
  Inquiry Data: 1403039C3336 Micron_P400e-MTFDDAK100MAR 0257

• Enter the smartctl command in order to determine the endurance:

  smartctl -l devstat -i -A -d sat+megaraid,5 /dev/sdb
  smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.4.27.0.1insieme-7] (local build) 
  			
  Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
  			
  === START OF INFORMATION SECTION ===
  			
  7 0x008 1 110 Percentage Used Endurance Indicator

Problem Symptom

The observed processes that run on APIC-L1 or APIC-L2 utilize higher than expected CPU resources and slow down the APIC cluster operations.

The root cause of the symptom was identified as a specific issue of SSD degradation in a large-scale Application Centric Infrastructure (ACI) network managed by APIC-L1 or APIC-L2.

The dmesg error when the SSD is fully worn out and becomes slow and or read-only is shown here:

[492316.851464] Buffer I/O error on dev sdb, logical block 0, async page read
[492316.857818] sdb: unable to read partition table
[492316.860970] sd 0:2:4:0: [sdb] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[492316.860975] sd 0:2:4:0: [sdb] tag#0 CDB: Read(10) 28 00 0d da 77 80 00 00 08 00
[492316.860977] blk_update_request: I/O error, dev sdb, sector 232421248
[492316.866882] sd 0:2:4:0: [sdb] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[492316.866885] sd 0:2:4:0: [sdb] tag#0 CDB: Read(10) 28 00 0d da 77 80 00 00 08 00
[492316.866888] blk_update_request: I/O error, dev sdb, sector 232421248
[492316.872732] Buffer I/O error on dev sdb, logical block 29052656, async page read

Workaround/Solution

Use the link in Upgrade Program Information to order a replacement SSD.

Complete these steps in order to replace the SSD. For details on each step, see Cisco APIC SSD Replacement.

1. Decommission the APIC.
2. If the Cisco Integrated Management Controller (CIMC) version is earlier than 2.0(9c), use Host Upgrade Utility (HUU) to upgrade the CIMC.
3. "Prepare for Removal" of the SSD from CIMC.
   Note: This step is conditional and is required if you already have a SSD installed. Perform this step before you replace an existing SSD with a new SSD. This step is not required for installation of a new SSD.
4. If there is an old SSD, physically remove it and add the new SSD.
5. Use the newly installed SSD to create RAID volume.
6. Install the APIC image with the virtual media. In this step, the SSD is partitioned and the APIC software is installed on the hard drive.
7. Recommission the APIC.

Upgrade Program Information

Click on the following link to open Support Case Manager in a new tab:
https://mycase.cloudapps.cisco.com/fieldnotice?fn=FN64329

1. Serial Numbers (SNs) must be provided and be affected.
   • SN Entitlement Check will be performed.
   • Order entry supports up to 50 SNs per request. If you have more than 50, you will need to submit more than 1 request.
2. One ship to address per request.
3. Service Request number (SR#) is not required, but if you have an Existing SR# please enter it for better tracking purposes.

Revision History

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email or telephone

Receive Email Notification For New Field Notices

My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.