Memory Component Issue

Cisco has been working with individual customers on an issue related to memory components manufactured by a single supplier between 2005 and 2010. These components are widely used across the industry, and have been included in some Cisco products.

Although the majority of Cisco products using these components are experiencing field failure rates below expected levels, the components are known to slowly degrade over time. A handful of our customers have recently experienced a higher number of failures, leading us to change our approach to managing this issue.

Cisco has decided to take a charge of $655 million related to the expected cost of managing these issues, including supporting our fix-on-fail and technology migration programs. We are taking this action to support our customers and partners.

•   •  To learn more about this topic, please read our blog post or click on the above FAQ tab.
•   •  To learn more about the products that may be impacted, please click on the above Focus Products tab.
•   •  To see the latest product advisories, click on the above Field Notices tab.

Q: What is the issue? 
A: Cisco has been working with individual customers on an issue related to memory components manufactured by a single supplier between 2005 and 2010. These components are widely used across the industry. The majority of affected Cisco products are experiencing field failure rates below expected levels:

•   •  Although the components are known to slowly degrade over time, not all components will fail.
•   •  Components that have failed have been in service for at least 2-3 years.
•   •  Component failures may have no immediate impact, but are exposed by a power cycle event.

Q: How are these components used? 
A: These components are commonly used as the main memory of the processor for the operating system. They include memory modules and discrete on-board memory (individual components soldered on a line card or similar printed circuit board assembly).

Q: What causes the issue? 
A: Typically, the memory components are more than 2-3 years old and suffer a functional failure when subjected to a power cycle event (turned off and on). The memory failure is attributed to a degradation mechanism in a specific memory circuit design.

Q: Is this a hard failure that causes a device to fail or a transient failure that could temporarily disrupt normal operations? 
A: We know that the components slowly degrade over time, but they require a power cycle to exhibit a hard failure. What we've seen is that within the population of components affected by this problem, some components may fail earlier than anticipated, while others will not exhibit failure within the component’s expected lifetime.

Component Supplier

Q: Who supplies the impacted memory components? 
A: As a matter of policy, Cisco stands behind the reputation of our products. We do not intend to publicly name the supplier.

Q: What makes you say this is an industry-wide issue? 
A: This issue relates to memory components manufactured by a single supplier between 2005 and 2010. The components are used across the industry and not just in Cisco devices.

Q: How are others in the industry handling this issue? 
A: While different systems vendors have chosen to address this issue in different ways, Cisco believes its approach is the best course of action for its customers. Despite the cost, we are demonstrating that we always make customer satisfaction a top priority.

Cisco Response

Q: Does this issue impact all memory components used in Cisco products? 
A: No, this issue is confined to memory components from specific lots from a specific supplier.

Q: What will happen to the inventory supply of these components? Will they be used in other products? 
A: Product using these components has not been shipped new since ~2012. The Service Depots have been purged of Focus List products, with exception of the SPA cards. Service Depot removal of exposed Non-Focus products is in progress and may take 12-18 months to complete.

Q: How long has Cisco been selling products with this component? 
A: Cisco products with the suspected memory components began shipping in 2005, and this continued until the final component was identified and removed from our inventory in 2012.

Q: How should a customer determine if this memory issue impacts products in their network? 
A: We are providing information on affected product families at www.cisco.com/go/memory, including a list of Focus Products and links to product-specific Field Notices (click on the tabs above for more information).

Q: What failure rates is Cisco seeing? 
A: We continue to closely monitor the data for Cisco products that have experienced field failures or customer escalations. The majority of our products using these components are experiencing field failure rates below expected levels. However a handful of our customers have recently experienced a higher number of failures in their networks, which is why we are changing our approach in managing this issue.

Q: Are these failure rates getting worse? 
A: The current data does not show a significant increase in failure rates, but we do know that the components are known to slowly degrade over time.

Timeline

Q: How and when did Cisco find out about this issue? 
A: Cisco first became aware of this issue in December 2010.

Q: What did Cisco do as a result? 
A: When Cisco first became aware of this issue in December 2010, we:

•   •  Verified the root cause for the failure.
•   •  Disqualified the impacted component from our list of usable components.
•   •  Purged any suspect remaining component inventory from our manufacturing sites.
•   •  Verified the supplier's corrective actions for eliminating the failure source.
•   •  Verified the supplier's changes to their component design verification process to minimize the potential for recurrence of a similar problem.
•   •  Verified that components from other component suppliers do not have the same issue.
•   •  Began ongoing reviews of field returns and field failures data.

Only in late 2012 did field failures and supplier review data point to a potential customer impact. Based on the information available at the time, Cisco created and managed an active inventory management of replacement line cards, and implements a "fix-on-fail" escalation process to support customers.

Advice to Customers

Q: How do I determine if this memory issue impacts products in my network? 
A: As always, customers should contact the Cisco Technical Assistance Center (TAC) if they experience a failure in a Cisco product. Cisco has established a process to help analyze a customer's installed base for exposure. We recommend reviewing the Focus Product list and Field Notices that are available through the above tabs.

Q: Do normal SmartNet and warranty entitlement rules apply? 
A: Yes. Normal SmartNet and warranty entitlement rules remain in place and will be applied by the Cisco TAC if you experience a failure in one of the products listed in the Field Notices. In other circumstances (eg. out of warranty or out or contract), we'd encourage you to raise your concern directly with your Cisco account team.

Q: What are Focus Products? 
A: Cisco has divided the relevant PIDs (orderable Product IDs) into two categories – Focus Products and All Other Products. The categories are based on the use of affected memory components (circuitry, number of components), observed field failure rates (including customer experience), and product replacements since 2012. Focus Products require careful management, while All Other Products have lower than expected failure rates.

Q: What action does Cisco recommend for Focus Products? 
A: Cisco recommends careful management of these devices, including:

•   •  Identifying the number and location of these PIDs in your network
•   •  Reviewing the appropriate Field Notices
•   •  Reviewing any product-related Cisco Security Advisories for recommended upgrade decisions (there are no known security implications associated
       with this issue)
•   •  Closely monitoring potential failure rates and trends in your network
•   •  Minimizing unnecessary power cycling

Cisco's recommendation is fix-on-fail for both product categories to minimize network disruption.

Q: What action does Cisco recommend for All Other Products? 
A: These products have lower than expected failure rates. Cisco's recommendation is fix-on-fail for both product categories to minimize network disruption

Q: Does this issue cause any security-related concerns? 
A: No. Although there are no known security implications associated with this issue, Cisco does recommends customers review the latest Security Advisories at www.cisco.com/go/psirt before making upgrade decisions.

Q: Will you be replacing products, repairing products, or offering discounted upgrades? 
A: Any currently available and covered products that experience an issue will be addressed through fix-on-fail, using normal support processes, or technology migration programs. End of Sale products will only be managed through a fix-on-fail, return-to-factory, or technology migration program. End of Support products will be managed through a technology migration program only.

The Issue

Cisco has been working with individual customers on an issue related to memory components manufactured by a single supplier between 2005 and 2010. These components are widely used across the industry, and have been included in some Cisco products. The majority of affected Cisco products are experiencing field failure rates below expected levels:

•   •  Although the components are known to slowly degrade over time, not all components will fail.
•   •  Components that have failed have been in service for at least 2-3 years.
•   •  Component failures may have no immediate impact, but are exposed by a power cycle event.

Cisco's Response

•   •  Currently available and covered products that experience an issue will be addressed through fix-on-fail or technology migration programs, using
       normal support processes.
•   •  End of Sale products will only be managed through fix-on-fail, return-to-factory, or technology migration programs.
•   •  End of Support products will be managed through a technology migration program only.

Product-by-Product Assessment

Cisco has assessed 105 impacted product families that use the impacted memory component. 95 of these product families have demonstrated lower than expected failure rates.

Cisco has divided the relevant PIDs (orderable Product IDs) into two categories:

•   •  Focus Products – 124 PIDs in 10 product families require careful management.
•   •  All Other Products – 1213 PIDs have lower than expected failure rates.
•   •  Note – These numbers include all orderable PIDs. The related Field Notices include only base level PIDs.
  
  

The categories are based on:

•   •  The use of affected memory components (circuitry, number of components).
•   •  Observed field failure rates, including customer experience.
•   •  Product replacements since 2012.
  
  

Cisco recommends careful management of the Focus Products, including:

•   •  Identifying the number and location of these PIDs in your network
•   •  Reviewing the appropriate Field Notices
•   •  Reviewing any product-related Cisco Security Advisories for recommended upgrade decisions (there are no known security implications associated
       with this issue)
•   •  Closely monitoring potential failure rates and trends in your network
•   •  Minimizing unnecessary power cycling
  
  

Click here to see a list of the 124 PIDs in the Focus Products category. Cisco's recommendation is fix-on-fail for both product categories to minimize network disruption.

Although Field Notices are normally issued for products with greater than expected failure rates, Cisco is providing updates for every product family that includes an impacted memory component. To minimize network disruption, Cisco recommends fix-on-fail for all related products.

Although there are no known security implications associated with this issue, Cisco also recommends customers review the latest Security Advisories at www.cisco.com/go/psirt before making upgrade decisions.

Collaboration Endpoints

Interfaces and Modules

Optical Networking

Routing

Security

Storage Networking

Switching

Unified Communications

Video

Wireless