Cisco strives to deliver technologies and services that exceed customers’ expectations and meet rigorous quality and customer experience standards. However recently, Cisco became aware of an issue related to a component manufactured by one supplier that affects some Cisco products. In some units, we have seen the clock signal component degrade over time. Although the Cisco products with this component are currently performing normally, we expect product failures to increase over the years, beginning after the unit has been in operation for approximately 18 months. Although the issue may begin to occur around 18 months in operation, we don’t expect a noticeable increase in failures until year three of runtime. Once the component has failed, the system will stop functioning, will not boot, and is not recoverable. This component is also used by other companies.
We have identified all Cisco products that have this component and worked with the supplier to quickly put a fix in place. All products shipping currently do not have this issue. In order to support those customers and partners with affected products under warranty as of November 16, 2016 or currently covered by any valid hardware services contract dated July 27, 2019 or prior, Cisco will replace affected products proactively.
An affected product that is not covered by a warranty or service contract is subject to Cisco’s Support Reinstatement Policy before a service contract can be purchased. An affected product that is covered by a valid hardware service contract dated after July 27, 2019 is subject to Cisco’s normal process for hardware replacement. Please see the Service and Support Descriptions for more information.
Given the timeframe of potential failures, customers have the flexibility to plan the timing of product replacements around maintenance activities to minimize business disruption. Due to the age-based nature of the failure and the volume of replacements, we will be prioritizing orders based on the products’ time in operation.
About the Issue
Q: What is the issue?
Cisco is aware of an issue related to a component supplied by one supplier. Although the Cisco products with these components are currently performing normally, we expect product failures to increase over the years beginning after the unit has been in operation for approximately 18 months. Once the component has failed, the system will stop functioning, will not boot, and is not recoverable.
Q: When did you become aware of this issue?
Cisco learned about the scope of potential customer impacts due to this issue in late November 2016. Cisco and the supplier identified the root cause, impact, and scope of the issue, created and tested product fixes, and put in place a remediation plan for our customers.
Q: Is this a hazardous issue?
No, there is no risk of fire or other hazards. The only symptoms are that once the component fails the system will stop functioning, will not reboot, and is not recoverable.
Q: Is this a product recall?
No, this is not a product recall. Although the Cisco products with this component are currently performing normally, we expect product failures to increase over the years, beginning after the unit has been in operation for approximately 18 months. Although the issue may begin to occur around 18 months in operation, we don’t expect a noticeable increase in failures until year three of runtime. For customers that determine proactive replacement is required, Cisco is offering to provide replacement products for those products under warranty as of November 16, 2016 or currently covered by any valid hardware services contract dated July 27, 2019 or prior, which have this component.
An affected product that is not covered by a warranty or service contract is subject to Cisco’s Support Reinstatement Policy before a service contract can be purchased. An affected product that is covered by a valid hardware service contract dated after July 27, 2019 is subject to Cisco’s normal process for hardware replacement. Please see the Service and Support Descriptions for more information.
Q: Who supplies the impacted component?
As a matter of policy, Cisco stands behind the reputation of our products. We do not intend to publicly name the supplier.
Q: Is this issue Cisco-specific?
No, other companies also use this component from the supplier in their products.
Q: What other companies are seeing this issue?
Cisco does not comment on other companies.
Q: What Cisco products are affected?
This component is used by other companies, and has also been included in a number of Cisco products including the ISR4300 and ASA product families. The full list of products is available on the Field Notices tab.
Q: Do you expect these products to fail at 18 months in operation?
Although the issue may occur beginning at 18 months in operation, based on information provided by the supplier, we don’t expect a noticeable spike in failures until year three of runtime.
Q: Is the 18 month window for potential increased failure rate based on 18 months after manufacturing, or 18 months after powered on by customer?
18 months refers to the runtime of the device (powered up). Although the issue may begin to occur around 18 months in operation, we don’t expect a noticable increase in failures until year three of runtime.
Q: When does the 18 month span (of the known issue) start?
The 18 month timeframe is referring to accumulated runtime rather than a specific per unit start date, so units that were in storage for a considerable time would be further from the 18 month timing than units from the same shipment that were powered up immediately. Before 18 months there will be no perceptible increase in failure rate so you need not worry. After that point the failure rate increases gradually over a period of years (not months). Although the issue may begin to occur around 18 months in operation, we don’t expect a noticable increase in failures until year three of runtime. Rebooting does not make a difference for this issue.
Q: Is there any way to proactively monitor or assess a device to see if it is nearing failure?
Unfortunately, no, there is no way to monitor and no warning signs.
Q: Is there any indication from a CLI perspective that would indicate a pending failure? Is there any metric that can be monitored that would show a degradation in the clock signal?
Unfortunately, there is no way to monitor via CLI or show commands. Runtime is the only potential indicator of failure.
Q: What failure rates is Cisco seeing?
Although the Cisco products with these components are currently performing normally, we expect product failures to increase over the years, beginning after the unit has been in operation for approximately 18 months.
Q: Which Cisco clients and/or customers are affected?
We do not comment on confidential client or customer information.
Q: How many customers are likely to be impacted? How many customers have already been impacted?
We do not comment on confidential customer information.
Q: What Cisco products are affected?
The list of affected Cisco products along with field notices are available on the Field Notices tab. This component is also used by other companies.
Q: Will Cisco continue to work with this third party?
We do not comment on our supplier relationships.
Q: Will Cisco seek compensation from the supplier for the failed component?
We do not comment on supplier relationships.
Q: Do your competitors use these products?
We cannot speak on behalf of other companies, but the components are used by other companies.
Q: Did Cisco investigate the issue with the supplier?
Yes, we worked closely with our supplier to ensure we offer customers the best solution going forward.
Q: What is Cisco doing to avoid this type of issue in the future?
Cisco has completed a comprehensive review and assessment of the technical and operational contributors to the clock signal component issue. We have taken actions with our supplier to implement alerts to deliver early notification, containment and mitigation of potential issues with the goal of continuous improvement and strengthened quality.
Cisco's Response
Q: Is Cisco replacing only products that have failed?
No, Cisco is offering to provide replacement products for those products under warranty as of November 16, 2016 or currently covered by any valid hardware services contract dated July 27, 2019 or prior, which have this component, even if they have not failed.
Q: How does Cisco plan to fix the affected products? Does Cisco intend to replace them?
To support our customers and partners, Cisco will provide replacement products for those products under warranty as of November 16, 2016 or currently covered by any valid hardware services contract dated July 27, 2019 or prior, which have this component.
An affected product that is not covered by a warranty or service contract is subject to Cisco’s Support Reinstatement Policy before a service contract can be purchased. An affected product that is covered by a valid hardware service contract dated after July 27, 2019 is subject to Cisco’s normal process for hardware replacement. Please see the Service and Support Descriptions for more information.
Q: Will you be replacing products, repairing products, or offering discounted upgrades?
In order to support our customers and partners, Cisco will provide replacement products for those affected products under warranty as of November 16, 2016 or currently covered by any valid hardware services contract dated July 27, 2019 or prior, which have this component.
An affected product that is not covered by a warranty or service contract is subject to Cisco’s Support Reinstatement Policy before a service contract can be purchased. An affected product that is covered by a valid hardware service contract dated after July 27, 2019 is subject to Cisco’s normal process for hardware replacement. Please see the Service and Support Descriptions for more information.
Q: How long will replacements take?
Due to the age-based nature of the failure and the volume of replacements, Cisco will be prioritizing requests based on the products’ time in operation. Therefore, fulfillment requests may be delayed and your request may be fulfilled in multiple shipments.
Q: Will the replacement product be factory sealed new, or reworked equipment?
Cisco will be providing the replacement product from service stock with the exception of the ISA-3000 and UCS-EN120E product families which will be fulfilled via manufacturing. Once the replacement order is initiated, customers will receive either a sales order or an RMA to provide the product. If the product is shipping through an RMA, the replacement product is coming from service stock, while a sales order would indicate that the unit would be coming from manufacturing.
Q: If my products were under warranty as of November 16, 2016, but are no longer covered, can I still get a replacement?
Yes, if your product was under warranty as of November 16, 2016, you are still eligible to replace your products, even if the products are no longer under warranty.
Q: What should customers do if their products are not covered by warranty or service contract?
An affected product that is not covered by a warranty or service contract is subject to Cisco’s Support Reinstatement Policy before a service contract can be purchased. An affected product that is covered by a valid hardware service contract dated after July 27, 2019 is subject to Cisco’s normal process for hardware replacement. Please see the Service and Support Descriptions for more information.
If you need to obtain service coverage for your products, please contact your Partner or Cisco Account Manager or use the Cisco Service Contract Center to create quotes and place orders for Cisco service contracts: http://www.cisco.com/c/en/us/services/ordering/cscc.html
Q: Will Cisco fund the labor to perform onsite replacements?
The funding for this replacement program is being used to proactively offer replacement products, even before the product fails for those products under warranty as of November 16, 2016 or currently covered by any valid hardware services contract dated July 27, 2019 or prior.
We believe this focus on providing the best quality products for all Cisco customers, is the right one. However, due to this, we are unable to fund onsite services to replace affected devices.
Q: Please advise if there is custom clearing/import requirement for the customer for receiving replacement equipment and, if so, who will bear the cost?
For those countries with VAT where Cisco has local service return depots, Cisco put in place a process to ship the products through the local Cisco depots so importation and customs charges are borne by Cisco.
For other countries, the costs will be the same as an RMA delivered to replace a failed device via customers support contracts.
Q: Will Smartnet contracts be automatically updated with the new serial numbers of replacement devices?
Yes – the serial number swap in service contracts would occur just as they do for typical RMA processes. Serial number swap on entitlement will occur when all the affected/replaced devices on a RMA are received.
Q: Will the replacement device come with a warranty?
The replacement device adopts whatever warranty time was remaining on the affected unit. In the event that the warranty on the affected unit had expired by the time the replacement device was received, a warranty period of 30 days will be in effect.
Q: For products which have a license that is unique to a serial number (such as the APPX license on ISR routers), how can customers proactively obtain a new license prior to replacement to avoid longer impact to the replacement process?
The process would be the same as during a normal fix-on-failure RMA. Whatever needs to be done from a licensing perspective would apply in this situation as well.
Q: What will happen to the inventory supply of these components? Will they be used in other products?
The component causing the issue are not in products that are shipping currently, and will not be used moving forward.
What Products are Affected
Q: What is the easiest way to determine whether a serial number is affected or not?
To validate whether a serial number is affected use the validation tool link within each Field Notice. Go to the Field Notices tab, open the appropriate Field Notice, and then click on the Cisco Serial Number Validation Tool/tab link under How to Identify Hardware Levels. Then, complete the Replacement Product Order Spreadsheet. Please note, this process is for Cisco products only.
Q: After viewing the Serial Number Validation Tool in the Field Notices, a customer is able to determine whether or not their devices are affected. Is there also a check done at that time as to whether or not the product(s) are covered by warranty or a valid maintenance contract?
Once you have validated whether a serial number is affected using the validation tool in the field notice, complete the Replacement Product Order Spreadsheet. After the spreadsheet is submitted to the alias, the team will check the entitlement and warranty status to determine which of the items is eligible (validation is not taking place in the spreadsheet), and the customer would be informed at that time.
Q: If the existing router has components such as WIC, memory upgrades, PVDM, SSD, etc. installed, will the replacement come with these as well or do they need to be swapped from the existing hardware?
They will need to be swapped from the existing hardware.
Q: Is this issue affecting any other bundle SKUs, not listed on the field notices (for example, voice gateways on the ISR4ks or security bundles on the ASAs)?
The product IDs listed in the field notices are the base SKUs of the products, so related bundles are exposed as well. The bundled SKUs are posted on the Field Notices tab on the cisco.com website for this issue. The best way to determine exposure is to go to www.cisco.com/c/en/us/support/web/clock-signal.html#~order and download the Replacement Product Order Spreadsheet. Once the product ID and serial number of a given product is entered, column Y will indicate whether or not it was exposed at time of shipment.
Q: When replacements are ordered, will all components, memory, and software be pre-installed or do customers need to move parts from the faulty device to the replacement?
Generally speaking, only the chassis will ship, not a fully configured, custom device although this would depend on the product ID being replaced. Cisco recommends examining the Sales Order or RMA information once it is created to confirm what will ship and what will be required to make the device whole again.
Q: What should I do if I have a failure and this component issue impacts my network?
As always, customers should contact the Cisco Technical Assistance Center (TAC) if they experience a failure in a Cisco product. Customers can review the field notices, which contain information on identifying any potentially impacted products.
Q: Is the replacement valid for the devices covered with software application support plus upgrades (SAU) contracts?
We would require customers to have a hardware service contract and not just the software support options to obtain replacement product under this program. If the device was under warranty as of November 16, 2016 or currently covered by any valid hardware services contract dated July 27, 2019 or prior they can obtain a replacement even if it is no longer covered.
An affected product that is not covered by a warranty or service contract is subject to Cisco’s Support Reinstatement Policy before a service contract can be purchased. An affected product that is covered by a valid hardware service contract dated after July 27, 2019 is subject to Cisco’s normal process for hardware replacement. Please see the Service and Support Descriptions for more information.
Customer Support
Q: Can Cisco arrange field support engineer for the replacement?
The funding for this replacement program is being used in order to proactively offer replacement products, even before the product fails. We believe this focus on providing the best quality products for all Cisco customers is the right one. However, due to this, we are unable to fund onsite services to replace affected devices. This limitation applies to customers with On-site Support (OSS) contracts. Unless otherwise stated, OSS contracts do not apply to proactive replacements, and any service requests requesting field engineering support will be denied.
Q: Are replacements going to be shipped to the address listed on the Replacement Order Form on a per S/N basis or are they going to be shipped to a single location per customer?
They will be shipped to the location on the SN line item.
Q: If a customer has some devices delivered to a central site then transported to the various sites around the world, will Cisco pay all the costs of shipping and how?
Cisco will pay to ship the devices one time. If a customer chooses to have shipments delivered to a central location, the customer will need to cover any shipping costs to other locations.
Q: In the “Customer Contact Information” section of the “Replacement_Order_form.xls” – should we enter the LCON (local contact) information or the actual customer info?
For the contact information, we would recommend the individual who will be able to address any overall logistical questions, scheduling, etc.
Q: My company has units overseas. Is Cisco going to ship the replacement units to those countries?
We will honor entitled replacements globally. However, there may be longer delays getting product delivered into some countries – just as in case of any product shipment.
Q: When replacements are ordered, will all components, memory, and software be pre-installed or do customers need to move parts from the faulty device to the replacement?
Generally speaking, only the chassis will ship, not a fully configured, custom device although this would depend on the product ID being replaced. Cisco recommends examining the Sales Order or RMA information once it is created to confirm what will ship and what will be required to make the device whole again.
Q: Will Smartnet contracts be automatically updated with the new serial numbers of replacement devices?
Yes – the serial number swap in service contracts would occur just as they do for typical RMA processes. Serial number swap on entitlement will occur when all the affected/replaced devices on a RMA are received.
Q: How will the licensing will be handled?
Licenses will be handled in the same fashion as traditional RMA and will not ship with the product. You may need to contact Global Licensing Operations (GLO) support team to re-host any existing licenses to the replacement unit as they are node locked via traditional PAK-based licensing using our 800# or via the licensing@cisco.com email alias. If you have Advanced Services or a TS Advantage contract a HTOM or FE could help with this.
Q: For products which have a license that is unique to a serial number (such as the APPX license on ISR routers), how can customers proactively obtain a new license prior to replacement to avoid longer impact to the replacement process?
The process would be the same as during a normal fix-on-failure RMA. Whatever needs to be done from a licensing perspective would apply in this situation as well.
Q: : How will Support contracts be handled?
Support contracts will be migrated; you may need some support from your service sales team with the clean up process, but as far as possible it will be automated based on the order form.
Q: What is the process for returning the affected devices back to Cisco?
It will be the same as with an RMA. Once you receive your new gear, you will be provided an RMA instruction to return the affected product.
Q: Once a replacement hardware has been provided, how long does the customer have to ship the original hardware back to Cisco?
We expect return of affected equipment within 30 days of receipt of the replacement equipment; however, we will work with customers to extend return times if needed.
Q: Once replaced, does Cisco pay the return cost of shipping the items back to Cisco from the various site around the world and how?
The costs will be the same as a RMA delivered to replace a failed device via customers support contracts. Return instructions will be provided when the new equipment is sent.
Q: I have a question not addressed here. Where do I go for an answer?
Please send any additional questions to component-questions@cisco.com.
To support our customers and partners, Cisco will provide replacement products for those affected products under warranty as of November 16, 2016 or currently covered by any valid hardware services contract dated July 27, 2019 or prior, which have this component.
An affected product that is not covered by a warranty or service contract is subject to Cisco’s Support Reinstatement Policy before a service contract can be purchased. An affected product that is covered by a valid hardware service contract dated after July 27, 2019 is subject to Cisco’s normal process for hardware replacement. Please see the Service and Support Descriptions for more information.
Be sure to review the information within the field notices and complete an order form on the Ordering Information tab. Due to the age-based nature of the failure and the volume of replacements, we will be prioritizing orders based on the products’ time in operation. Therefore, fulfillment requests may be delayed and your request may be fulfilled in multiple shipments.
View the full list of product IDs that have the affected component.
Although there are no known security implications associated with this issue, Cisco also recommends customers review the latest Security Advisories at www.cisco.com/go/psirt.
Optical Networking
| Title | Field Notice | |
|---|---|---|
| FN - 64230 - NCS1K-CNTLR Might Fail After 18 Months or Longer Due to Clock Signal Component Failure - Replace on Failure | http://www.cisco.com/c/en/us/support/docs/field-notices/642/fn64230.html |
Routing
| Title | Field Notice | |
|---|---|---|
| FN - 64231 - NCS5500 Line Cards Might Fail After 18 Months or Longer Due to Clock Signal Component Failure - Replace on Failure | http://www.cisco.com/c/en/us/support/docs/field-notices/642/fn64231.html | |
| FN - 64252 - IR809/IR829 Industrial Integrated Services Routers Might Fail After 18 Months or Longer Due to Clock Signal Component Failure - Replace on Failure | http://www.cisco.com/c/en/us/support/docs/field-notices/642/fn64252.html | |
| FN - 64253 - ISR4331, ISR4321, ISR4351 and UCS-E120 Might Fail After 18 Months or Longer Due to Clock Signal Component Failure - Replace on Failure | http://www.cisco.com/c/en/us/support/docs/field-notices/642/fn64253.html ISR4321, 4331 & 44351 Replacement Video Tutorial |
Security
| Title | Field Notice | |
|---|---|---|
| FN - 64228 - ASA 5506, ASA 5506W, ASA 5506H, ASA 5508, and ASA 5516 Might Fail After 18 Months or Longer Due to Clock Signal Component Failure - Replace on Failure | http://www.cisco.com/c/en/us/support/docs/field-notices/642/fn64228.html ASA 5506X, 5508X & 5516X Replacement Video Tutorial |
|
| FN - 64250 - Cisco ISA3000 Industrial Security Appliance Might Fail After 18 Months or Longer Due to Clock Signal Component Failure - Replace on Failure | http://www.cisco.com/c/en/us/support/docs/field-notices/642/fn64250.html | |
| Meraki Notification - MX 84, Clock Signal Component Issue | https://meraki.cisco.com/blog/clock-signal-component-issue/ |
Switches
| Title | Field Notice | |
|---|---|---|
| FN - 64251 - Nexus 9000 Series N9K-C9504-FM-E/N9K-C9508-FM-E/N9K-X9732C-EX Might Fail After 18 Months or Longer Due to Clock Signal Component Failure - Replace on Failure | http://www.cisco.com/c/en/us/support/docs/field-notices/642/fn64251.html | |
| Meraki Notification - MS350 Series, Clock Signal Component Issue | https://meraki.cisco.com/blog/clock-signal-component-issue/ |
For proactive product replacements, to validate whether a serial number is affected use the validation tool link within each Field Notice. Go to the Field Notices tab, open the appropriate Field Notice, and then click on the Cisco Serial Number Validation Tool/tab link under How to Identify Hardware Levels. Then, complete the Replacement Product Order Spreadsheet. Please note, this process is for Cisco products only.
For Meraki products, go to https://meraki.cisco.com/blog/clock-signal-component-issue/ for more information.
Please send the completed replacement order spreadsheet to clock-signal-replacement-request@cisco.com. Once Cisco receives the request, we will validate that the provided Serial Numbers contain this clock signal component and are entitled to be replaced. A case number will be sent to you to track your request.
Be aware, Cisco is offering to provide replacement products for those products under warranty as of November 16, 2016 or currently covered by any valid hardware services contract dated July 27, 2019 or prior, which have this component.
An affected product that is not covered by a warranty or service contract is subject to Cisco’s Support Reinstatement Policy before a service contract can be purchased. An affected product that is covered by a valid hardware service contract dated after July 27, 2019 is subject to Cisco’s normal process for hardware replacement. Please see the Service and Support Descriptions for more information.
NOTE: Attachments sent via email to clock-signal-replacement-request@cisco.com might not be secure in transit. Cisco offers an alternate file transfer method via shares.cisco.com and the Aspera platform. In order to learn more about using this method, please click here.
Due to the age-based nature of the failure and the volume of replacements, Cisco will be prioritizing requests based on the products’ time in operation. Therefore, fulfillment requests may be delayed and your request may be fulfilled in multiple shipments. Note that proactive replacement orders do NOT adhere to normal SLAs or Service Contracts.
Once replacement units are ready to be delivered to you, you will receive either a Sales Order Number or an RMA number for the shipment.
| • | If you receive a Sales Order Number for the shipment of your replacement units, please refer to the Sales Order Status Tool (you must have a CCO User ID and password to access this site): https://cisco-apps.cisco.com/cisco/psn/commerce. You will also receive a Return Only RMA to return the impacted product to Cisco. |
| • | If you receive an RMA Number for the shipment of your replacement units, please refer to the "Service Order QuickSearch" Tool at the following location. (Please note: you must have a CCO User ID and password to access this site): http://tools.cisco.com/support/serviceordertool/home.svo |
Please be sure to return your affected equipment within 30 days of receiving your replacement. You can use the following tool to help return impacted units to Cisco: http://www.cisco.com/web/ordering/cs_info/or3/o32/Return_a_Product/WebReturns/product_Online_web_returns.html.
Send any general questions to component-questions@cisco.com.
NOTE: Cisco cannot ship to P.O. Boxes with the exception of U.S. military sites.
If your “Ship To” address is in the following countries, please expect additional delays depending on importation regulations: All LATAM countries, All APAC countries EXCEPT Australia, New Zealand, Hong Kong, Singapore, and Japan, All non EU Countries (for example: UAE, Russia, Turkey, etc.).