Field Notice: FN - 70169 - Invalid IPv6 Address Created by NX-OS on Cisco MDS 9706 and 9710 Leads to Bootflash and ISSU/D Issues - Workaround Provided

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.0	09-May-18	Initial Release
2.0	19-Jun-18	Updated All Sections

Products Affected

Affected OS Type	Affected Release	Affected Release Number	Comments
NON-IOS	8	8.1(1)
NON-IOS	7	7.3(1)D1(1), 7.3(1)DY(1)
NON-IOS	6	6.2(21)	Affects all releases 6.2.1 to 6.2.21

Defect Information

Defect ID	Headline
CSCvf99665	ISSU/D fails with bootflash busy err; seg fault mounting bootflash (not mounted)
CSCvb93701	ISSU/D fails on MDS 9706/9710 with 'bootflash busy' error; seg fault while mounting bootflash
CSCvg05230	An invalid IPv6 address may be created on mgmt0 interface on MDS 9706 and MDS 9710
CSCvj40760	ISSU from 6.2(21) to 8.1(1a) is incorrectly disruptive for all FC ports all platforms
CSCvj76876	NVRAM config not saved to standby sup when config is saved

Problem Description

Cisco MDS NX-OS Releases 6.(x) and 7.3(x) might create an invalid IPv6 address on the mgmt0 interface of MDS 9706 and MDS 9710 switches. If an affected switch is upgraded to Cisco MDS NX-OS Release 8.1(1) it will not be able to upgrade (In-Service Software Upgrade, ISSU) or downgrade (In-Service Software Downgrade, ISSD) nondisruptively to any other version of NX-OS.

Background

Cisco MDS 9706 and Cisco MDS 9710 switches that run Cisco MDS NX-OS Release 6.2(x) or 7.3(x) are vulnerable to Cisco bug ID CSCvg05230. This defect causes an invalid IPv6 address to be created in the mgmt0 interface configuration. In these releases the only symptom is that the invalid address cannot be removed by the user.

This defect can then trigger Cisco bug ID CSCvf99665 while running Cisco MDS NX-OS Release 8.1(1). An affected switch cannot complete a nondisruptive upgrade or downgrade. Data switching is unaffected unless a disruptive upgrade or downgrade is attempted. This defect manifests with many symptoms, all related to the bootflash device on either the active or standby supervisor being inaccessible. The root cause of the bootflash issue is the random and invalid IPv6 address and mask installed on the mgmt0 interface by Cisco bug ID CSCvg05230, mentioned previously. This configuration has not been installed by the user, however it might have existed unnoticed for a long time. Due to changes that start in Cisco MDS NX-OS Release 8.1(1) the system behavior is different, which leads to the issues described in Cisco bug ID CSCvf99665.

Problem Symptom

In all affected Cisco MDS NX-OS releases, the signature of this issue is an invalid IPv6 address and mask in the configuration of the mgmt0 interface that the user has not configured. The address has the format of ‘::d.d.d.d’ where:

• ‘d’ is a decimal IP address octet
• there are no octets before the ‘::’ delimiters and the mask is always ‘/0’

switch# show running-config interface mgmt 0
interface mgmt0
  ipv6 address ::1.1.1.255/0

In Cisco MDS NX-OS Release 8.1(1) only, various other symptoms might occur. In the upgrade to Cisco MDS NX-OS Release 8.1(1), the upgrade might abort after the first linecard has been upgraded. While running Cisco MDS NX-OS Release 8.1(1), bootflash specific commands such as dir bootflash: or system health check bootflash will fail. One of these filesystem mount entries for active (/bootflash) or standby (/bootflash_sup-remote) supervisor bootflash will not be present:

switch# show system internal flash | include bootflash
/bootflash                3653168    3031428    436168    88    /dev/sde3
/bootflash_sup-remote     3653168    2880928    586672    84    127.1.1.6:/bootflash/

Workaround/Solution

In general, once a switch has been affected, complete these steps:

1. The invalid IPv6 address must be cleared from the mgmt0 configuration. This will be done automatically and nondisruptively by an ISSU to Cisco MDS NX-OS Release 6.2(23) or 8.1(1a). For other releases, contact the Cisco Technical Assistance Center (TAC) for assistance to clear the address nondisruptively.
2. The configuration must be saved on the active supervisor. This is done with the copy running-config startup-config command.
3. The 2nd supervisor must be made the active supervisor via a switchover with the system switchover command.
4. The configuration must be resaved on the newly active supervisor with the copy running-config startup-config command again.

The purpose of this approach is to save the mgmt0 IPv6 configuration specifically to the NVRAM of each supervisor. The IPv6 NVRAM configuration is only updated on the active supervisor when the switch configuration is saved due to Cisco bug ID CSCvj76876.

If your switch runs Cisco NX-OS Release 8.1(1) and has an invalid IPv6 address on the mgmt0 interface, then ISSU/D is no longer possible. Contact the TAC for help to clear it nondisruptively and restore ISSU/D functionality. Although this issue can be cleared in Cisco NX-OS Release 8.1(1), the recovery process is laborious because it cannot be done through normal NX-OS commands and it is thus preferable to clear it prior to the upgrade to this release. If there is no invalid IPv6 address on the mgmt0 interface, then no action is required.

If your switch runs any Cisco MDS NX-OS 7.3(x) release or any Cisco MDS NX-OS 6.2(13a) or later release except 6.2(21), upgrade directly to Cisco MDS NX-OS Release 8.1(1a) or a later 8.1(x) release. Then switch over to the 2nd supervisor. Once the supervisors reach the active/ha-standby state, save the configuration again. These releases will automatically delete any invalid IPv6 address on the mgmt0 interface which will prevent an occurence of the ISSU or other issues. Any switch that runs Cisco MDS NX-OS Release 6.2(13) or earlier must first be upgraded to Cisco MDS NX-OS Release 6.2(13a) before you start this step.

If your switch runs Cisco MDS NX-OS Release 6.2(21) and there is no invalid IPv6 address on the mgmt0 interface, then upgrade directly to any Cisco MDS NX-OS 8.1(x) release except 8.1(1a). It is not possible to upgrade nondisruptively from Cisco MDS NX-OS 6.2(21) to Cisco MDS NX-OS 8.1(1a) due to Cisco bug ID CSCvj40760. No further action is required.

If your switch runs Cisco NX-OS 6.2(21) and has an invalid IPv6 address on the mgmt0 interface, then it must be corrected before Cisco MDS NX-OS Release 8.1(1) is loaded in order to prevent the ISSU and other issues. To do this, ISSU to Cisco MDS NX-OS Release 6.2(23) in order to remove the invalid IPv6 address automatically. Save the configuration and switch over to the other supervisor. Once the supervisors reach the active/ha-standby state, save the configuration again.

Additional Information

For more information, which includes further details on how to determine if a switch is affected and recovery procedures, see these defects:

• Cisco bug IDs CSCvg05230 and CSCvj76876 for Cisco MDS NX-OS Release 6.2(x) or 7.3(x)
• Cisco bug IDs CSCvf99665, CSCvb93701, CSCvj40760 and CSCvj76876 for Cisco MDS NX-OS Release 8.1(1)

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email
• By telephone

Receive Email Notification For New Field Notices

Cisco Notification Service—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.