-
When upgrading from Cisco NX-OS Release 9.3(3) to Cisco NX-OS Release 9.3(6), if you do not retain configurations of the TRM
enabled VRFs from Cisco NX-OS Release 9.3(3), or if you create new VRFs after the upgrade, the auto-generation of ip multicast multipath s-g-hash next-hop-based CLI, when feature ngmvpn is enabled, will not happen. You must enable the CLI manually for each TRM enabled VRF. For the configuration instructions,
see Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 9.3(x).
-
When you use install all with no-reload option, the saved configuration cannot be used before you reload the device. Saving configuration in this state can result
in incorrect start-up configuration once you reload the device with new version of NX-OS.
-
When you upgrade a Cisco Nexus 9000 device to Cisco NX-OS Release 9.3(x), if a QSFP port is configured with the manual breakout
command and is using a QSA, the configuration of the interface Ethernet 1/50/1 is no longer supported and must be removed.
To restore the configuration, you must manually configure the interface Ethernet 1/50 on the device.
-
Enhanced ISSU: Non-disruptive enhanced ISSU to Cisco NX-OS Release 9.3(x) is not supported as there are kernel fixes that
cannot take effect without reloading the underlying kernel. The upgrade will be disruptive. However, a non-disruptive enhanced
ISSU from Cisco NX-OS Release 9.3(x) to later releases is supported in fallback mode only, even in cases of kernel incompatibility.
-
When upgrading from Cisco NX-OS Release 9.2(2) or earlier releases to Cisco NX-OS Release 9.3(x), you need to make sure that
ingress RACL TCAM region is not more than 50% full. Otherwise, the atomic update feature will be enabled after the upgrade
and interfaces with RACLs that exceed 50% of TCAM allocation will remain down.
-
When upgrading from Cisco NX-OS Release 9.2(4) or earlier releases to Cisco NX-OS Release 9.3(4) or later, running configuration
contains extra TCAM configuration lines. You can ignore these extra lines as they do not have an effect on the upgrade and
configuration.
-
When performing an ISSU from Cisco NX-OS Release 9.3(1) or 9.3(2) to Cisco NX-OS Release 9.3(3) or later, ensure that the
features with user-defined ports, such as <ssh port>, are within the prescribed port range. If the port range is incorrect, follow the syslog message recommendation. For more
information about the port range, see Cisco Nexus 9000 Series NX-OS IP SLAs Configuration Guide, Release 9.3(x).
-
Before upgrading from Cisco NX-OS Release 7.0(3)I7(5) to Cisco NX-OS Release 9.3(5), make sure that you configure TCAM region
Egress Layer3/VLAN QOS [egr-l3-vlan-qos].
-
Beginning with Cisco NX-OS Release 9.3(5), ISSU is supported on FC/FCoE switch mode on N9K-C93180YC-FX. For more information
about the FC/FCoE switch mode and supported hardware, see Cisco Nexus 9000 Series NX-OS SAN Switching Configuration Guide, Release 9.3(x)
-
Beginning with Cisco NX-OS Release 9.3(5), ISSU is supported with FC/FCoE NPV mode on N9K-C93180YC-FX and N9K-C93360YC-FX2.
For more information about the FC/FCoE NPV mode and supported hardware, see Cisco Nexus 9000 Series NX-OS FC-NPV and FCoE-NPV Configuration Guide
-
Software image compaction is only supported on Cisco Nexus 9300-series platform switches.
-
The compressed image of Cisco Nexus 3000-series is hardware dependent and can only be used on the same device that it got
compressed or downloaded from CCO. Do not use the Nexus 3000-series compressed image on Nexus 9000-series
-
The following limitation applies to software upgrades from 7.0(3)I5 to 9.3(x) or 9.2(3) to 9.3(x):
If you have the same NetFlow configuration in both VLAN and SVI, you must remove the NetFlow flow monitor from the VLAN configuration
prior to the upgrade. Once upgraded, reconfigure NetFlow by creating a new flow monitor and adding it to the VLAN configuration.
Failure to perform these steps results in error messages and the inability to modify the VLAN NetFlow configuration in the
upgraded software.
-
When upgrading from Cisco NX-OS Releases 7.0(3)I4(8), 7.0(3)I5(3), and 7.0(3)I6(1) to Cisco NX-OS Release 9.3(x) results in
a disruptive upgrade. If syncing images to standby SUP failed during the disruptive upgrade from Cisco NX-OS Releases 7.0(3)I4(8),
7.0(3)I5(3,) or 7.0(3)I6(1) to 9.3(x), you should manually copy the image to the standby SUP and perform the disruptive upgrade.
-
When upgrading to Cisco NX-OS Release to 9.3(x) from any release prior to 7.0(3)I2(3) an intermediate upgrade to 7.0(3)I4(x),
7.0(3)I5(x), 7.0(3)I6(x), or 7.0(3)I7(x) is required. We recommend using Cisco NX-OS Release 7.0(3)I4(8) or 7.0(3)I7(4) as
the interim release to aid in a smooth migration.
-
When upgrading from Cisco NX-OS Release 7.0(3)I6(1) or 7.0(3)I7(1) to Cisco NX-OS Release 9.3(x), if the Cisco Nexus 9000
Series switches are running vPC and they are connected to an IOS-based switch via Layer 2 vPC, there is a likelihood that
the Layer 2 port channel on the IOS side will become error disabled. The workaround is to disable the spanning-tree etherchannel
guard misconfig command on the IOS switch before starting the upgrade process.
Once both the Cisco Nexus 9000 Series switches are upgraded, you can re-enable the command.
-
If you are upgrading from Cisco NX-OS Release 7.0(3)I5(2) to Cisco NX-OS Release 9.3(x) using the install all command, BIOS
will not be upgraded due to CSCve24965. When the upgrade to Cisco NX-OS Release 9.3(x) is complete, use the install all command
again to complete the BIOS upgrade, if applicable.
-
An upgrade that is performed via the install all command for Cisco NX-OS Release 7.0(3)I2(2b) to Release 9.3(x) might result
in the VLANs being unable to be added to the existing FEX HIF trunk ports. To recover from this, the following steps should
be performed after all FEXs have come online and the HIFs are operationally up:
-
Enter the copy run bootflash:fex_config_restore.cfg command at the prompt.
-
Enter the copy bootflash:fex_config_restore.cfg running-config echo-commands command at the prompt.
-
In Cisco NX-OS Release 7.0(3)I6(1) and earlier, performing an ASCII replay or running the copy file run command on a FEX HIF
configuration requires manually reapplying the FEX configuration after the FEX comes back up.
-
When upgrading to Cisco NX-OS Release 9.3(x) from 7.0(3)I2(x) or before and running EVPN VXLAN configuration, an intermediate
upgrade to 7.0(3)I4(x) or 7.0(3)I5(x) or 7.0(3)I6(x) is required.
-
To perform an EPLD upgrade after an ISSU upgrade from Cisco NX-OS Release 7.x to Cisco NX-OS Release 9.3(x), before starting
the EPLD upgrade, add the copy run start command.
-
Before enabling the FHS on the interface, we recommend that you carve the ifacl TCAM region on Cisco Nexus 9300 and 9500 platform
switches. If you carved the ifacl TCAM region in a previous release, you must reload the system after upgrading to Cisco NX-OS
Release 9.3(x). Uploading the system creates the required match qualifiers for the FHS TCAM region, ifacl.
-
When redistributing static routes, Cisco NX-OS requires the default-information originate command to successfully redistribute the default static route starting in 7.0(3)I7(6).
-
Before enabling the FHS, we recommend that you carve the ing-redirect TCAM region on Cisco Nexus 9200 and 9300-EX platform
switches. If you carved the ing-redirect TCAM region in a previous release, you must reload the system after upgrading to
Cisco NX-OS Release 9.3(x). Uploading the system creates the required match qualifiers for the FHS TCAM region, ing-redirect.
-
Upgrading from Cisco NX-OS Release 9.3(1), 9.3(2) or 9.3(3) to a higher release, with Embedded Event Manager (EEM) configurations
that are saved to the running configuration, may cause a DME error to be presented. The error is in the output of the show consistency-checker dme running-config enhanced command, specifically, the event manager commands. If this error occurs, delete all EEM applet configurations after completing
the ISSU, then reapply the EEM configurations.
-
For any prior release version upgrading to Cisco NX-OS Release 9.3(5) using ISSU, if the following logging level commands
are configured, they are missing in the upgraded version and must be reconfigured:
-
For any prior release version upgrading to Cisco NX-OS Release 9.3(6) using ISSU, if the following logging level commands
are configured, they are missing in the upgraded version and must be reconfigured:
-
logging level evmc
value
-
logging level mvsh
value
-
An error occurs when you try to perform an ISSU if you changed the reserved VLAN without entering the copy running-config
save-config and reload commands.
-
During an ISSU, there is a drop for all traffic to and from 100-Mb ports 65–66 on the Cisco Nexus 92304QC switch.
-
The install all command is the recommended method for software upgrades and downgrades because it performs configuration compatibility
checks and BIOS upgrades automatically. In contrast, changing the boot variables and reloading the device bypasses these checks
and the BIOS upgrade and therefore it is not recommended.
-
Upgrading from Cisco NX-OS Release 7.0(3)I1(2), Release 7.0(3)I1(3), or Release 7.0(3)I1(3a) requires installing a patch for
Cisco Nexus 9500 platform switches only. For more information on the upgrade patch, see Patch Upgrade Instructions.
-
When upgrading to Cisco NX-OS Release 9.3(x), Guest Shell automatically upgrades from 1.0 to 2.0. In the process, the contents
of the guest shell 1.0 root filesystem are lost. To keep from losing important content, copy any needed files to /bootflash
or an off-box location before upgrading to Cisco NX-OS Release 9.3(x).
-
An ISSU can be performed only from a Cisco NX-OS Release 7.0(3)I4(1) to a later image.
-
While performing an ISSU, VRRP and VRRPv3 displays the following messages:
-
If VRRPv3 is enabled:
2015 Dec 29 20:41:44 MDP-N9K-6 %$ VDC-1 %$ %USER-0-SYSTEM_MSG: ISSU ERROR: Service
"vrrpv3" has sent the following message: Feature vrrpv3 is configured. User can change
vrrpv3 timers to 120 seconds or fine tune these timers based on upgrade time on all Vrrp
Peers to avoid Vrrp State transitions. – sysmgr
-
If VRRP is enabled:
2015 Dec 29 20:45:10 MDP-N9K-6 %$ VDC-1 %$ %USER-0-SYSTEM_MSG: ISSU ERROR: Service "vrrp-
eng" has sent the following message: Feature vrrp is configured. User can change vrrp
timers to 120 seconds or fine tune these timers based on upgrade time on all Vrrp Peers to
avoid Vrrp State transitions. – sysmgr
-
Guest Shell is disabled during an ISSU and reactivated after the upgrade. Any application running in the Guest Shell is affected.
-
If you have ITD probes configured, you must disable the ITD service (using the shutdown command) before upgrading to Cisco
NX-OS Release 9.3(x). After the upgrade, enter the feature sla sender command to enable IP SLA for ITD probes and then the no shutdown command to re-enable the ITD service. (If you upgrade without
shutting down the service, you can enter the feature sla sender command after the upgrade.)
-
Schedule the upgrade when your network is stable and steady.
-
Avoid any power interruption, which could corrupt the software image, during the installation procedure.
-
On devices with dual supervisor modules, both supervisor modules must have connections on the console ports to maintain connectivity
when switchovers occur during a software upgrade. See the Hardware Installation Guide for your specific chassis.
-
Perform the installation on the active supervisor module, not the standby supervisor module.
-
The
install all
command is the recommended method for software upgrades because it performs configuration compatibility checks and BIOS upgrades
automatically. In contrast, changing the boot variables and reloading the device bypasses these checks and the BIOS upgrade
and therefore is not recommended.
Note
|
For Cisco Nexus 9500 platform switches with -R line cards, you must save the configuration and reload the device to upgrade
from Cisco NX-OS Release 7.0(3)F3(5) to 9.3(1). To upgrade from Cisco NX-OS Release 9.2(2) or 9.2(3), we recommend that you
use the
install all
command.
|
-
You can detect an incomplete or corrupt NX-OS software image prior to performing an upgrade by verifying the MD5 or SHA256
checksum of the software image.
To verify the MD5 checksum of the software image, run the show file bootflash:<IMAGE-NAME> md5sum command and compare the resulting value to the published MD5 checksum for the software image on Cisco’s Software Download
website.
To verify the SHA256 checksum of the software image, run the show file bootflash:<IMAGE-NAME> sha256sum command and compare the resulting value to the published SHA256 checksum for the software image on Cisco’s Software Download
website.
-
When upgrading from Cisco Nexus 94xx, 95xx, and 96xx line cards to Cisco Nexus 9732C-EX line cards and their fabric modules,
upgrade the Cisco NX-OS software before inserting the line cards and fabric modules. Failure to do so can cause a diagnostic
failure on the line card and no TCAM space to be allocated. You must use the write_erase command followed by the reload command.
-
If you upgrade from a Cisco NX-OS release that supports the CoPP feature to a Cisco NX-OS release that supports the CoPP feature
with additional classes for new protocols, you must either run the setup utility using the setup
command or use the copp profile
command for the new CoPP classes to be available. For more information on these commands, see the "Configuring Control Plane
Policing" chapter in the Cisco Nexus 9000 Series NX-OS Security Configuration Guide, Release 9.3(x).
-
For secure POAP, ensure that DHCP snooping is enabled and set firewall rules to block unintended or malicious DHCP servers.
For more information on POAP, see the Cisco Nexus 9000 Series Fundamentals Configuration Guide, Release 9.3(x).
-
When you upgrade from an earlier release to a Cisco NX-OS release that supports switch profiles, you have the option to move
some of the running-configuration commands to a switch profile. For more information, see the Cisco Nexus 9000 Series NX-OS System Management Configuration Guide, Release 9.3(x).
-
By default, the software upgrade process is disruptive.
-
OpenFlow and LACP fast timer rate configurations are not supported for ISSU.
-
Guest Shell is disabled during an ISSU and reactivated after the upgrade.
-
ISSU supports only default hold timers for BGP peers.
-
During an ISSU on a Cisco Nexus 3164Q, 31128PQ, or 9300 Series switch, all First-Hop Redundancy Protocols (FHRPs) will cause
the other peer to become active if the node undergoing the ISSU is active.
-
Make sure that both vPC peers are in the same mode (regular mode or enhanced mode) before performing a nondisruptive upgrade.
Note
|
vPC peering between an enhanced ISSU mode (boot mode lxc) configured switch and a non-enhanced ISSU mode switch is not supported.
|
-
During an ISSU, the software reload process on the first vPC device locks its vPC peer device by using CFS messaging over
the vPC communications channel. Only one device at a time is upgraded. When the first device completes its upgrade, it unlocks
its peer device. The second device then performs the upgrade process, locking the first device as it does so. During the upgrade,
the two vPC devices temporarily run different releases of Cisco NX-OS; however, the system functions correctly because of
its backward compatibility support.
-
ISSU is not supported when onePK is enabled. You can run the show feature | include onep command to verify that this feature is disabled before performing an ISSU or enhanced ISSU.
-
In general, ISSUs are supported for the following:
-
From a major release to any associated maintenance release.
-
From the last two maintenance releases to the next two major releases.
-
From an earlier maintenance release to the next two major releases.
- After performing ISSU on Cisco Nexus 9300 platform switches and the Cisco Nexus 3164Q switches, you may see the MTS_OPC_CLISH
message on the vPC peers. MTS_OPC_CLISH is the last MTS code that is sent from the back-end component to the VSH to specify
the end of the show command output.
If the user executes a show command that produces more output and keeps the session on for more than 3 minutes, the following
warning message may be displayed on the console. As a workaround, you can set the terminal length as 0 using the
terminal length 0
command or the
show <command> | no-more
option.
--More--2018 Jun 5 19:11:21 Th-agg1 %$ VDC-1 %$ Jun 5 19:11:20 %KERN-2-SYSTEM_MSG: [12633.219113]
App vsh.bin on slot 1 vdc 1 SUP sap 64098(cli_api queue) did not drop MTS_OPC_CLISH with
msg_id 0x675ecf from sender sap 64132(NULL) in 180 sec, contact app owner - kernel
(config)# show ip mroute detail
IP Multicast Routing Table for VRF "default"
Total number of routes: 4801
Total number of (*,G) routes: 2400
Total number of (S,G) routes: 2400
Total number of (*,G-prefix) routes: 1
(*, 225.0.0.1/32), uptime: 00:09:32, igmp(1) pim(0) ip(0)
RPF-Source: 10.10.10.3 [11/110]
Data Created: No
VPC Flags
RPF-Source Forwarder
Stats: 15/720 [Packets/Bytes], 0.000 bps
Stats: Inactive Flow
Incoming interface: Ethernet1/1, RPF nbr: 12.0.0.2
LISP dest context id: 0 Outgoing interface list: (count: 1) (bridge-only: 0)
Vlan2001, uptime: 00:09:32, igmp (vpc-svi)
(60.60.60.2/32, 225.0.0.1/32), uptime: 00:09:31, ip(0) mrib(1) pim(0)
RPF-Source: 60.60.60.2 [20/110]
Data Created: Yes
VPC Flags
--More--2018 Jun 5 19:11:21 Th-agg1 %$ VDC-1 %$ Jun 5 19:11:20 %KERN-2-SYSTEM_MSG: [12633.219113] App vsh.bin on slot 1 vdc 1 SUP
sap 64098(cli_api queue) did not drop MTS_OPC_CLISH with msg_id 0x675ecf from sender sap 64132(NULL) in 180 sec,
contact app owner - kernel
There is no functionality impact or traffic loss due to this issue. All the MTS messages are drained once the show command
displays the complete output, the user enters CTRL+c, or the session gets closed.
-
Occasionally, while the switch is operationally Up and running, the Device not found logs are displayed on the console. This issue is observed because the switch attempts to find an older ASIC version and the
error messages for the PCI probe failure are enabled in the code. There is no functionality impact or traffic loss due to
this issue.
-
ISSU is not supported if EPLD is not at Cisco NX-OS Release 7.0(3)I3(1) or later.
-
A simplified NX-OS numbering format is used for platforms that are supported in Cisco NX-OS 9.3(x) releases. In order to support
a software upgrade from releases prior to Cisco NX-OS Release 7.0(3)I7(4) that have the old release format, an installer feature
supplies an I9(x) label as a suffix to the actual release during the
install all
operation. This label is printed as part of the image during the install operation from any release prior to Cisco NX-OS
Release 7.0(3)I7(4) to 9.3(x), and it can be ignored. See the following example.
switch# install all nxos bootflash:nxos.9.3.1.bin
Installer will perform compatibility check first. Please wait.
Installer is forced disruptive
Verifying image bootflash:/nxos.9.3.1.bin for boot variable "nxos".
[####################] 100% -- SUCCESS
Verifying image type.
[####################] 100% -- SUCCESS
Preparing "nxos" version info using image bootflash:/nxos.9.3.1.bin.
[####################] 100% -- SUCCESS
Preparing "bios" version info using image bootflash:/nxos.9.3.1.bin.
[####################] 100% -- SUCCESS
Performing module support checks.
[####################] 100% -- SUCCESS
Notifying services about system upgrade.
[####################] 100% -- SUCCESS
Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- ------------ ------------ ------
1 yes disruptive reset Incompatible image for ISSU
Images will be upgraded according to following table:
Module Image Running-Version(pri:alt) New-Version Upg-Required
------ ------- -------------------------------------- -------------------- ------------
1 nxos 7.0(3)I7(3) 9.3(1)I9(1) yes
1 bios v07.61(04/06/2017):v07.61(04/06/2017) v05.33(09/08/2018) yes
Switch will be reloaded for disruptive upgrade.
Do you want to continue with the installation (y/n)? [n] y
-
A nondisruptive standard ISSU is supported from Cisco NX-OS Release 7.0(3)I7(4), 7.0(3)I7(5), 7.0(3)I7(6), or 9.2(x) to Cisco
NX-OS Release 9.3(1). For more information, see the ISSU Support Matrix.
-
Beginning with Cisco NX-OS Release 9.3(5), standard, nondisruptive ISSU, on switches that are configured with uRPF, is supported on the following:
-
Cisco Nexus 9300-EX platform switches
-
Cisco Nexus 9300-FX/FX2 platform switches
-
Cisco Nexus 9300-GX platform switches
Note
|
Prior to Cisco NX-OS Release 9.3(5), if any of the above switches were configured with uRPF, standard, nondisruptive ISSU
was not supported.
|
-
ISSU is blocked if
boot poap enable
is configured.
-
On performing a non-disruptive ISSU from Cisco NX-OS Release 7.0(3)I6(1) to any higher version, a traffic loss might occur
based on the number of VLANs configured. To avoid traffic loss, it is recommended to increase the routing protocol's graceful
restart timer to higher value. The recommended value of the graceful restart timer is 600 seconds. You can further increase
or decrease this value based on the scale of the configuration.
-
If MACsec configuration is present, ND ISSU from versions of Cisco NX-OS Release 9.3(x) to Cisco NX-OS Release 9.3(11) is
not supported on single SUP Nexus 9000 switch.
-
Beginning with Cisco NX-OS Release 9.3(13), for Nexus 9300-R platform, to upgrade bios to the latest version you should first
upgrade to nxos image. This release onwards, the install all nxos command only upgrades the nxos sw to the latest version
but the bios image will be upgraded to the last bios released prior to 9.3(13) version.
To upgrade to bios released with 9.3(13) or higher version, first upgrade the nxos image and then use bios-force option to
upgrade the bios. For example,
-
Install all nxos bootflash:nxos64-msll.9.3.13.bin.
The system reloads and boots up with 9.3(13) image.
-
Install all nxos bios-force.
Note
|
The switch reloads twice, once for nxos upgrade and then again for bios upgrade.
|
-
When upgrading from 7.0(3)I7(9) and 7.0(3)I7(10) to 9.3(5) and above releases with 9.2(1) through 9.3(4) as an intermediate
release, certain PTP configurations such as no ptp management and ptp mean-path-delay 0 may be added to the running configuration. These configurations appear in the show run ptp output and can impact the configure replace operation. To fix this issue, perform one of the following:
-
Configure default values, that is, ptp management and ptp mean-path-delay 1000000000 and then perform copy running-config startup-config .
-
Remove the PTP feature using the no feature ptp command, re-apply the pre-upgrade PTP configuration, and then perform the copy running-config startup-config command.
Note
|
The recommended approach (if the upgrade has not been performed yet) is to go from 7.0(3)I7(9) and 7.0(3)I7(10) to 9.3(5)
and above releases directly without an intermediate upgrade to 9.2(1) through 9.3(4).
|