System Upgrade

On Cisco NCS 4000 routers, system upgrade and package installation processes are executed using install commands. The processes involve adding and activating the iso images (.iso), feature packages (.pkg), and software maintenance upgrade files (.smu) on the router. These files are accessed from a network server and then activated on the router. If the installed package or SMU causes any issue on the router, it can be uninstalled.

This chapter provides details of how to upgrade the system using ISSU and OLR.

In-Service Software Upgrade

In-Service Software Upgrade (ISSU) provides the ability to upgrade the router software with no traffic outage. The OTN traffic is hitless whereas packet traffic is impacted.

ISSU is a user-initiated and user-controlled process that uses Cisco nonstop forwarding (NSF) and non-stop routing (NSR). ISSU supports upgrading an image from a lower to a higher version and downgrading an image from a higher version to a lower version.

Processes of ISSU

ISSU on Cisco NCS 4000 enables the Virtual Machines (VM) to run two independent copies of the system software (current version, Version 1; upgraded version, Version 2). The RP VM and the LC VM are upgraded simultaneously. Upgrade using ISSU involves RP switchover and hence two RPs are required.

The upgrade or downgrade using ISSU installation involves:

  • Prepare phase―The installable files are pre-checked and loaded on to the router before activation.

  • Activate phase―The new image (Version 2) is downloaded to all nodes in the router replacing the old image (Version 1). This phase can be run in step-by-step phases too, such as, Load, Run and Cleanup or by using a one-shot activate phase.

  • Commit phase―The ISSU installation is complete with Version 2 on all nodes.

Limitations of ISSU

The limitations of ISSU are:

  • Hitless upgrade(s) using ISSU is possible only when the SDKs are compatible. Change in SDK results in the traffic getting affected. OLR is the available solution, see Orchestrated Linecard Reload.

    SDK changes are applicable only to packet features and hence OLR is implemented for packet-features; for OTN-only nodes, OLR is not required.

  • Multiple simultaneous failures of critical components during an ISSU operation may result in ISSU rollback that will not be hitless.

  • Telnet/SSH connectivity will be momentarily lost during switchover to the new software during ISSU.

  • Some FPGA and other firmware updates may not be hitless.

Implementing ISSU

ISSU supports upgrading the System Admin and the XR VM individually. It is mandatory to upgrade the System Admin first and then the XR VM.

System Admin ISSU

  • Packages can be System Admin SMUs, Host SMUs, System Admin ISO

  • The route processor must have redundancy

  • Preparing the installable files before activation is mandatory

  • Terminating the process is not supported after the activation starts. Reload the system to restore the old version

  • When the image is used to upgrade, the System Admin ISO must be passed along with the host ISO, XR and Sysadmin SMUs

  • Commit command will freeze the new version (V2)

  • Activation of standby RP is trigerred and then the activation of active RP

XR ISSU

  • Packages can be SMUs

  • If the image is used, the image must be compatible with the current active image

  • The route processor must have redundancy

  • Terminating the process is not supported after the activation starts. Reload the system to restore the old version

For upgrade, System Admin ISSU is performed first, followed by XR ISSU. For downgrade, XR ISSU is performed first, followed by System Admin ISSU.

Upgrading SMUs

A Software Maintenance Update (SMU) is a software patch that is installed on the IOS XR device. A SMU is an emergency point fix, which is positioned for expedited delivery and which addresses a network that is down or a problem that affects revenue. A SMU is built on a per release and per component basis and is specific to the platform.

Depending on the process(es) to which the fix is being applied, applying a SMU is non-traffic impacting and the device operation is not compromised.

The two most common SMU upgrades are:

  • Process Restart SMU: specific processes are impacted as part of this fix; critical processes remain unimpacted.

  • ISSU Reload SMU: specific processes, including critical processes are impacted as part of this fix. The upgrade procedures are discussed in the subesequent pages. OLR-ISSU is implemented for SMUs with SDK changes.

Orchestrated Linecard Reload

Orchestrated Linecard Reload (OLR) is a procedure which enables the user to reload the line cards at different times. This allows a hitless software upgrade for both OTN and packet during ISSU. This overcomes the problem encountered by ISSU wherein, all the line cards are upraded simultaneously and hence causing an outage in cases where there is a SDK change in the software. OLR supports software upgrade involving SDK changes.

Implementing OLR

This section explains the OLR process. Let us consider, upgrading the software from Version-1 to Version-2, where Version-2 has a new SDK.

The working and the protect paths need to be non-overlapping for implementing OLR. The solution requires the network administrator to design the NCS 4000 network with redundant cards in the chassis i.e. each LC is backed up by another LC in the chassis.

For ease of understanding, let us consider, the LCs in the chassis are split into two sets, Red and Green. Each traffic working path needs to have a backup path. The key requirement is that the working and protect paths need to be on LCs that belong to different sets. The first step is to force all traffic on to one set of cards, say, green. This can be done in a controlled manner by setting the admin weights. This can cause a 50ms switchover glitch for certain streams moving to their protect paths in the green set. LCs in the green set are ignored and hence data traffic is not impacted. The LCs in the red set are brought up with Version-2. Once the red set is completely functional, the administrator now switches over the traffic from the green to red set cards by reloading the LCs in the green set. This is the second instance of a 50ms glitch due to the protection switchover. Now all the traffic is on red cards while green cards are upgraded to Version-2. At the end, the administrator can rebalance the traffic streams between the two sets of cards to match the original traffic profile.

Starting from Release 6.5.32, you can upgrade the FPDs of the line cards before performing shutdown and reload of the line cards during OLR workflow. See Performing OLR (Single Chassis System).

System Setup (Single Chassis System)

Pre-checks

Perform the following pre-checks:

  1. Check for failed configuration using the following command:

    RP/0/RP0:ios#show configuration failed startup

    If there is a configuration failure re-apply the configuration. If the configuration failure persists, use the following command: 

    RP/0/RP0:ios#clear configuration inconsistency

Creating any missing directories in Configuration File system...OK
Initializing Configuration Version Manager...OK
Syncing commit database with running configuration...OK

  2. All line cards must be installed in plane A or plane B. If any card is not part of the MPLS-TE topology, place it in plane B.

    NCS 4009:

    RP/0/RP0:ios#show running-config  | in hw-mod
Building configuration...
hw-module olr plane A rack 0 nodes 0,1,2,3,4
hw-module olr plane B rack 0 nodes 5,6,7,8

    NCS 4016:

    RP/0/RP0:ios#show running-config | in hw-module 
Building configuration...
hw-module olr plane A rack 0 nodes 0,1,2,3,4,5,6,7
hw-module olr plane B rack 0 nodes 8,9,10,11,12,13,14,15

  3. Configure the route policy for BGPLU neighbor. To check for BGP-LU neighbors, use the following command:

    RP/0/RP0:ios#show bgp sessions

Neighbor        VRF                   Spk    AS   InQ  OutQ  NBRState     NSRState
100.1.1.1      default                 0     1     0     0  Established  NSR Ready
53.0.0.2        default                 0     1     0     0  Established  NSR Ready
54.0.0.2        default                 0     1     0     0  Established  NSR Ready
RP/0/RP0:R1#

RP/0/RP0:ios#show running-config route-policy OLR_PLANE_A
route-policy OLR_PLANE_A 
if destination in (100.1.1.1/32) then 
set weight 6000 
else 
pass 
endif 
end-policy 
! 
RP/0/RP0:ios#show running-config route-policy OLR_PLANE_B
route-policy OLR_PLANE_B 
if destination in (100.1.1.1/32) then 
set weight 5000 
else 
pass 
endif 
end-policy 
!

    Note

    We recommend you to set a higher weight for a BGP-LU session.

    The running configuration of BGP is shown below:

    show running-config route-policy OLR_PLANE_A
router bgp 1
neighbor 53.0.0.2
  remote-as 1
  update-source Bundle-Ether2
  address-family ipv4 labeled-unicast
   route-policy PLANE_A in
   route-reflector-client
   next-hop-self
  !
 !
 neighbor 54.0.0.2
  remote-as 1
  update-source Bundle-Ether3
  address-family ipv4 labeled-unicast
   route-policy PLANE_B in
   route-reflector-client
   next-hop-self
  !

  4. Traffic must be switched from plane A to plane B.

    • Switch traffic on Active/Active LAG from Plane A to Plane B

      int TenGigE 0/0/0/2   >>>>>>>>> Plane A interface 
shut
int TenGigE 0/1/0/2  >>>>>>>>>>> Plane B interface 
no shut

    • To check the traffic from the router, use the following command:

      RP/0/RP0:ios#show inter TenGigE 0/1/0/1 | in pack
  5 minute input rate 1000 bits/sec, 0 packets/sec
  5 minute output rate 1000 bits/sec, 0 packets/sec
     31 packets input, 27399 bytes, 0 total input drops
     Received 0 broadcast packets, 26 multicast packets
     29 packets output, 27245 bytes, 0 total output drops
     Output 0 broadcast packets, 0 multicast packets

    • Switch traffic on BGPLU Plane A to Plane B 

      route-policy OLR_PLANE_B
  if destination in (100.1.1.1/32) then
    set weight 7000
  else
    pass
  endif
end-policy
!

      Note

      A higher weight is recommended.

    • Switch traffic on MPLS enabled Plane A interface to Plane B interface

      mpls traffic-eng
 interface HundredGigE0/0/0/5.100
admin-weight 16777200      
 !

      Note

      16777200 is the lockout metric used for plane A interfaces.

    • Switch traffic on CORE interfaces

      To check for MPLS-TE interface neighbors, use the following command:

      RP/0/RP0:ios#show mpls traffic-eng  link-management igp-neighbors 
Fri Feb 3 14:13:56.515 IST

  Link ID:: HundredGigE0/0/0/5.100
    Neighbor ID: 0000.0000.000b.03 (IS-IS 100 level 2, link address: 48.0.0.2)
Link ID:: HundredGigE0/0/0/6.100
    Neighbor ID: 0000.0000.000b.03 (IS-IS 100 level 2, link address: 49.0.0.2)
Link ID:: HundredGigE0/4/0/6.100
    Neighbor ID: 0000.0000.000b.03 (IS-IS 100 level 2, link address: 58.0.0.2)
Link ID:: HundredGigE0/1/0/6.100
    Neighbor ID: 0000.0000.000b.03 (IS-IS 100 level 2, link address: 59.0.0.2)

      To apply plane A admin-weight on the MPLS-TE interfaces:

      ** config on UUT **
mpls traffic-eng
 interface HundredGigE0/0/0/5.100
admin-weight   16777200
 !
 interface HundredGigE0/0/0/6.100
admin-weight   16777200
 !


** config on Peer Nodes **
mpls traffic-eng
 interface HundredGigE0/0/0/2.100
admin-weight   16777200
 !


** config on Peer Nodes **
mpls traffic-eng
 interface HundredGigE0/0/0/2.100
admin-weight   16777200
 !

    • Check all the traffic moved to plane B interfaces

      Verify that both Ingress IF and Egress IF in command output "show mpls traffic-eng forwarding" belongs to the same Plane B. 

      RP/0/RP0:RP1#show mpls traffic-eng forwarding
P2P tunnels:
Tunnel ID                  Ingress IF     Egress IF      In lbl  Out lbl        Backup 
-------------------------- -------------- -------------- ------- -------------- -------
49.49.49.49 42149_2227     Hu0/13/0/1.112  Te0/6/0/6/3.1 28597   0              unknown
49.49.49.49 42147_2253     Hu0/13/0/1.112  Te0/6/0/6/3.1 28595   0              unknown
49.49.49.49 42145_2273     Hu0/13/0/1.112  Te0/6/0/6/3.1 28593   0              unknown
49.49.49.49 42150_2283     Hu0/13/0/1.112  Te0/6/0/6/3.1 28598   0              unknown
49.49.49.49 42143_2299     Hu0/13/0/1.112  Te0/6/0/6/3.1 28591   0              unknown
49.49.49.49 42148_2302     Hu0/13/0/1.112  Te0/6/0/6/3.1 28596   0              unknown

    • OTN tunnel plane configuration

      OTN line cards must be part of OLR. For OTN, an additional duplicate circuit must be present in plane B. To configure it, place the active circuit in plane A and the create the duplicate circuit in plane B. The client ports are placed in their respective planes.

      Sample Configuration

      mpls traffic-eng
   controller Odu-Group-Te 0
   logging events lsp-status state
   logging events lsp-status signalling-state
   logging events lsp-status switch-over
   logging events lsp-status cross-connect
   logging events lsp-status insufficient-bandwidth
   signalled-bandwidth ODU3
   static-uni ingress-port controller OTU30/8/0/5 egress-port unnumbered 55
   destination ipv4 unicast 10.106.201.222
   path-option 1 dynamic attribute-set otn_olr_planeB_W_orange protected-by 2 lockdown
   path-option 2 dynamic attribute-set otn_olr_planeA_P_brown lockdown
  !
 attribute-set path-option otn_olr_planeA_R_blue
  affinity include blue
 !
 attribute-set path-option otn_olr_planeA_P_brown
  affinity include brown
 !
 attribute-set path-option otn_olr_planeB_W_orange
  affinity include orange

      Note

      The affinities used in the sample configuration are generic and used only for traffic switching.

      Verification

      RP/0/RP0:ios#show controllers Odu-Group-Te0 protection-detail
ODU  Group Information
--------------------------------------------------------------
LOCAL
                Request State                  : Do Not Revert State
                Request signal                 : 1
                Bridge signal                    : 1
                Bridge Status                   : 1+1
REMOTE
                Request State                 : Do Not Revert State
                Request signal                : 1
                Bridge signal                   : 1
                Bridge Status                  : 1+1
WORKING
                Controller Name            : ODU31_1_0_0_43
                ODU STATE                        : Active_tx
                Local Failure                       : State Ok
                Remote Failure                  : Not Applicable
                WTR Left                             : 0 ms
PROTECT
                Controller Name                : ODU30_8_0_1_83
                ODU STATE                        : Active
                Local Failure                       : State Ok
                Remote Failure                  : Not Applicable
                WTR Left                             : 0 ms

      Note

      The ODU-Te 0 has both an active and protect path.

      Note

      The reoptimize timers delay installation parameter is set to 180 seconds. Hence, wait for 180 Seconds after applying the lockout metric on Plane A interfaces.

    • Apply the Max metric on Plane B interfaces 

      ** config on UUT **
mpls traffic-eng
 interface HundredGigE0/4/0/6.100
admin-weight   4294967295
 !
 interface HundredGigE0/1/0/6.100
admin-weight   4294967295
 !


** config on Peer Node**
mpls traffic-eng
 interface HundredGigE0/1/0/2.100
admin-weight   4294967295
 !


** config on Peer Node **
mpls traffic-eng
 interface HundredGigE0/1/0/2.100
admin-weight   4294967295
 !

      Note

      4294967295 is the max metric used for plane B interfaces.

    • Add the V2 image to the router from the sftp path.

      RP/0/RP0:ios: install add source  sftp://test@10.127.60.201://nobackup/tftpboot/images/MC_DT/6533_I/ ncs4k-mpls.pkg-6.5.33 ncs4k-mgbl.pkg-6.5.33 ncs4k-k9sec.pkg-6.5.33 ncs4k-mini-x-6.5.33.iso <SMUs>

      After it is done, check the admin repository to verify.

      RP/0/RP1:ios#show install repository all
2 package(s) in Host repository:
    host-6.5.33
    host-6.5.32
4 package(s) in Admin repository:
    ncs4k-mini-x-6.5.33
    ncs4k-sysadmin-6.5.33 
    ncs4k-mini-x-6.5.32
    ncs4k-sysadmin-6.5.32
17 package(s) in XR repository:
    ncs4k-mini-x-6.5.33
    ncs4k-mgbl-6.5.33
    ncs4k-k9sec-6.5.33
    ncs4k-6.5.32.CSCwe17425-0.0.3.i 
    ncs4k-mpls-6.5.33
    ncs4k-mpls-6.5.32 
    ncs4k-6.5.33.CSCvz67358-0.0.7.i
    ncs4k-xr-6.5.32
    ncs4k-6.5.32.CSCwd69083-0.0.13.i
    ncs4k-mgbl-6.5.32
    ncs4k-6.5.32.CSCvz67358-0.0.6.i
    ncs4k-xr-6.5.33
    ncs4k-k9sec-6.5.32
    ncs4k-mini-x-6.5.32
    ncs4k-6.5.33.CSCwe11655-0.0.8.i
    ncs4k-6.5.32.CSCwc68365-0.0.6.i
    ncs4k-6.5.33.CSCwe17425-0.0.5.i

  5. Verify the cross plane traffic. See the OLR MOP document, Release 6.5.33 for more information.

Install System Admin Package Using ISSU (Single Chassis System)

This task enables the user to upgrade the System Admin package. While performing ISSU, the System Admin package is upgraded first, followed by the XR packages. The System Admin upgrade must be performed node by node.

Procedure

Step 1

Use the show install repository all to display the mini package and the other packages of the new software version.

Example:

 sysadmin-vm:0_RP1# show install repository all 
Fri Feb  3  07:34:43.264 UTC+00:00
 Admin repository
--------------------- 
 ncs4k-mini-x-6.5.33 
 ncs4k-sysadmin-6.5.33 
 ncs4k-mini-x-6.5.32 
 ncs4k-sysadmin-6.5.32
  XR repository 
------------------ 
 ncs4k-mini-x-6.5.33
  ncs4k-mgbl-6.5.33
  ncs4k-k9sec-6.5.33
  ncs4k-6.5.32.CSCwe17425-0.0.3.i
   ncs4k-mpls-6.5.33
   ncs4k-mpls-6.5.32
   ncs4k-6.5.33.CSCvz67358-0.0.7.i
   ncs4k-xr-6.5.32
   ncs4k-6.5.32.CSCwd69083-0.0.13.i
   ncs4k-mgbl-6.5.32
   ncs4k-6.5.32.CSCvz67358-0.0.6.i
   ncs4k-xr-6.5.33
   ncs4k-k9sec-6.5.32
   ncs4k-mini-x-6.5.32
   ncs4k-6.5.33.CSCwe11655-0.0.8.i
   ncs4k-6.5.32.CSCwc68365-0.0.6.i
   ncs4k-6.5.33.CSCwe17425-0.0.5.i 

  

  Host repository 
--------------------
host-6.5.33 
host-6.5.32 
sysadmin-vm:0_RP1#

Step 2

Run the command install extract mini_package from System Admin VM to extract the host and ISO file for System Admin installation.

Example:

sysadmin-vm:0_RP1#  install extract ncs4k-mini-x-<release-version>

Step 3

Prepare the installable files before activation using the command install prepare ncs4k-sysadmin-<release-version>host-<release-version>sysadminSMU<release-version>

Example:

sysadmin-vm:0_RP1# install prepare ncs4k-sysadmin-<release-version> host-<release-version>
Package list:
result Fri Feb 03 07:50:50 2023 Install operation 73 (install prepare) started by user 'root' will continue asynchronously.
sysadmin-vm:0_RP1#

Step 4

Check the current status of the RP1 and RP0 using the command show redundancy summary

Example:

RP/0/RP0:R1 #show redundancy summmary
    Active Node    Standby Node
    -----------    ------------
          0/RP0           0/RP1 (Node Ready, NSR:Ready)
          0/LC0           0/LC1 (Node Ready, NSR:Not Configured)

Step 5

Use the install activate nodes 0/standbyRP to enable the package configurations to be made active on the router so new features and software fixes take effect.

Standby RP reloads and comes up with version2 host and sysadmin. Redundancy is established and NSR is also ready.

  1. Log in to Active RP System Admin console.

    Example:

    telnet 10.106.201.XX 20XX
Trying 10.106.201.13...
Connected to 10.106.201.XX.
Escape character is '^]'.
System Admin Username: root
Password: 
root connected from 127.0.0.1 using console on sysadmin-vm:0_RP0
sysadmin-vm:0_RP0#

  2. Activate the node to the new version using the command install activate Standby RP 

    sysadmin-vm:0_RP0# install activate nodes 0/RP1 
This install operation will result in system reload
Do you want to proceed [yes/no]: yes
Proceeding with operation
result Fri Feb 03 07:13:35 2023 Install operation 74 (install activate) started by user 'root' will continue asynchronously.
sysadmin-vm:0_RP0# 
Fri Feb 03 07:15:36 2023 Install operation 74 completed successfully.

The nodes must be upgraded one by one. Ensure that redundancy is established.

Note

 

Before upgrading the Active RP System Admin, execute the mpls traffic-eng reoptimize all command and wait for 10 mins, because the reoptimize timers delay installation and reoptimize timers delay cleanup are set to 180 seconds in the router. This makes sure to avoid any reoptimization being triggered during ISSU upgrade and system can take 10 minutes to handle the manual reoptimization.

Step 6

Activate the node.

Active RP reloads and comes up with version2 host and sysadmin. Redundancy is established as both RPs are on now same images.

  1. Log in to System Admin active RP console.

    telnet 10.106.201.XX 20XX
Trying 10.106.201.13...
Connected to 10.106.201.XX.
Escape character is '^]'.
System Admin Username: root
Password: 
root connected from 127.0.0.1 using console on sysadmin-vm:0_RP0
sysadmin-vm:0_RP0#

  2. Activate the node to the new version using the command install activate nodes Active RP 

    sysadmin-vm:0_RP0# install activate nodes 0/RP0 
Do you want to proceed [yes/no]: yes 
Proceeding with operation

result Fri Feb 03 08:49:28 2023 Install operation 75 (install activate) started by user 'root' will continue asynchronously. 
Fri Feb 03 08:49:49 2023 Install operation 75 completed successfully.
Fri Feb 03 08:51:28 2023 Card will now reload as part of the install operation.

Step 7

Commit the new System Admin and host images.

  1. Log in to System Admin RP console:

    Example:

    telnet 10.106.201.XX 20XX
Trying 10.106.201.13...
Connected to 10.106.201.XX.
Escape character is '^]'.
System Admin Username: root
Password: 
root connected from 127.0.0.1 using console on sysadmin-vm:0_RP0
sysadmin-vm:0_RP0#

  2. Commit the newly activated software using the command install commit .

    Example:

    sysadmin-vm:0_RP1# show install active
Fri Feb  3  09:03:56.187 UTC+00:00
 Node 0/RP0 [RP]
    Active Packages: 1
       ncs4k-sysadmin-6.5.33 version=6.5.33 [Boot image]    
 	 

 Node 0/RP1 [RP] 
    Active Packages: 1
       ncs4k-sysadmin-6.5.33 version=6.5.33 [Boot image] 

 	 
sysadmin-vm:0_RP1# install commit

result Fri Feb 03 10:28:23 2023 Install operation 76 (install commit) started by user 'root' will continue asynchronously.
sysadmin-vm:0_RP1# Fri Feb 03 10:30:23 2023 Install operation 76 completed successfully.

Step 8

Verify the activated software using the command show install committed .

Example:

sysadmin-vm:0_RP1# show install committed 
Fri Feb  3  09:03:57.187 UTC+00:00
 Node 0/RP0 [RP]
    Active Packages: 1
       ncs4k-sysadmin-6.5.33 version=6.5.33 [Boot image]
 	
 Node 0/RP1 [RP]
    Active Packages: 1
       ncs4k-sysadmin-6.5.33 version=6.5.33 [Boot image]

Install XR Packages Using ISSU

Complete this task to upgrade the system or install a patch. The system upgrade is done using an ISO image file, while the patch installation is done using packages and SMUs.

Before you begin

  • Verify the status of route processor redundancy.

    RP/0/RP0:R1#show redundancy summary 
    Active Node    Standby Node 
    -----------    ------------ 
          0/RP0        0/RP1 (Node Ready, NSR:Ready)
          0/LC0        0/LC1 (Node Ready, NSR:Not Configured) 

RP/0/RP0:R1#

  • Verify Cross plane traffic. See OLR MOP document, Release 6.5.33.

  • Make sure that the V2 image is present in the repository.

Procedure

Step 1

To extract the XR image from ncs4k-x.iso and place it in the repository use the command install extract package_name .

Example:

P/0/RP1:router#install extract ncs4k-mini-x-release-version

Step 2

Verify that the XR image files are properly extracted to repository using the command show install repository all .

Example:

P/0/RP1:router#show install repository all
2 package(s) in Host repository:
    host-6.5.33
    host-6.5.32
4 package(s) in Admin repository:
    ncs4k-mini-x-6.5.33
    ncs4k-sysadmin-6.5.33
    ncs4k-mini-x-6.5.32
    ncs4k-sysadmin-6.5.32
17 package(s) in XR repository:
    ncs4k-mini-x-6.5.33
    ncs4k-mgbl-6.5.33
    ncs4k-k9sec-6.5.33
    ncs4k-6.5.32.CSCwe17425
    ncs4k-mpls-6.5.33
    ncs4k-mpls-6.5.32
    ncs4k-6.5.33.CSCvz67358-0.0.7.i
    ncs4k-xr-6.5.32
    ncs4k-6.5.32.CSCwd69083-0.0.13.i
    ncs4k-mgbl-6.5.32
    ncs4k-6.5.32.CSCvz67358-0.0.6.i
    ncs4k-xr-6.5.33
    ncs4k-k9sec-6.5.32
    ncs4k-mini-x-6.5.32
    ncs4k-6.5.33.CSCwe11655-0.0.8.i
    ncs4k-6.5.32.CSCwc68365-0.0.6.i
    ncs4k-6.5.33.CSCwe17425-0.0.5.i 

RP/0/RP1:Router#

Step 3

Activate the upgrade to the new version using the command install activate issu package_name .

Example:

: router # install activate issu ncs4k-mpls-6.5.33 ncs4k-xr-6.5.33
 ncs4k-mgbl-6.5.33 ncs4k-k9sec-6.5.33
Feb 03 14:26:52 Package list:
Feb 03 14:26:52     ncs4k-mgbl-6.5.33
Feb 03 14:26:52     ncs4k-k9sec-6.5.33
Feb 03 14:26:52     ncs4k-mpls-6.5.33
Feb 03 14:26:52     ncs4k-6.5.33.CSCvz67358-0.0.7.i
Feb 03 14:26:52     ncs4k-xr-6.5.33
Feb 03 14:26:52     ncs4k-6.5.33.CSCwe11655-0.0.8.i
Feb 03 14:26:52     ncs4k-6.5.33.CSCwe17425-0.0.5.i
Feb 03 14:26:53 Action 1: install prepare action started
Feb 03 14:26:53 Install operation will continue in the background
Feb 03 14:27:38 The prepared software is set to be activated with ISSU
Feb 03 14:28:03 Checking compatibility with sysadmin
Feb 03 14:28:03 This install operation will start the issu, continue?
 [yes/no]:[yes] yes

Step 4

Commit the newly activated software using the command install commit .

Example:

RP/0/RP0:R1##install commit 
Feb 03 15:28:30 Install operation 67 started by root:
  install commit 
Feb 03 15:28:31 Install operation will continue in the background
RP/3/RP1:R1#Feb 03 15:28:53 Install operation 67 finished successfully

RP/3/RP1:R1#

Commits the package.

Performing OLR (Single Chassis System)

Procedure

Step 1

Use the command show controllers fia driver location all | in fia to displays the status of the NPU on all the cards.

Example:

RP/0/RP1:R1##show controllers fia driver location all  | in fia
Asics :
HP - HotPlug event, PON - Power On reset
HR - Hard Reset,    WB  - Warm Boot
+----------------------------------------------------------------------------------+
| Asic inst. | fap|HP|Slice|Asic|Admin|Oper | Asic state |   Last   |PON|HR |MODE  |
|  (R/S/A)   | id |  |state|type|state|state|            |   init   |(#)|(#)|STATE |
+----------------------------------------------------------------------------------+
| 0/0/0      |   0| 1| NA  | fia| UP  | NA  |ONLINE      |Sdkless   |  0|  0|Fabric|
| 0/2/0      |   2| 1| NA  | fia| UP  | NA  |ONLINE      |Sdkless   |  0|  0|Fabric|
| 0/8/0      |   4| 1| NA  | fia| UP  | NA  |ONLINE      |Sdkless   |  0|  0|Fabric|
| 0/1/0      |   6| 1| NA  | fia| UP  | NA  |ONLINE      |Sdkless   |  0|  0|Fabric|
| 0/4/0      |   8| 1| NA  | fia| UP  | NA  |ONLINE      |Sdkless   |  0|  0|Fabric|
| 0/11/0     |  10| 1| NA  | fia| UP  | NA  |ONLINE      |Sdkless   |  0|  0|Fabric|

The line cards listed under plane A and plane B will be indicated as SDKLESS.

Step 2

Perform FPD upgrade for plane A line cards:

Use this step when FPD upgrade is required. For upgrade from 6.5.32 to 6.5.33, FPD upgrade is not required.

  1. Verify LC distribution between plane A and plane B

    :   
RP/0/RP0:R1#show running-config  | in hw-mod
Building configuration...
hw-module olr plane A rack 0 nodes 0,2,8
hw-module olr plane B rack 0 nodes 1,4,11

  2. Check the FPDs for the line cards:

    RP/0/RP0:R1#show hw-module fpd | e CURRENT
                                                               FPD Versions
                                                               =================Location   Card type        HWver FPD device       ATR Status   Running Programd
------------------------------------------------------------------------------
0/0       NCS4K-4H-OPW-QC2  0.1   CCC-FPGA             NEED UPGD  1.01    1.01  
0/0       NCS4K-4H-OPW-QC2  0.1   Primary-ZYNQ      S  NEED UPGD  4.11    4.11  
0/2       NCS4K-4H-OPW-QC2  0.1   CCC-FPGA             NEED UPGD  1.01    1.01  
0/2       NCS4K-4H-OPW-QC2  0.1   Primary-ZYNQ      S  NEED UPGD  4.11    4.11  
0/8       NCS4K-4H-OPW-QC2  0.1   CCC-FPGA             NEED UPGD  1.01    1.01  
0/8       NCS4K-4H-OPW-QC2  0.1   Primary-ZYNQ      S  NEED UPGD  4.11    4.11  
0/1       NCS4K-4H-OPW-QC2  0.1   CCC-FPGA             NEED UPGD  1.01    1.01  
0/1       NCS4K-4H-OPW-QC2  0.1   Primary-ZYNQ      S  NEED UPGD  4.11    4.11  
0/4       NCS4K-4H-OPW-QC2  0.1   CCC-FPGA             NEED UPGD  1.01    1.01  
0/4       NCS4K-4H-OPW-QC2  0.1   Primary-ZYNQ      S  NEED UPGD  4.11    4.11
0/11      NCS4K-4H-OPW-QC2  0.1   Primary-ZYNQ      S  NEED UPGD  4.11    4.11  
0/11      NCS4K-4H-OPW-QC2  0.1   CCC-FPGA             NEED UPGD  1.01    1.01  
0/RP0     NCS4K-RP          0.1   Timing-FPGA       S  NEED UPGD  4.42    4.42  
0/RP1     NCS4K-RP          0.1   Timing-FPGA       S  NEED UPGD  4.42    4.42

    Note

     

    During OLR, FPD upgrade is done only on the line cards. And, only one card upgrade is done at a time.

  3. Upgrade the FPD of plane A line cards using one of the following commands:

    Command to upgrade the FPDs: 
    upgrade hw-module location <slot > fpd all
 upgrade hw-module location <slot > fpd <fpd name>

Step 3

Shut down the plane A cards using the hw-module location location-id shutdown command.

Example:

RP/0/RP0:ios#admin

root connected from 127.0.0.1 using console on xr-vm
sysadmin-vm:0_RP1# hw-module location 0/0 shutdown 
Shutdown hardware module ? [no,yes] yes
result Card graceful shutdown request on 0/0 succeeded.
sysadmin-vm:0_RP1# 

** wait for 30 seconds **

sysadmin-vm:0_RP1# hw-module location 0/2 shutdown
Shutdown hardware module ? [no,yes] yes
result Card graceful shutdown request on 0/2 succeeded.
sysadmin-vm:0_RP1# 

** wait for 30 seconds **

sysadmin-vm:0_RP1# hw-module location 0/8 shutdown
Shutdown hardware module ? [no,yes] yes
result Card graceful shutdown request on 0/8 succeeded.
sysadmin-vm:0_RP1#

** wait for 30 seconds **

Step 4

Reload the plane A cards using the hw-module location location-id reload command.

Example:


sysadmin-vm:0_RP1# hw-module location 0/0 reload  
Reload hardware module ? [no,yes] yes
result Card graceful reload request on 0/0 succeeded.
sysadmin-vm:0_RP1# 

** wait for 30 seconds **

sysadmin-vm:0_RP1# hw-module location 0/2reload
Reload hardware module ? [no,yes] yes
result Card graceful reload request on 0/2 succeeded.
sysadmin-vm:0_RP1# 

** wait for 30 seconds **

sysadmin-vm:0_RP1# hw-module location 0/8 reload 
Reload hardware module ? [no,yes] yes
result Card graceful reload request on 0/8 succeeded.
sysadmin-vm:0_RP1#

** wait for 30 seconds **

Note

 
Shutting down and reloading the cards is done one by one. We recommend to have a time interval of one minute before shutting down and reloading the next card.

Before proceeding to the next step, wait till all the interfaces and L3 protocols come up.

RP/0/RP1:R1##show controllers fia driver location all  | in fia
<< snip >>

Asics :
HP - HotPlug event, PON - Power On reset
HR - Hard Reset,    WB  - Warm Boot
+----------------------------------------------------------------------------------+
| Asic inst. | fap|HP|Slice|Asic|Admin|Oper | Asic state |   Last   |PON|HR |MODE  |
|  (R/S/A)   | id |  |state|type|state|state|            |   init   |(#)|(#)|STATE |
+----------------------------------------------------------------------------------+
| 0/0/0      |   0| 1| NA  | fia| UP  | UP  |ONLINE      |PON  |  1|  0|Fabric|
| 0/2/0      |   2| 1| NA  | fia| UP  | UP  |ONLINE      |PON   |  1|  0|Fabric|
| 0/8/0      |   4| 1| NA  | fia| UP  | UP  |ONLINE      |PON   |  1|  0|Fabric|
| 0/1/0      |   6| 1| NA  | fia| UP  | NA  |ONLINE      |Sdkless   |  0|  0|Fabric|
| 0/4/0      |   8| 1| NA  | fia| UP  | NA  |ONLINE      |Sdkless   |  0|  0|Fabric|
| 0/11/0     |  10| 1| NA  | fia| UP  | NA  |ONLINE      |Sdkless   |  0|  0|Fabric|

Step 5

Switch traffic on the active-active LAG interface from plane B to plane A.

Example:

int TenGigE 0/0/0/2
no shut
int TenGigE 0/1/0/2
shut

Step 6

Switch traffic on the active-standby LAG interface from plane B to plane A.

Example:

int TenGigE 0/0/0/1
bundle port-priority 33000
int TenGigE 0/1/0/1
bundle port-priority 34000

Step 7

Switch traffic on the BGPLU Plane B to Plane A

Example:

route-policy OLR_PLANE_B
  if destination in (100.1.1.1/32) then
    set weight 5000
  else
    pass
  endif
end-policy
!

Step 8

Switch traffic from plane B to plane A (bundle).

  1. In the case of VPWS LAG or BGP LU and MPLS-TE, perform these steps:

    • Remove the lockout metric on plane A core interface (TE) and then apply lockout on plane B interfaces. Wait for 180 seconds. Wait for the TE to switch from plane B to plane A.

    • Switch AC LAG or BGP LU from plane B to plane A.

  2. In case of VPWS LAG and FLEX (revertive), remove lockout metric on plane A and switch AC LAG from plane B to plane A.

  3. In case of VPWS LAG and FLEX (nonrevertive), remove lockout on plane A, apply lockout on plane B, and switch AC LAG from plane B to plane A.

Step 9

Switch traffic from plane B to A. Remove plane A lockout metric and apply lockout metric on plane B interface.

Example:

** config on UUT **
mpls traffic-eng
 interface HundredGigE0/0/0/5.100
admin-weight   1000
 !
 interface HundredGigE0/0/0/6.100
admin-weight   1000
 !


** config on Peer Nodes **
mpls traffic-eng
 interface HundredGigE0/0/0/2.100
admin-weight   1000
 !


** config on Peer Nodes **
mpls traffic-eng
 interface HundredGigE0/0/0/2.100
admin-weight   1000
 !

For the flex-LSP , wait till the backup path (plane A interface) comes up. Then apply the lockout metric on plane B interfaces.

sh mpls lsd  forwarding  tunnels  151
Tunnel_Intf, Path_Info: <Type>
tunnel-te151, (TE-Control), local_lbl=24138, 1 Paths, 
        Owner=TE-Control(A)
   1/1: TEv4, 'default':4U, Hu0/4/0/6.100, nh=202.202.202.1, lbl=24030, tun=tt151, weight=0x0, class=0x0 bkup=Hu0/0/0/5.100 mrg_lbl=3, bkup_local_lbl=24432, bkup_nh=102.102.102.1, nnh=0.0.0.0 
            flags=0x200 ()

In the output displayed above, the bkup=Hu0/0/0/5.10 (Plane A interface) has come up. Apply the lockout metric to plane B. 

** config on UUT **
mpls traffic-eng
 interface HundredGigE0/4/0/6.100
admin-weight   16777200
 !
 interface HundredGigE0/1/0/6.100
admin-weight   16777200
 !


** config on Peer Nodes **
mpls traffic-eng
 interface HundredGigE0/1/0/2.100
admin-weight   16777200
 !


** config on Peer Nodes **
mpls traffic-eng
 interface HundredGigE0/1/0/2.100
admin-weight   16777200

Step 10

Perform FPD upgrade for plane B line cards:

  1. Verify LC distribution between plane A and plane B

    :   RP/0/RP0:R1#sh running-config  | in hw-mod
Building configuration...
hw-module olr plane A rack 0 nodes 0,2,8
hw-module olr plane B rack 0 nodes 1,4,11

  2. Check the FPDs for the line cards:

    RP/3/RP1:R1#sh hw-module fpd | e CURRENT
                                                               FPD Versions
                                                               =================Location   Card type        HWver FPD device       ATR Status   Running Programd
------------------------------------------------------------------------------
0/1       NCS4K-4H-OPW-QC2  0.1   CCC-FPGA             NEED UPGD  1.01    1.01  
0/1       NCS4K-4H-OPW-QC2  0.1   Primary-ZYNQ      S  NEED UPGD  4.11    4.11  
0/4       NCS4K-4H-OPW-QC2  0.1   CCC-FPGA             NEED UPGD  1.01    1.01  
0/4       NCS4K-4H-OPW-QC2  0.1   Primary-ZYNQ      S  NEED UPGD  4.11    4.11
0/11      NCS4K-4H-OPW-QC2  0.1   CCC-FPGA             NEED UPGD  1.01    1.01  
0/11      NCS4K-4H-OPW-QC2  0.1   Primary-ZYNQ      S  NEED UPGD  4.11    4.11  
0/RP0     NCS4K-RP          0.1   Timing-FPGA       S  NEED UPGD  4.42    4.42  
0/RP1     NCS4K-RP          0.1   Timing-FPGA       S  NEED UPGD  4.42    4.42

    Note

     

    During OLR, FPD upgrade is done only on the line cards. And, only one card upgrade is done at a time.

  3. Upgrade the FPD of plane B line cards using one of the following commands:

    Command to upgrade the FPDs: 
    upgrade hw-module location <slot > fpd all
 upgrade hw-module location <slot > fpd <fpd name>

Step 11

Shut and reload all the line cards of the plane B interface.

Example:

RP/0/RP0:ios#admin

root connected from 127.0.0.1 using console on xr-vm
sysadmin-vm:0_RP1# hw-module location 0/1 shutdown 
Shutdown hardware module ? [no,yes] yes
result Card graceful shutdown request on 0/1 succeeded.
sysadmin-vm:0_RP1# 

** wait for 30 seconds **

sysadmin-vm:0_RP1# hw-module location 0/4 shutdown
Shutdown hardware module ? [no,yes] yes
result Card graceful shutdown request on 0/4 succeeded.
sysadmin-vm:0_RP1# 

** wait for 30 seconds **

sysadmin-vm:0_RP1# hw-module location 0/11  shutdown
Shutdown hardware module ? [no,yes] yes
result Card graceful shutdown request on 0/11 succeeded.
sysadmin-vm:0_RP1#

** wait for 30 seconds **

sysadmin-vm:0_RP1# hw-module location 0/1 reload  
Reload hardware module ? [no,yes] yes
result Card graceful reload request on 0/1 succeeded.
sysadmin-vm:0_RP1# 

** wait for 30 seconds **

sysadmin-vm:0_RP1# hw-module location 0/4 reload
Reload hardware module ? [no,yes] yes
result Card graceful reload request on 0/4 succeeded.
sysadmin-vm:0_RP1# 

** wait for 30 seconds **

sysadmin-vm:0_RP1# hw-module location 0/11 reload 
Reload hardware module ? [no,yes] yes
result Card graceful reload request on 0/11succeeded.
sysadmin-vm:0_RP1#

  ** wait for 30 seconds **

RP/0/RP1:R1##show controllers fia driver location all  | in fia
<< snip >>

Asics :
HP - HotPlug event, PON - Power On reset
HR - Hard Reset,    WB  - Warm Boot
+----------------------------------------------------------------------------------+
| Asic inst. | fap|HP|Slice|Asic|Admin|Oper | Asic state |   Last   |PON|HR |MODE  |
|  (R/S/A)   | id |  |state|type|state|state|            |   init   |(#)|(#)|STATE |
+----------------------------------------------------------------------------------+
| 0/0/0      |   0| 1| NA  | fia| UP  | UP  |ONLINE      |PON  |  1|  0|Fabric|
| 0/2/0      |   2| 1| NA  | fia| UP  | UP  |ONLINE      |PON   |  1|  0|Fabric|
| 0/8/0      |   4| 1| NA  | fia| UP  | UP  |ONLINE      |PON   |  1|  0|Fabric|
| 0/1/0      |   6| 1| NA  | fia| UP  | UP  |ONLINE      |PON   |  1|  0|Fabric|
| 0/4/0      |   8| 1| NA  | fia| UP  | UP  |ONLINE      |PON   |  1|  0|Fabric|
| 0/11/0     |  10| 1| NA  | fia| UP  | UP  |ONLINE      |PON   |  1|  0|Fabric|
----------------------------------------------------------------------------------------------------------------------

Ensure all the line cards of the plane A and B interfaces are in PON state.

Step 12

Remove the plane B lockout metric.

Example:

** config on UUT **
mpls traffic-eng
 interface HundredGigE0/4/0/6.100
admin-weight   1000
 !
 interface HundredGigE0/1/0/6.100
admin-weight   1000
 !


** config on Peer nodes**
mpls traffic-eng
 interface HundredGigE0/1/0/2.100
admin-weight   1000
 !


** config on Peer nodes **
mpls traffic-eng
 interface HundredGigE0/1/0/2.100
admin-weight   1000
 !

Step 13

Remove the route policy for the BGP LU neighbor.

Example:

Conf t
no route-policy OLR_PLANE_A
       no route-policy OLR_PLANE_B
       commit
router bgp 1
 neighbor 101.6.1.2
  address-family ipv4 labeled-unicast
  no  route-policy OLR_PLANE_B in
 neighbor 101.6.1.6
  address-family ipv4 labeled-unicast
   no route-policy OLR_PLANE_A in
commit
end

Step 14

Verify that all services and up. ISSU and OLR processes are complete.

Note

 

  • Y1564 test is not supported, when the line cards are in sdkless state.

  • Connectivity Fault Management (CFM) peer Maintenance End Points (MEPs) are in timed out state when the line cards are in sdkless state.

Step 15

Verify and upgrade the FPDs.

Use this step when FPD upgrade is required. For upgrade from 6.5.32 to 6.5.33, FPD upgrade is not required.

Example:

RP/0/RP0:R1#show hw-module fpd | e CURRENT
                                                               FPD Versions
                                                           ==================
 Location   Card Type                      HWver FPD device    ATR Status    Running Programd
------------------------------------------------------------------------------
0/RP0     NCS4K-RP          0.1             Timing-FPGA       S  NEED UPGD  4.42    4.42  
0/RP1     NCS4K-RP          0.1             Timing-FPGA       S  NEED UPGD  4.42    4.42

To upgrade the FPDs, use the command, upgrade hw-module location slot fpd all.

Note

 

Only one card upgrade is done at a time.

While upgrading the FPDs, upgrade the line cards first, followed by the fabric cards, and finally the RP cards. After each upgrade, reload the card.