Perform NCS 6000 Disaster Recovery without using USB

Introduction

This document describes a way to recover Network Convergence System 6000 (NCS6K) after failed upgrade without using a USB drive. Recovery with USB drive requires physical access to the device which most of the time can be a challenge and time consuming.

The procedure described in this document would use a Linux Machine acting as a TFTP and DHCP server to recover the NCS6K via the RP management ethernet port.

Prerequisites

Requirements

Cisco recommends that you have basic knowledge of Linux, TFTP, DHCP and Cisco XR CLI.

Components Used

This document is restricted to NCS6K Platform.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Step by Step Procedure

1. Download the required XR USB boot file from Cisco Website.
   
   
2. Upload the downloaded ZIP file to Linux server and unzip it in /tftpboot :

   root@xxxr:/tftpboot# unzip ncs6k-usb-boot-5.2.3.zip
   Archive:  ncs6k-usb-boot-5.2.3.zip
     inflating: EFI/boot/bootx64.efi
     inflating: EFI/boot/grub.cfg
     inflating: boot/install-image.iso

3. Find the mac address of the NCS6K Route Processor (RP) management ethernet port. It can be found in the console logs :

   Press F12 to go to Boot Manager..
   
   Booting System Host OS..
   Verifying Image for Secure Boot failed with status 15
   
   System Host OS boot failed.
   
   Booting Int Network 0 for IPv4 (4C-4E-35-B6-63-33)..
   >>Start PXE over IPv4.
     PXE-E18: Server response timeout.
   
   Int Network 0 for IPv4 (4C-4E-35-B6-63-33) boot failed.
   
   Booting Ext Network 0 for IPv4 (4C-4E-35-B6-63-33)..

4.  Add the following to dhcpd.conf file. This will allocate a static IP address to RP mgmt ethernet port when booting ( eg : 10.48.32.160) :

   root@xxxr:/tftpboot/0A3020A0# cat /etc/dhcp/dhcpd.conf
   
   allow bootp;
   allow booting;
   
   subnet 10.48.32.0 netmask 255.255.255.0 {
   
          option routers 10.48.32.1;
          next-server 10.48.32.93;
          host pani0-rp {
                  hardware ethernet  4c:4e:35:b6:63:33;
                  fixed-address 10.48.32.160;
                  filename "EFI/boot/bootx64.efi" ;
       }
   }

   Note: 10.48.32.93 is the TFTP and DHCP server address.

5. Make a copy of grub.cfg (file generated in step 2). Name the file after the IP address NCS6K will get from DHCP:

   root@xxxr:/tftpboot# cp /tftpboot/EFI/boot/grub.cfg  /tftpboot/10.48.32.160.cfg

6. Edit the file generated above to ensure ISO is picked from the network (remove leading /):

   root@xxxr:/tftpboot# diff /tftpboot/EFI/boot/grub.cfg  /tftpboot/10.48.32.160.cfg
   11,12c11,12
   <         echo "Booting from USB.."
   <         loopback loop /boot/install-image.iso
   ---
   >         echo "Booting from network.."
   >         loopback loop boot/install-image.iso

   For 5.2.3, it would look something like this :

   root@xxxr:/tftpboot# cat /tftpboot/10.48.32.160.cfg
   set default=0
   
   serial --unit=0 --speed=115200
   
   terminal_input console
   terminal_output serial
   
   set timeout=2
   
   menuentry "System Install OS" {
           echo "Booting from network..."
           loopback loop boot/install-image.iso
           root=loop
           echo "Loading Kernel.."
           linux (loop)/boot/bzImage root=/dev/ram install=/dev/sda console=ttyS0,115200 prod=1 crashkernel=192M@0 bigphysarea=10M quiet pci=assign-busses noissu aer=off pci=hpmemsize=0M,hpiosize=0M
           echo "Loading initrd.."
           initrd (loop)/boot/initrd.img signfile=/boot/signature.initrd.img
   }

7. Configuration on Linux server is complete. At next PXE boot attempt, DHCP will allocate 10.48.32.160 to the NCS6K RP. It will then get the grub .efi and .cfg using TFTP. After this, GRUB would start automatically and load the ISO using TFTP.
   
   

   Note: ISO file is normally around 700Mb. It will take some time (up to 10 mins) after "Booting from network.." message is displayed.

   Complete logs of the activity:

   Cisco BIOS version : SB.Panini.0014.00
   BIOS Build Date : 07/10/2014 by lchinnad
   System Memory Speed : 1600 MHz
   Processor Type : Intel(R) Xeon(R) CPU E5-2448L @ 1.80GHz
   
   Press F12 to goto Boot Manager..
   
   Booting System Host OS..
   Verifying Image for Secure Boot failed with status 15
   
   System Host OS boot failed.
   
   Booting Int Network 0 for IPv4 (4C-4E-35-B6-63-33)..
    Start PXE over IPv4.
     PXE-E18: Server response timeout.
   
   Int Network 0 for IPv4 (4C-4E-35-B6-63-33) boot failed.
   
   Booting Ext Network 0 for IPv4 (4C-4E-35-B6-63-33)..
     Start PXE over IPv4.
     Station IP address is 10.48.32.160
     Server IP address is 10.48.32.93
     NBP filename is bootx64.efi
    Downloading NBP file...
   
     Succeed to download NBP file.
   
   
   GNU GRUB version 2.00
   Press F2 to goto grub Menu..
   Booting from network..
   
   [    6.338259] i8042: No controller found
   Starting udev: [  OK  ]
   Actual changes:
   large-receive-offload: off [requested on]
   ntuple-filters: on
   Setting hostname host:  [  OK  ]
   Checking filesystems:[  OK  ]
   Remounting root filesystem in read-write mode:  [  OK  ]
   Entering non-interactive startup
   Bringing up loopback interface:  [  OK  ]
   Starting system logger: [  OK  ]
   Starting kernel logger: [  OK  ]
   Starting kdump:[  OK  ]
   Starting system message bus: [  OK  ]
   Starting smartd: [  OK  ]
   Generating SSH1 RSA host key: [  OK  ]
   Generating SSH2 RSA host key: [  OK  ]
   Generating SSH2 DSA host key: [  OK  ]
   Starting sshd: [  OK  ]
   Starting xinetd: [  OK  ]
   Checking PCI block device /dev/sdb disk space
   Thu Jun 25 14:07:13 UTC 2015: Detected /iso/host.iso
   mount: block device /iso/host.iso is write-protected, mounting read-only
   Thu Jun 25 14:07:13 UTC 2015: Mounted /iso/host.iso to /tmp/isomnt.iV1833
   Thu Jun 25 14:07:13 UTC 2015: Found /tmp/isomnt.iV1833/rpm/ncs6k-sysadmin-hostos.all-5.2.3-Default.x86_64.rpm in host.iso
   Thu Jun 25 14:07:13 UTC 2015: Installing /tmp/isomnt.iV1833/rpm/ncs6k-sysadmin-hostos.all-5.2.3-Default.x86_64.rpm
   Preparing packages for installation...
   ncs6k-sysadmin-hostos.all-5.2.3-Default.x86_64
   hushd_static: no process killed
   hushd restarted
   Thu Jun 25 14:07:13 UTC 2015: Did not detect new pxe install script, keep going with old xrnginstall
   Thu Jun 25 14:07:13 UTC 2015: Running in  Data LV support model
   /etc/rc3.d/S60xrnginstall: line 239: SIMULATION: readonly variable
   Thu Jun 25 14:07:13 UTC 2015: Prepping System with calvados.iso
   Thu Jun 25 14:07:13 UTC 2015: Installer will install image on sda
   Thu Jun 25 14:07:13 UTC 2015: Running in LVM support model
   Thu Jun 25 14:07:15 UTC 2015: Partition creation on /dev/sda took 1 seconds
   Thu Jun 25 14:07:15 UTC 2015: File system creation on /dev/sda1 took 0 seconds
   Thu Jun 25 14:07:15 UTC 2015: Install host image on /dev/sda1
   Thu Jun 25 14:07:23 UTC 2015: Installing host image size of 183M took 8 seconds
   Thu Jun 25 14:07:33 UTC 2015: File system creation on /dev/sda2 took 4 seconds
   Thu Jun 25 14:08:38 UTC 2015: Copying XR iso to repository took 65 seconds
   Partitioning PCI block device /dev/sdb
   Added VLAN with VID == 513 to IF -:eth-pf1:-
   Thu Jun 25 14:08:40 UTC 2015: Copying boot/install-image.iso from tftpserver 10.48.32.93
   Thu Jun 25 14:16:58 UTC 2015: Copying Pxeboot files from tftpserver 10.48.32.93 took 498 seconds
   Thu Jun 25 14:17:28 UTC 2015: File system creation on /dev/panini_vol_grp/calvados_lv0 took 5 seconds
   Thu Jun 25 14:17:28 UTC 2015: Install sysadmin-vm image on /dev/panini_vol_grp/calvados_lv0
   mount: block device /iso/ncs6k-sysadmin.iso is write-protected, mounting read-only
   Thu Jun 25 14:17:35 UTC 2015: sysadmin-vm: RP based installation
   Thu Jun 25 14:18:22 UTC 2015: Installing sysadmin-vm image size of 444M took 54 seconds
   Install EFI on /dev/sda4
   Thu Jun 25 14:18:24 UTC 2015: Install finished on sda
   Resetting BIOS Boot Mode register ...
   Automatic rebooting system after installation ...
   
   Cisco BIOS version : SB.Panini.0014.00
   BIOS Build Date : 07/10/2014 by lchinnad
   System Memory Speed : 1600 MHz
   Processor Type : Intel(R) Xeon(R) CPU E5-2448L @ 1.80GHz
   
   Press F12 to goto Boot Manager..
   
   Booting System Host OS..
   
   GNU GRUB version 2.00
   Press F2 to goto grub Menu..
   Booting from Disk..
   Loading Kernel..
   Loading initrd..
   Starting udev: [  OK  ]
   Setting hostname sysadmin-vm:  [  OK  ]
   Checking filesystems:[  OK  ]
   Mount /dev/vdd at /misc/disk1
   Entering non-interactive startup
   Bringing up loopback interface:  [  OK  ]
   Starting system logger: [  OK  ]
   Starting kernel logger: [  OK  ]
   Starting system message bus: [  OK  ]
   Starting smartd: [FAILED]
   Generating SSH1 RSA host key: [  OK  ]
   Generating SSH2 RSA host key: [  OK  ]
   Generating SSH2 DSA host key: [  OK  ]
   Starting sshd: [  OK  ]
   Starting xinetd: [  OK  ]
   Starting crond: [  OK  ]
   Starting libvirtd daemon: [  OK  ]
   Starting NCS6k programs for RP on sysadmin-vm: [  OK  ]
   starting pm
   sysadmin_startup: Starting Cisco Login Program on ttyS0
   sysadmin initialized
   sysadmin_startup: Starting Cisco Login Program on ttys1
   sysadmin initialized
   
   0_0_0Jun 25 14:19:32 : Send To Helper Failed - Msg : aaad[2600]: %MGBL-AAAD-7-DEBUG : AAA Init successful
   0_0_0Jun 25 14:19:33 : Send To Helper Failed - Msg : vm_manager[2628]: %INFRA-VM_MANAGER-4-INFO : Info: VM Manager started. arguments  -W
   0_0_0Jun 25 14:19:34 : Send To Helper Failed - Msg : sdr_mgr[2619]: %SM-SDR_MANAGER-4-INFO : Info: SDR Manager started.
   
   SYSTEM IS NOT READY FOR LOGIN
   
   !!!NO root-system username is configured. Need to configure root-system username!!!
   
   --- Administrative User Dialog ---
   
   
   Enter root-system Username: 0_0_0Jun 25 14:20:58 : Send To Helper Failed - Msg : plx_fpd[2616]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : Driver missing fpd obfl log function for fpd PLX-8748, FPD init continues but debugability impacted
   0/RP0/ADMIN0:Jun 25 14:20:58.410 : envmon[2609]: %PKT_INFRA-FM-4-FAULT_MINOR : ALARM_MINOR :Unsupported power module detected :DECLARE :0/PT0-PM0:
   0/RP0/ADMIN0:Jun 25 14:20:58.417 : envmon[2609]: %PKT_INFRA-FM-4-FAULT_MINOR : ALARM_MINOR :Unsupported power module detected :DECLARE :0/PT0-PM1:
   0/RP0/ADMIN0:Jun 25 14:20:58.418 : envmon[2609]: %PKT_INFRA-FM-4-FAULT_MINOR : ALARM_MINOR :Unsupported power module detected :DECLARE :0/PT0-PM2:
   0/RP0/ADMIN0:Jun 25 14:20:58.434 : envmon[2609]: %PKT_INFRA-FM-4-FAULT_MINOR : ALARM_MINOR :Unsupported power module detected :DECLARE :0/PT3-PM0:
   0/RP0/ADMIN0:Jun 25 14:20:58.445 : envmon[2609]: %PKT_INFRA-FM-4-FAULT_MINOR : ALARM_MINOR :Unsupported power module detected :DECLARE :0/PT3-PM1:
   0/RP0/ADMIN0:Jun 25 14:20:58.451 : envmon[2609]: %PKT_INFRA-FM-4-FAULT_MINOR : ALARM_MINOR :Unsupported power module detected :DECLARE :0/PT3-PM2:
   0/RP0/ADMIN0:Jun 25 14:20:58.517 : zen[2630]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : Driver missing fpd obfl log function for fpd CPU Complex FPGA, FPD init continues but debugability impacted
   
   
   Enter root-system Username: root
   
   Enter secret:
   
   Enter secret again:
   
   
    Successfully created root-system user
   
   
   System Admin Username: root
   
   Password:
   
   root connected from 127.0.0.1 using console on sysadmin-vm:0_RP0
   
   sysadmin-vm:0_RP0# show platform
   Thu Jun  25 14:21:33.150 UTC
   Location  Card Type               HW State      SW State      Config State
   ----------------------------------------------------------------------------
   0/1       NC6-60X10GE-M-S         POWERED_ON    SW_INACTIVE   NSHUT
   0/7       NC6-10X100G-M-P         POWERED_ON    SW_INACTIVE   NSHUT
   0/RP0     NC6-RP                  OPERATIONAL   OPERATIONAL   NSHUT
   0/RP1     NC6-RP                  POWERED_ON    SW_INACTIVE   NSHUT
   0/FC0     P-L-FC-S                POWERED_ON    N/A           NSHUT
   0/FC1     P-L-FC-S                POWERED_ON    N/A           NSHUT
   0/FC2     P-L-FC-S                POWERED_ON    N/A           NSHUT
   0/FC3     P-L-FC-S                POWERED_ON    N/A           NSHUT
   0/FC4     P-L-FC-S                POWERED_ON    N/A           NSHUT
   0/FC5     P-L-FC-S                POWERED_ON    N/A           NSHUT
   0/CI0     NCS-CRFT=               OPERATIONAL   N/A           NSHUT
   0/FT0     NC6-FANTRAY             OPERATIONAL   N/A           NSHUT
   0/FT1     NC6-FANTRAY             OPERATIONAL   N/A           NSHUT
   0/PT0     NCS-AC-PWRTRAY          OPERATIONAL   N/A           NSHUT
   0/PT1     NCS-AC-PWRTRAY          OPERATIONAL   N/A           NSHUT
   0/PT2     NCS-AC-PWRTRAY          OPERATIONAL   N/A           NSHUT
   0/PT3     NCS-AC-PWRTRAY          OPERATIONAL   N/A           NSHUT
   0/PT4     NCS-AC-PWRTRAY          OPERATIONAL   N/A           NSHUT
   0/PT5     NCS-AC-PWRTRAY          OPERATIONAL   N/A           NSHUT

   After some time, other RP and Line Cards will also start booting.