Disaster Recovery

The topics covered in this chapter are:

Boot using USB Drive

The bootable USB drive is used to re-image the router for the purpose of system upgrade or boot the router in case of boot failure. The bootable USB drive can be created using a compressed boot file.

Create a Bootable USB Drive Using Compressed Boot File

A bootable USB drive is created by copying a compressed boot file into a USB drive. The USB drive becomes bootable after the contents of the compressed file are extracted.


Note

In case of failure to read or boot from USB drive, ensure that the drive is inserted correctly. If the drive is inserted correctly and still fails to read from USB drive, check the contents of the USB on another system.

This task can be completed using Windows, Linux, or MAC operating systems available on your local machine. The exact operation to be performed for each generic step outlined here depends on the operating system in use.

Before you begin

  • You have access to a USB drive with a storage capacity that is between 8GB (min) and 32 GB (max). USB 2.0 and USB 3.0 are supported.

    Note

    The NCS-5501-SE PID supports a USB device with a storage capacity of 128 GB (max).

  • Copy the compressed boot file from the software download page at cisco.com to your local machine. The file name for the compressed boot file is in the format ncs5500-usb-boot-<release_number>.zip.

Procedure

Step 1

Connect the USB drive to your local machine and format it with FAT32 or MS-DOS file system using the Windows Operating System or Apple MAC Disk Utility.

Step 2

Copy the compressed boot file to the USB drive.

Step 3

Verify that the copy operation is successful. To verify, compare the file size at source and destination. Additionally, verify the MD5 checksum value.

Step 4

Extract the content of the compressed boot file by unzipping it inside the USB drive. This converts the USB drive to a bootable drive.

Note 
The content of the zipped file ("EFI" and "boot" directories) should be extracted directly into root of the USB drive. If the unzipping application places the extracted files in a new folder, move the "EFI" and "boot" directories to root of the USB drive.
Step 5

Eject the USB drive from your local machine.

What to do next

Use the bootable USB drive to boot the router or upgrade its image.

Boot the Router Using USB

The router can be booted using an external bootable USB drive. This might be required when the router is unable to boot from the installed image. A boot failure may happen when the image gets corrupted. During the USB boot, process the router gets re-imaged with the version available on the USB drive.


Note

During the USB boot process, the router is completely re-imaged with the ISO image version present in the bootable USB drive. All existing configurations are deleted because the disk 0 content is erased. No optional packages are installed during the upgrade process; they need to be installed after the upgrade is complete.

Before you begin

Create a bootable USB drive. See Create a Bootable USB Drive Using Compressed Boot File.

Procedure

Use one of the two methods to boot the router from USB:

  • From Admin EXEC mode - Use this method if Admin LXC is up and Admin Exec prompt is accessible:

    1. Run the show controller card-mgr inventory summary command and identify the active RP with the Master chip.

    2. Insert the USB drive to the active RP.

    3. Run hw-module location {<loc> | all} bootmedia usb reload . The RP boots the image from USB and installs the image onto the hard disk. The router boots from the hard disk after installation.

  • From RP BIOS boot manager menu - Use this method if Admin LXC is not running:
    Note 

    Use this procedure only on active RP; the standby RP must either be powered OFF or removed from the chassis. After the active RP is installed with images from USB, insert or power ON the standby RP as appropriate.

    1. Insert the USB drive.

    2. Connect to the console.

    3. Power the router.

    4. Press Esc or Del to pause the boot process and get the RP to BIOS menu.

    5. Select the USB from the boot menu on the RP to which the USB is connected to. The RP boot the image from USB and installs the image onto the hard disk. The router boots from the hard disk after installation.

Note 

If there is no space in the RP, a prompt to either cancel the installation, or to continue with formatting the disk is displayed.

What to do next

  • After the booting process is complete, specify the root username and password.

  • Install the required optional packages.

Boot the Router Using iPXE

iPXE is a pre-boot execution environment that is included in the network card of the management interfaces and works at the system firmware (UEFI) level of the router. iPXE is used to re-image the system, and boot the router in case of boot failure or in the absence of a valid bootable partition. iPXE downloads the ISO image, proceeds with the installation of the image, and finally bootstraps inside the new installation.

iPXE acts as a boot loader and provides the flexibility to choose the image that the system will boot based on the Platform Identifier (PID), the Serial Number, or the management mac-address. iPXE must be defined in the DHCP server configuration file.


Note

PID and serial number is supported only if iPXE is invoked using the command (admin) hw-module location all bootmedia network reload all. If iPXE is selected manually from BIOS, PID and serial number is not supported.

Zero Touch Provisioning

Zero Touch Provisioning (ZTP) helps in auto provisioning after the software installation of the router using iPXE.

ZTP auto provisioning involves:

  • Configuration: Downloads and executes the configuration file. The first line of the file must contain !! IOS XR for ZTP to process the file as a configuration.

  • Script: Downloads and executes the script files. The script files include a programmatic approach to complete a task. For example, scripts created using IOS XR commands to perform patch upgrades. The first line of the file must contain #! /bin/bash or #! /bin/sh for ZTP to process the file as a script.

Setup DHCP Server

A DHCP server must be configured for IPv4, IPv6 or both communication protocols. The following example shows ISC-DHCP server running on Linux system.

Before you begin

  • Consult your network administrator or system planner to procure IP addresses and a subnet mask for the management interface.

  • Physical port Ethernet 0 on RP is the management port. Ensure that the port is connected to management network.

  • Enable firewall to allow the server to process DHCP packets.

  • For DHCPv6, a Routing advertisement (RA) message must be sent to all nodes in the network that indicates which method to use to obtain the IPv6 address. Configure Router-advertise-daemon (radvd, install using yum install radvd) to allow the client to send DHCP request. For example: 
    interface eth3
{
        AdvSendAdvert on;
        MinRtrAdvInterval 60;
        MaxRtrAdvInterval 180;
        AdvManagedFlag on;
        AdvOtherConfigFlag on;
        prefix 2001:1851:c622:1::/64
        {
                AdvOnLink on;
                AdvAutonomous on;
                AdvRouterAddr off;
        };
};

  • The HTTP server can be in the same server as that of the DHCP server, or can be on a different server. After the IP address is assigned from DHCP server, the router must connect to the HTTP server to download the image.

Procedure

Step 1

Create the dhcpd.conf file (for IPv4, IPv6 or both communication protocols), dhcpv6.conf file (for IPv6) or both in the /etc/ or /etc/dhcp directory. This configuration file stores the network information such as the path to the script, location of the ISO install file, location of the provisioning configuration file, serial number, MAC address of the router.

Step 2

Test the server once the DHCP server is running. For example, for IPv4:

  • Use MAC address of the router:
    Note 

    Using the host statement provides a fixed address that is used for DNS, however, verify that option 77 is set to iPXE in the request. This option is used to provide the bootfile to the system when required. 

    host ncs5500 
{
hardware ethernet <router-mac-address>;
if exists user-class and option user-class = "iPXE" {
	filename = "http://<httpserver-address>/<path-to-image>/ncs5500-mini-x.iso";
}
    Ensure that the above configuration is successful.
  • Use serial number of the router: 
    host ncs5500 
{
option dhcp-client-identifier "<router-serial-number>";
  filename "http://<IP-address>/<path-to-image>/ncs5500-mini-x.iso";
  fixed-address <IP-address>;
}
    The serial number of the router is derived from the BIOS and is used as an identifier.
Step 3

Restart DHCP. 

killall dhcpd
/usr/sbin/dhcpd -f -q -4 -pf /run/dhcp-server/dhcpd.pid 
-cf /etc/dhcp/dhcpd.conf ztp-mgmt &

Example

The example shows a sample dhcpd.conf file: 

allow bootp;
allow booting;
ddns-update-style interim;
option domain-name "cisco.com";
option time-offset -8;
ignore client-updates;
default-lease-time 21600;
max-lease-time 43200;
option domain-name-servers <ip-address-server1>, <ip-address-server2>;
log-facility local0;
 :
subnet <subnet> netmask <netmask> {
  option routers <ip-address>;
  option subnet-mask <subnet-mask>;
  next-server <server-addr>;
}
  :
host <hostname> {
  hardware ethernet e4:c7:22:be:10:ba;
  fixed-address <address>;
  filename "http://<address>/<path>/<image.bin>";
}
The example shows a sample dhcpd6.conf file: 

option dhcp6.name-servers <ip-address-server>;
option dhcp6.domain-search "cisco.com";
dhcpv6-lease-file-name "/var/db/dhcpd6.leases";
option dhcp6.info-refresh-time 21600;
option dhcp6.bootfile-url code 59 = string;
subnet6 <subnet> netmask <netmask> {
       range6 2001:1851:c622:1::2 2001:1851:c622:1::9;
        option dhcp6.bootfile-url "http://<address>/<path>/<image.bin>";

What to do next

Invoke ZTP.

Invoke ZTP

ZTP runs within the XR namespace, and within the global VPN routing/forwarding (VRF) namespace for management interfaces and line card interfaces.

Before you begin

Ensure that a DHCP server is setup. For more information, see Setup DHCP Server.

Procedure

Edit the dhcpd.conf file to utilize the capabilities of ZTP.

The following example shows a sample DHCP server configuration including iPXE and ZTP: 

host <host-name>
{
hardware ethernet <router-serial-number or mac-id>;
fixed-address <ip-address>;
  if exists user-class and option user-class = "iPXE" {
  # Image request, so provide ISO image
  filename "http://<ip-address>/<directory>/ncs5500-mini-x.iso";
  } else 
{
  # Auto-provision request, so provide ZTP script or configuration
  filename "http://<ip-address>/<script-directory-path>/ncs5500-ztp.script";
  #filename "http://<ip-address>/<script-directory-path>/ncs5500-ztp.cfg
  }
}
Note 

Either the ZTP .script file or the .cfg file can be provided at a time for auto-provisioning.

With this configuration, the system boots using ncs5500-mini-x.iso during installation, and then download and execute ncs5500-ztp.script when XR VM is up.

Invoke ZTP Manually

ZTP can also be invoked manually with the modified one touch provisioning approach. The process involves:

Before you begin
A configuration file can be used to specify a list of interfaces that will be brought up in XR and DHCP will be invoked on. /pkg/etc/ztp.config is a platform specific file that allows the platform to specify which if any additional interfaces will be used. 

#
# List all the interfaces that ZTP will consider running on. ZTP will attempt
# to bring these interfaces. At which point dhclient will be able to use them.
#
# Platforms may add dynamically to this list.
#
#ZTP_DHCLIENT_INTERFACES=" \
#    Gi0_0_0_0 \
#"
...
Procedure
Step 1

Boot the router.

Step 2

Login manually.

Step 3

Enable interfaces.

Step 4

Invoke a new ZTP DHCP session manually using the ztp initiate command. 


Router#ztp initiate

For example, to send DHCP requests on the GigabitEthernet interface 0/0/0/0, run the command: 


Router#ztp initiate debug verbose interface GigabitEthernet0/0/0/0

ZTP will run on the management port by default unless the platform has configured otherwise. The logs will be logged in /disk0:/ztp/ztp/log location.

Note 

To configure a 40G interface into 4 separate 10G interfaces, use the ztp breakout nosignal-stay-in-breakout-mode command.

Note 
To enable dataport breakouts and invoke DHCP sessions on all dataport and line card interfaces that are detected, use the ztp breakout command. 

Router#ztp breakout debug verbose
Router#ztp initiate dataport debug verbose
Invoke ZTP?(this may change your configuration) [confirm] [y/n]:
To override the prompt: 

Router#ztp initiate noprompt
Invoke ZTP?(this may change your configuration) [confirm] [y/n]:

ZTP will now run in the background.
Please use "show logging" or look at /disk0:/ztp/ztp/log to check progress.
ZTP runs on the management interfaces that are UP by default.
Step 5

To terminate the ZTP session, use the ztp terminate command.

What to do next

Boot the router using iPXE.


Note

While ZTP executes, intermediate configuration is created to control interface addressing and routing information. When the configuration file is downloaded, this immediate configuration is removed and downloaded configuration will be applied. But, when the script file is downloaded intermediate configuration is kept for scripts to communicate with remote hosts. Once the script is ended, the final configuration needs to be applied to the router using the commit replace command. This ensures that the intermediate configuration is replaced. If the commit replace command is not applied after the script execution, intermediate configuration will remain and the final configuration will not take effect.

Boot the Router Using iPXE

Before you use the iPXE boot, ensure that:

  • DHCP server is set and is running.

  • You have logged in to the System Admin console using the admin command.

Run the following command to invoke the iPXE boot process to reimage the router: 
hw-module location all bootmedia network reload
Example: 
sysadmin-vm:0_RP0# hw-module location all bootmedia network reload
Wed Dec 23 15:29:57.376 UTC
Reload hardware module ? [no,yes]
The following example shows the output of the command: 
 
iPXE 1.0.0+ (3e573) -- Open Source Network Boot Firmware -- http://ipxe.org
Features: DNS HTTP TFTP VLAN EFI ISO9660 NBI Menu
Trying net0...
net0: c4:72:95:a6:14:e1 using dh8900cc on PCI01:00.1 (open)
[Link:up, TX:0 TXE:0 RX:0 RXE:0]
Configuring (net0 c4:72:95:a6:14:e1).................. Ok << Talking to DHCP/PXE server to obtain network information
net0: 10.37.1.101/255.255.0.0 gw 10.37.1.0
net0: fe80::c672:95ff:fea6:14e1/64
net0: 2001:1800:5000:1:c672:95ff:fea6:14e1/64 gw fe80::20c:29ff:fefb:b9fe
net1: fe80::c672:95ff:fea6:14e3/64 (inaccessible)
Next server: 10.37.1.235
Filename: http://10.37.1.235/ncs5500/ncs5500-mini-x.iso


http://10.37.1.235/ ncs5500/ncs5500-mini-x.iso... 58% << Downloading file as indicated by DHCP/PXE server to boot install image

Disaster Recovery Using Manual iPXE Boot

Manually booting the system using iPXE can be used to reinstall a clean system in case of a corrupt install or recover lost password. However, all the disks will be wiped out and the configuration will be removed.

Procedure

Step 1

Press Del or Esc key to enter the Boot manager.

Step 2

Use the arrow keys (up, down) to select UEFI: Built-in EFI IPXE to enable iPXE boot. The iPXE boot launches the auto boot.

If the standby RP is being recovered and an active RP is present, the image is pulled from the active RP and auto boot is launched. In case of a single RP, or the other RP is in BIOS or unavailable, iPXE iteratively tries to configure the available interfaces in a loop. The following message is displayed at the end of every iteration:

Press Ctrl-B for the iPXE command line...

To manually boot using iPXE, press Ctrl-B keys to reach the iPXE command line.

Step 3

Identify the management interface. If the management interface is connected properly and is UP, it displays Link:up in the following output:

Example:

iPXE> ifstat
net0: 00:a0:c9:00:00:00 using i350-b on PCI01:00.0 (closed)
  [Link:up, TX:0 TXE:0 RX:0 RXE:0]
net1: 00:a0:c9:00:00:01 using i350-b on PCI01:00.1 (closed)
  [Link:up, TX:0 TXE:0 RX:0 RXE:0]
net2: 00:a0:c9:00:00:02 using i350-b on PCI01:00.2 (closed)
  [Link:down, TX:0 TXE:0 RX:0 RXE:0]
  [Link status: Down (http://ipxe.org/38086193)]
net3: 00:a0:c9:00:00:03 using i350-b on PCI01:00.3 (closed)
  [Link:down, TX:0 TXE:0 RX:0 RXE:0]
  [Link status: Down (http://ipxe.org/38086193)]
net4: 00:00:00:00:00:04 using dh8900cc on PCI02:00.1 (closed)
  [Link:down, TX:0 TXE:0 RX:0 RXE:0]
  [Link status: Down (http://ipxe.org/38086193)]
net5: 00:00:00:00:00:05 using dh8900cc on PCI02:00.2 (closed)
  [Link:down, TX:0 TXE:0 RX:0 RXE:0]
  [Link status: Down (http://ipxe.org/38086193)]
net6: 04:62:73:08:57:86 using dh8900cc on PCI02:00.3 (closed)
  [Link:up, TX:0 TXE:0 RX:0 RXE:0]

iPXE> set net6/ip 10.x.x.y
iPXE> set net6/netmask 255.x.x.x
iPXE> set net6/gateway 10.x.x.x
iPXE>
iPXE> ifopen net6

iPXE> ping 10.x.x.z
64 bytes from 10.x.x.z: seq=1
64 bytes from 10.x.x.z: seq=2
Finished: Operation canceled (http://ipxe.org/0b072095)

iPXE> boot http://10.x.x.z/<dir-to-iso>/ncs5500-mini-x.iso-<version>_IMAGE
http://10.x.x.z/<dir-to-iso>/ncs5500-mini-x.iso-<version>_IMAGE... ok

Choose the net interface that shows Link:up. If there are multiple interfaces that show the status as UP, identify the management interface with MAC address.

iPXE also supports HTTP, TFTP and FTP. For more information, see https://ipxe.org/cmd.

Note 

Keep the standby RP in BIOS while installing the active RP.

After installing the mini ISO image, the system reboots. After successful reboot, specify the root username and password. Once you get back to the XR prompt, you can load the configuration and install remaining packages.