Configuring Graceful Insertion and Removal

This chapter contains the following sections:

About Graceful Insertion and Removal

You can use graceful insertion and removal to gracefully eject a switch and isolate it from the network in order to perform debugging or upgrade operations. The switch is removed from the regular forwarding path with minimal traffic disruption. When you are finished performing debugging or upgrade operations, you can use graceful insertion to return the switch to its fully operational (normal) mode.

In graceful removal, all protocols and vPC domains are gracefully brought down and the switch is isolated from the network. In graceful insertion, all protocols and vPC domains are restored.

The following protocols are supported (for both IPv4 and IPv6 address families):

  • Border Gateway Protocol (BGP)

  • Enhanced Interior Gateway Routing Protocol (EIGRP)

  • Intermediate System-to-Intermediate System (ISIS)

  • Open Shortest Path First (OSPF)

  • Protocol Independent Multicast (PIM)

  • Routing Information Protocol (RIP)


Note
For graceful insertion and removal, the PIM protocol is applicable only to vPC environments. During graceful removal, the vPC forwarding role is transferred to the vPC peer for all northbound sources of multicast traffic.

Profiles

By default, the system isolates all enabled protocols during graceful removal and restores them during graceful insertion. The protocols are isolated and restored in a predefined order.

If you want to isolate, shut down, or restore the protocols individually (or perform additional configurations), you can create a profile with configuration commands that can be applied during graceful removal or graceful insertion. However, you need to make sure that the order of the protocols is correct and any dependencies are considered.

The switch supports the following profiles:

  • Maintenance-mode profile—Contains all the commands that will be executed during graceful removal, when the switch enters maintenance mode.

  • Normal-mode profile—Contains all the commands that will be executed during graceful insertion, when the switch returns to normal mode.

The following commands (along with any configuration commands) are supported in the profiles:

Command

Description

isolate

Isolates the protocol from the switch and puts the protocol in maintenance mode.

no isolate

Restores the protocol and puts the protocol in normal mode.

shutdown

Shuts down the protocol or vPC domain.

no shutdown

Brings up the protocol or vPC domain.

system interface shutdown [exclude fex-fabric]

Shuts down the system interfaces (except the management interface).

no system interface shutdown [exclude fex-fabric]

Brings up the system interfaces.

sleep instance instance-number seconds

Delays the execution of the command by a specified number of seconds. You can delay multiple instances of the command.

The range for the instance-number and seconds arguments is from 0 to 2177483647.

python instance instance-number uri [python-arguments]

Example: python instance 1 bootflash://script1.py

Configures Python script invocations to the profile. You can add multiple invocations of the command to the profile.

You can enter a maximum of 32 alphanumeric characters for the Python arguments.

Snapshots

In Cisco NX-OS, a snapshot is the process of capturing the running states of selected features and storing them on persistent storage media.

Snapshots are useful to compare the state of a switch before graceful removal and after graceful insertion. The snapshot process consists of three parts:

  • Creating a snapshot of the states of a few preselected features on the switch and storing them on the persistent storage media

  • Listing the snapshots taken at various time intervals and managing them

  • Comparing snapshots and showing the differences between features

GIR Workflow

Follow these steps to complete the graceful insertion and removal (GIR) workflow:

  1. (Optional) Create the maintenance-mode profile. (See Configuring the Maintenance-Mode Profile.)

  2. (Optional) Create the normal-mode profile. (See Configuring the Normal-Mode Profile.)

  3. Take a snapshot before triggering graceful removal. (See Creating a Snapshot.)

  4. Trigger graceful removal to put the switch in maintenance mode. (See Triggering Graceful Removal.)

  5. Trigger graceful insertion to return the switch to normal mode. (See Triggering Graceful Insertion.)

  6. Take a snapshot after triggering graceful insertion. (See Creating a Snapshot.)

  7. Use the show snapshots compare command to compare the operational data before and after the graceful removal and insertion of the switch to make sure that everything is running as expected. (See Verifying the GIR Configuration.)

Configuring the Maintenance-Mode Profile

You can create a maintenance-mode profile with configuration commands that can be applied during graceful removal or graceful insertion.

SUMMARY STEPS

  1. configure maintenance profile maintenance-mode
  2. end
  3. show maintenance profile maintenance-mode

DETAILED STEPS

  Command or Action Purpose

Step 1

configure maintenance profile maintenance-mode

Example:

switch# configure maintenance profile maintenance-mode
Enter configuration commands, one per line. End with CNTL/Z.
switch(config-mm-profile)#

Enters a configuration session for the maintenance-mode profile.

Depending on which protocols you have configured, you must now enter the appropriate commands to bring down the protocols. For a list of supported commands, see Profiles.

Step 2

end

Example:

switch(config-mm-profile)# end
switch#

Closes the maintenance-mode profile.

Step 3

show maintenance profile maintenance-mode

Example:

switch# show maintenance profile maintenance-mode

Displays the details of the maintenance-mode profile.

Example

This example shows how to create a maintenance-mode profile: 

switch# configure maintenance profile maintenance-mode
Enter configuration commands, one per line. End with CNTL/Z.
switch(config-mm-profile)# ip pim isolate
switch(config-mm-profile)# vpc domain 10
switch(config-mm-profile-config-vpc-domain)# shutdown
switch(config-mm-profile)# router bgp 100
switch(config-mm-profile-router)# shutdown
switch(config-mm-profile)# router eigrp 10
switch(config-mm-profile-router)# shutdown
switch(config-mm-profile-router)# address-family ipv6 unicast
switch(config-mm-profile-router-af)# shutdown
switch(config-mm-profile)# system interface shutdown
switch(config-mm-profile)# end
Exit maintenance profile mode.
switch# show maintenance profile maintenance-mode
[Maintenance Mode]
ip pim isolate
vpc domain 10
		shutdown
router bgp 100
  shutdown
router eigrp 10
  shutdown
  address-family ipv6 unicast
    shutdown
system interface shutdown

Configuring the Normal-Mode Profile

You can create a normal-mode profile with configuration commands that can be applied during graceful removal or graceful insertion.

SUMMARY STEPS

  1. configure maintenance profile normal-mode
  2. end
  3. show maintenance profile normal-mode

DETAILED STEPS

  Command or Action Purpose

Step 1

configure maintenance profile normal-mode

Example:

switch# configure maintenance profile normal-mode
Enter configuration commands, one per line. End with CNTL/Z.
switch(config-mm-profile)#

Enters a configuration session for the normal-mode profile.

Depending on which protocols you have configured, you must now enter the appropriate commands to bring up the protocols. For a list of supported commands, see Profiles.

Step 2

end

Example:

switch(config-mm-profile)# end
switch#

Closes the normal-mode profile.

Step 3

show maintenance profile normal-mode

Example:

switch# show maintenance profile normal-mode

Displays the details of the normal-mode profile.

Example

This example shows how to create a maintenance-mode profile: 

switch# configure maintenance profile normal-mode
switch(config-mm-profile)# no system interface shutdown
switch(config-mm-profile)# router eigrp 10
switch(config-mm-profile-router)# no shutdown
switch(config-mm-profile-router)# address-family ipv6 unicast
switch(config-mm-profile-router-af)# no shutdown
switch(config-mm-profile)# router bgp 100
switch(config-mm-profile-router)# no shutdown
switch(config-mm-profile)# vpc domain 10
switch(config-mm-profile-config-vpc-domain)# no shutdown
switch(config-mm-profile)# no ip pim isolate
switch(config-mm-profile)# end
Exit maintenance profile mode.
switch# show maintenance profile normal-mode
[Normal Mode]
no system interface shutdown
router eigrp 10
  no shutdown
  address-family ipv6 unicast
    no shutdown
router bgp 100
  no shutdown
vpc domain 10
		no shutdown
no ip pim isolate

Creating a Snapshot

You can create a snapshot of the running states of selected features. When you create a snapshot, a predefined set of show commands are run and the outputs are saved.

SUMMARY STEPS

  1. snapshot create snapshot-name description
  2. show snapshots
  3. show snapshots compare snapshot-name-1 snapshot-name-2 [summary | ipv4routes | ipv6routes]

DETAILED STEPS

  Command or Action Purpose

Step 1

snapshot create snapshot-name description

Example:

switch# snapshot create snap_before_maintenance Taken before maintenance
Executing 'show interface'... Done
Executing 'show ip route summary vrf all'... Done
Executing 'show ipv6 route summary vrf all'... Done
Executing 'show bgp sessions vrf all'... Done
Executing 'show ip eigrp topology summary'... Done
Executing 'show ipv6 eigrp topology summary'... Done
Feature 'vpc' not enabled, skipping...
Executing 'show ip ospf vrf all'... Done
Feature 'ospfv3' not enabled, skipping...
Feature 'isis' not enabled, skipping...
Feature 'rip' not enabled, skipping...
Snapshot 'snap_before_maintenance' created

Captures the running state or operational data of selected features and stores the data on persistent storage media.

You can enter a maximum of 64 alphanumeric chapters for the snapshot name and a maximum of 254 alphanumeric characters for the description.

Use the snapshot delete {all | snapshot-name} command to delete all snapshots or a specific snapshot.

Step 2

show snapshots

Example:

switch# show snapshots
Snapshot Name             Time                      Description
----------------------------------------------------------------------------
snap_before_maintenance   Wed Aug 19 13:53:28 2015  Taken before maintenance

Displays snapshots present on the switch.

Step 3

show snapshots compare snapshot-name-1 snapshot-name-2 [summary | ipv4routes | ipv6routes]

Example:

switch# show snapshots compare snap_before_maintenance snap_after_maintenance

Displays a comparison of two snapshots.

The summary option displays just enough information to see the overall changes between the two snapshots.

The ipv4routes and ipv6routes options display the changes in IPv4 and IPv6 routes between the two snapshots.

Example

The following example shows a summary of the changes between two snapshots: 

switch# show snapshots compare snapshot1 snapshot2 summary
feature                             snapshot1      snapshot2     changed
basic summary
    # of interfaces                 16             12            *
    # of vlans                      10             4             *
    # of ipv4 routes                33             3             *
    …………

interfaces
    # of eth interfaces             3              0             *
    # of eth interfaces up          2              0             *
    # of eth interfaces down        1              0             *
    # of eth interfaces other       0              0

    # of vlan interfaces            3              1             *
    # of vlan interfaces up         3              1             *
    # of vlan interfaces down       0              0
    # of vlan interfaces other      0              1             *
    …………

The following example shows the changes in IPv4 routes between two snapshots: 

switch# show snapshots compare snapshot1 snapshot2 ipv4routes
metric                          snapshot1      snapshot2       changed
# of routes                     33             3               *
# of adjacencies                10             4               *

Prefix             Changed Attribute
------             -----------------
23.0.0.0/8         not in snapshot2
10.10.10.1/32      not in snapshot2
21.1.2.3/8         adjacency index has changed from 29 (snapshot1) to 38 (snapshot2)
………

There were 28 attribute changes detected

Adding Show Commands to Snapshots

You can specify additional show commands to be captured in snapshots. These show commands are defined in user-specified snapshot sections.

SUMMARY STEPS

  1. snapshot section add section "show-command" row-id element-key1 [element-key2]
  2. show snapshots sections
  3. show snapshots compare snapshot-name-1 snapshot-name-2 [summary | ipv4routes | ipv6routes]

DETAILED STEPS

  Command or Action Purpose

Step 1

snapshot section add section "show-command" row-id element-key1 [element-key2]

Example:

switch# snapshot section add myshow “show ip interface brief” ROW_intf intf-name

Adds a user-specified section to snapshots. The section is used to name the show command output. You can use any word to name the section.

The show command must be enclosed in quotation marks. Non-show commands will not be accepted.

The row-id argument specifies the tag of each row entry of the show command's XML output. The element-key1 and element-key2 arguments specify the tags used to distinguish among row entries. In most cases, only the element-key1 argument needs to specified to be able to distinguish among row entries.

Note

 
To delete a user-specified section from snapshots, use the snapshot section delete section command.

Step 2

show snapshots sections

Example:

switch# show snapshots sections

Displays the user-specified snapshot sections.

Step 3

show snapshots compare snapshot-name-1 snapshot-name-2 [summary | ipv4routes | ipv6routes]

Example:

switch# show snapshots compare snap1 snap2

Displays a comparison of two snapshots.

The summary option displays just enough information to see the overall changes between the two snapshots.

The ipv4routes and ipv6routes options display the changes in IPv4 and IPv6 routes between the two snapshots.

Example

The following example adds the show ip interface brief command to the myshow snapshot section. It also compares two snapshots (snap1 and snap2) and shows the user-specified sections in both snapshots. 

switch# snapshot section add myshow “show ip interface brief” ROW_intf intf-name
switch# show snapshots sections
user-specified snapshot sections
--------------------------------
[myshow]
  cmd:  show ip interface brief
  row:  ROW_intf
  key1: intf-name
  key2: -

[sect2]
  cmd:  show ip ospf vrf all
  row:  ROW_ctx
  key1: instance_number
  key2: cname

switch# show snapshots compare snap1 snap2
=============================================================================
Feature              Tag                  snap1                 snap2                
=============================================================================

[bgp]
-----------------------------------------------------------------------------
…………

[interface]
-----------------------------------------------------------------------------

       [interface:mgmt0]   
                     vdc_lvl_in_pkts      692310             **692317**        
                     vdc_lvl_in_mcast     575281             **575287**        
                     vdc_lvl_in_bcast     77209              **77210**         
                     vdc_lvl_in_bytes     63293252           **63293714**      
                     vdc_lvl_out_pkts     41197              **41198**         
                     vdc_lvl_out_ucast    33966              **33967**         
                     vdc_lvl_out_bytes    6419714            **6419788**       
………… 

[ospf]
-----------------------------------------------------------------------------
………… 

[myshow]
-----------------------------------------------------------------------------

       [interface:Ethernet1/1]   
                     state                up                 **down**        
                     admin_state          up                 **down**        
…………

Triggering Graceful Removal

In order to perform debugging or upgrade operations, you can trigger a graceful removal of the switch, which will eject the switch and isolate it from the network.

Before you begin

If you want the system to use a maintenance-mode profile that you create, see Configuring the Maintenance-Mode Profile.

SUMMARY STEPS

  1. configure terminal
  2. system mode maintenance [dont-generate-profile | timeout value | shutdown | on-reload reset-reason reason]
  3. (Optional) show system mode
  4. (Optional) copy running-config startup-config

DETAILED STEPS

  Command or Action Purpose

Step 1

configure terminal

Example:

switch# configure terminal
switch(config)#

Enters global configuration mode.

Step 2

system mode maintenance [dont-generate-profile | timeout value | shutdown | on-reload reset-reason reason]

Example:

switch(config)# system mode maintenance
Following configuration will be applied:

   ip pim isolate
   router bgp 65502
     isolate
   router ospf p1
     isolate
   router ospfv3 p1
     isolate

Do you want to continue (y/n)? [no] y

Generating a snapshot before going into maintenance mode

Starting to apply commands...

Applying : ip pim isolate
Applying : router bgp 65502
Applying :   isolate
Applying : router ospf p1
Applying :   isolate
Applying : router ospfv3 p1
Applying :   isolate

Maintenance mode operation successful.

Puts all enabled protocols in maintenance mode (using the isolate command).

The following options are available:

  • dont-generate-profile—Prevents the dynamic searching of enabled protocols and executes commands configured in a maintenance-mode profile. Use this option if you want the system to use a maintenance-mode profile that you have created.

  • timeout value —Keeps the switch in maintenance mode for a specified number of minutes. The range is from 5 to 65535. Once the configured time elapses, the switch returns to normal mode automatically. The no system mode maintenance timeout command disables the timer.

  • shutdown—Shuts down all protocols, vPC domains, and interfaces except the management interface (using the shutdown command). This option is disruptive while the default (which uses the isolate command) is not.

  • on-reload reset-reason reason —Boots the switch into maintenance mode automatically in the event of a specified system crash. The no system mode maintenance on-reload reset-reason command prevents the switch from being brought up in maintenance mode in the event of a system crash.

    The maintenance mode reset reasons are as follows:

    • HW_ERROR—Hardware error

    • SVC_FAILURE—Critical service failure

    • KERN_FAILURE—Kernel panic

    • WDOG_TIMEOUT—Watchdog timeout

    • FATAL_ERROR—Fatal error

    • LC_FAILURE—Line card failure

    • MATCH_ANY—Any of the above reasons

The system prompts you to continue. Enter y to continue or n to terminate the process.

Step 3

(Optional) show system mode

Example:

switch(config)# show system mode
System Mode: Maintenance
(Optional)

Displays the current system mode.

The switch is in maintenance mode. You can now perform any desired debugging or upgrade operations on the switch.

Step 4

(Optional) copy running-config startup-config

Example:

switch(config)# copy running-config startup-config
 (Optional)

Copies the running configuration to the startup configuration. This command is required if you want to preserve maintenance mode following a reboot.

Example

This example shows how to shut down all protocols, vPC domains, and interfaces on the switch: 

switch(config)# system mode maintenance shutdown

Following configuration will be applied:

   vpc domain 10
     shutdown
   router bgp 65502
     shutdown
   router ospf p1
     shutdown
   router ospfv3 p1
     shutdown
   system interface shutdown

Do you want to continue (y/n)? [no] y

Generating a snapshot before going into maintenance mode

Starting to apply commands...

Applying : vpc domain 10
Applying :   shutdown
Applying : router bgp 65502
Applying :   shutdown
Applying : router ospf p1
Applying :   shutdown
Applying : router ospfv3 p1
Applying :   shutdown

Maintenance mode operation successful.

This example shows how to automatically boot the switch into maintenance mode if a fatal error occurs: 

switch(config)# system mode maintenance on-reload reset-reason fatal_error

Triggering Graceful Insertion

When you finish performing any debugging or upgrade operations, you can trigger a graceful insertion to restore all protocols.

Before you begin

If you want the system to use a normal-mode profile that you create, see Configuring the Maintenance-Mode Profile.

SUMMARY STEPS

  1. configure terminal
  2. no system mode maintenance [dont-generate-profile]
  3. (Optional) show system mode

DETAILED STEPS

  Command or Action Purpose

Step 1

configure terminal

Example:

switch# configure terminal
switch(config)#

Enters global configuration mode.

Step 2

no system mode maintenance [dont-generate-profile]

Example:

switch(config)# no system mode maintenance dont-generate-profile
Following configuration will be applied:

   no ip pim isolate
   router bgp 65502
     no isolate
   router ospf p1
     no isolate
   router ospfv3 p1
     no isolate
 
Do you want to continue (y/n)? [no] y

Starting to apply commands...

Applying : no ip pim isolate
Applying : router bgp 65502
Applying :   no isolate
Applying : router ospf p1
Applying :   no isolate
Applying : router ospfv3 p1
Applying :   no isolate

Maintenance mode operation successful.

Generating Current Snapshot

Puts all enabled protocols in normal mode (using the no isolate command).

The dont-generate-profile option prevents the dynamic searching of enabled protocols and executes commands configured in a normal-mode profile. Use this option if you want the system to use a normal-mode profile that you have created.

The system prompts you to continue. Enter y to continue or n to terminate the process.

Step 3

(Optional) show system mode

Example:

switch(config)# show system mode
System Mode: Normal
(Optional)

Displays the current system mode. The switch is now in normal mode and is fully operational.

Maintenance Mode Enhancements

The following maintenance mode enhancements are added to Cisco Nexus 3600 platform switches:

  • In the system maintenance shutdown mode, the following message is added: 

    NOTE: The command system interface shutdown will shutdown all interfaces excluding mgmt 0.

  • Entering the CLI command, system mode maintenance checks and sends alerts for the orphan ports.

  • In isolate mode, when the vPC is configured, the following message is added: 

    NOTE: If you have vPC orphan interfaces, please ensure vpc orphan-port suspend is configured under them, before proceeding further.

  • Custom Profile Configuration: A new CLI command, system mode maintenance always-use-custom-profile is added for custom profile configuration. A new CLI command, system mode maintenance non-interactive is added under #ifdef for Cisco Nexus 9000 Series switches only.

    When you create a custom profile (in maintenance or normal mode), it displays the following message: 

    Please use the command system mode maintenance always-use-custom-profile  if you want to always use the custom profile.

  • A delay has been added before the after_maintenance snapshot is taken. The no system mode maintenance command exits once all the configuration for the normal mode has been applied, the mode has been changed to normal mode, and a timer has been started to take the after_maintenance snapshot. Once the timer expires, the after_maintenance snapshot is taken in the background and a new warning syslog, MODE_SNAPSHOT_DONE is sent once the snapshot is complete.

    The final output of the CLI command no system mode maintenance indicates when the after_maintenance snapshot is generated:

    The after_maintenance snapshot will be generated in <delay> seconds. After that time, please use show snapshots compare before_maintenance after_maintenance to check the health of the system. The timer delay for the after_maintenance snapshot is defaulted to 120 seconds but it can be changed by a new configuration command.

    The new configuration command to change the timer delay for the after_maintenance snapshot is system mode maintenance snapshot-delay <seconds> . This configuration overrides the default setting of 120 seconds to any value between 0 and 65535 and it is displayed in the ASCII configuration.

    A new show command, show maintenance snapshot-delay has also been added to display the current snapshot-delay value. This new show command supports the XML output.

  • A visible CLI indicator has been added to display when the system is in the maintenance mode, for example, switch(m-mode)#.

  • Support for the SNMP traps has been added when the device moves from the maintenance mode to the normal mode and vice-versa through CLI reload, or system reset. The snmp-server enable traps mmode cseMaintModeChangeNotify trap is added to enable changing to the maintenance mode trap notification. The snmp-server enable traps mmode cseNormalModeChangeNotify is added to enable changing to the normal mode trap notification. Both the traps are disabled by default.

Verifying the GIR Configuration

To display the GIR configuration, perform one of the following tasks:

Command Purpose

show interface brief

Displays abbreviated interface information.

show maintenance on-reload reset-reasons

Displays the reset reasons for which the switch comes up in maintenance mode. For a description of the maintenance mode reset reasons, see Triggering Graceful Removal.

show maintenance profile [maintenance-mode | normal-mode]

Displays the details of the maintenance-mode or normal-mode profile.

show maintenance timeout

Displays the maintenance-mode timeout period, after which the switch automatically returns to normal mode.

show {running-config | startup-config} mmode [all]

Displays the maintenance-mode section of the running or startup configuration. The all option includes the default values.

show snapshots

Displays snapshots present on the switch.

show snapshots compare snapshot-name-1 snapshot-name-2 [summary | ipv4routes | ipv6routes]

Displays a comparison of two snapshots.

The summary option displays just enough information to see the overall changes between the two snapshots.

The ipv4routes and ipv6routes options display the changes in IPv4 and IPv6 routes between the two snapshots.

show snapshots dump snapshot-name

Displays the content of each file that was generated when the snapshot was taken.

show snapshots sections

Displays the user-specified snapshot sections.

show system mode

Displays the current system mode.