Replacing the Fabric Interconnect

This chapter contains the following topics:

Replacing the Fabric Interconnect

Through Intersight Managed Mode, fabric interconnects can be configured and managed as a pair of peer systems in a UCS domain.

In a UCS domain, both fabric interconnects are active and share connection to other resources. When you remove one fabric interconnect, connectivity and data fails over to the other active fabric interconnect in the domain.

Replacing a fabric interconnect that is managed by Cisco Intersight Managed Mode (IMM) is a straight forward process that occurs by selecting the fabric interconnect(s) and using either of the following Replace options to start the replacement workflow.

  • For single fabric interconnect replacement, use the Replace Fabric Interconnect option, which is available by selecting the fabric interconnect to be replaced.

  • For dual-fabric interconnect replacement in a UCS domain, use the Replace UCS Domain option, which is available by selecting either of the fabric interconnects to be replaced.

Use the following topics to replace a fabric interconnect. When you receive your new system, install it by following the instructions in Installing the Fabric Interconnect.

Preparing a Single Fabric Interconnect for Removal, Intersight

Use Cisco IMM to perform the following task.

Procedure

Ensure that the fabric interconnect you will add has no configuration information.

  1. If you are installing a new fabric interconnect from the factory, go to Replacing a Single Fabric Interconnect, Intersight.

  2. If you are using a fabric interconnect that was previously configured, erase configuration on the fabric interconnect.

    Through the fabric interconnect CLI, you can use the erase configuration command.

What to do next

Go to Replacing a Single Fabric Interconnect, Intersight

Replacing a Single Fabric Interconnect, Intersight

Use the following procedure when you need to replace a fabric interconnect with the same model of fabric interconnect.

Use Cisco UCS Intersight Managed Mode (IMM) to perform software tasks after the new fabric interconnect comes online. For more information, see "Upgrades and RMA of Servers and Fabric Interconnects" in the Cisco Intersight Managed Mode Configuration Guide.


Note

Verify that all vNICs are either redundant or that the fabric failover is enabled.

Procedure

Step 1

Label the ports and the cables that you are using so that you can refer to this information later. You should use the same cabling and port numbering for the replacement fabric interconnect.

Step 2

Disconnect all the cable connections, including servers, FEX fabrics, and blade chassis, from the old fabric interconnect.

Step 3

Power down the fabric interconnect by unplugging it from the power source.

Step 4

Remove the fabric interconnect from the rack. Follow the instructions in Removing a Cisco UCS Fabric Interconnect From a Rack.

Step 5

Install the replacement fabric interconnect into the rack. Follow the instructions in Installing the Fabric Interconnect.

Step 6

Connect all the cable connections, including servers, FEX fabrics, and blade chassis, from the old fabric interconnect to the new fabric interconnect.

  1. Connect the L1/L2 cables that were disconnected to the replacement fabric interconnect.

  2. Connect the data cable according to the labels that you created in Step 2.

  3. Connect the power cable to the fabric interconnect and it will automatically boot and run POST tests.

Step 7

Complete the Basic System Configuration Dialog for the new fabric interconnect, replying to the dialog's prompts as necessary.

For an example of the Basic System Configuration Dialog, see Configuring Fabric Interconnect B Using the Console.

  • If there is no explicit upgrade required, the newly replaced fabric interconnect syncs the system, management plugin, and device connector images with the peer fabric interconnect while the Basic System Configuration Dialog runs after you answer the prompts.

  • If an explicit upgrade is required, proceed to the next step.

Step 8

If necessary, upgrade the software bundles so that the fabric interconnects are in sync.

The Basic System Configuration Dialog will prompt you if updates are required.

Example:

---- Basic System Configuration Dialog ----

  This setup utility will guide you through the basic configuration of
  the system. Only minimal configuration including IP connectivity to
  the Fabric interconnect and its clustering mode is performed through 
  these steps.

  Type Ctrl-C at any time to abort configuration and reboot system.
  To back track or make modifications to already entered values,
  complete input till end of section and answer no when prompted
  to apply configuration.


  Enter the configuration method. (console/gui) ? console

  Installer has detected the presence of a peer Fabric interconnect. 
  This Fabric interconnect will be added to the cluster. Continue (y/n) ? y

  Enter the admin password of the peer Fabric interconnect:
    Connecting to peer Fabric interconnect... done
    Retrieving config from peer Fabric interconnect... done
   Management Mode is  :  intersight

    Local Fabric Interconnect
      Kickstart version            : 9.3(5)I42(1b)
      Management Plugin version    : 1.0.9-255
      local_model_no               : UCS-FI-6536

    Peer Fabric Interconnect
      Kickstart version            : 9.3(5)I42(1b)
      Management Plugin version    : 1.0.9-281
      local_model_no               : UCS-FI-6536


  Do you wish to update firmware on this Fabric Interconnect to the 
  Peer's version? (y/n): y
  Updating firmware of Fabric Interconnect.......  [ Please don't press Ctrl+c while 
    updating firmware ]

 Updating images
 Please wait for firmware update to complete...
 
                                    <output truncated>
Step 9

Allow the new fabric interconnect to come online.

Step 10

In IMM, find the fabric interconnect that you removed in the Fabric Interconnects list.

Step 11

In the Fabric Interconnects list, click the checkbox to select the fabric interconnect that you removed.

Step 12

Click the Replace Fabric Interconnect icon.

This step starts the Replace workflow, which guides you through decommissioning the fabric interconnect.

Step 13

Follow the workflow to completion.
Step 14

When the workflow is completed:

  1. Verify that the Domain Profile is deployed.

  2. Check that all discovery and inventory workflows are triggered and successful.

  3. Verify that the Server Profile is deployed.

  4. Verify that the Chassis Profile is deployed.

  5. Ensure that all the correct switch policies are configured.

  6. Verify that all required ports, port channels, virtual Ethernet and virtual Fibre Channel interfaces are configured and up on both fabric interconnects.

  7. Verify that the removed fabric interconnect is deleted from the Fabric Interconnects list.

  8. Verify that end-to-end Ethernet connectivity is up.

  9. Verify that end-to-end Fibre Channel connectivity is up.

  10. Verify that you can launch a virtual KVM session.

Preparing Fabric Interconnects in a UCS Domain For Removal, Intersight

Use Cisco IMM to perform the following tasks:

Procedure

Ensure that the fabric interconnects you will add have no configuration information.

  1. If you are installing new fabric interconnects from the factory, go to Replacing Fabric Interconnects in a UCS Domain, Intersight.

  2. If you are using fabric interconnects that were previously configured, erase the configuration on both fabric interconnects.

    Through the fabric interconnect CLI, you can use the erase configuration command.

Replacing Fabric Interconnects in a UCS Domain, Intersight

Use the following procedure when you need to replace a pair of fabric interconnects in the same domain that are the same model of fabric interconnect. For example, a Cisco UCS 6400 Series Fabric Interconnect cannot be paired with a Cisco UCS 6500 Series Fabric Interconnect in the same domain.

In a domain, each fabric interconnect operate as an active online system, not as one live systems and one standby system. If needed, this documentation will differentiate the fabric interconnects by naming them fabric interconnect A and fabric interconnect B.

Use Cisco UCS Intersight Managed Mode (IMM) to perform software tasks after the new fabric interconnects come online. For more information, see "Upgrades and RMA of Servers and Fabric Interconnects" in the Cisco Intersight Managed Mode Configuration Guide.


Note

Verify that all vNICs are either redundant or that the fabric failover is enabled.

Procedure

Step 1

Label the ports and the cables that you using so that you can refer to this information later. The cabling and port numbering should be same for the replacement fabric interconnects.

Step 2

Disconnect all the cable connections, including servers, FEX fabrics, and blade chassis, from the old fabric interconnects.

Step 3

Power down the fabric interconnects by unplugging it from the power source.

Step 4

Remove the fabric interconnects from the rack. Follow the instructions in Removing a Cisco UCS Fabric Interconnect From a Rack.

Step 5

Install the replacement fabric interconnects into the rack. Follow the instructions in Installing the Fabric Interconnect.

Step 6

Connect all the cable connections, including servers, FEX fabrics, and blade chassis, from the old fabric interconnects to the new fabric interconnects.

  1. Connect the L1/L2 cables that were disconnected to the replacement fabric interconnects.

  2. Connect the data cable according to the labels that you created in Step 2.

  3. Connect the power cable to the fabric interconnects, and it will automatically boot and run POST tests.

Step 7

Complete the Basic System Configuration Dialog for the new fabric interconnects, replying to the dialog's prompts as necessary.

Important 

When configuring two fabric interconnects in the same UCS domain, make sure to configure the same IP address and domain name.
For an example of the Basic System Configuration Dialog, see Configuring Fabric Interconnect B Using the Console.

  • If there is no explicit upgrade required, the newly replaced fabric interconnects can be upgraded through IMM after they are claimed by Intersight, but before the RMA workflow is activated.

  • If an explicit upgrade is required, you can perform the upgrade through IMM after the fabric interconnects are claimed by IMM.

Step 8

Complete the Basic System Configuration Dialog for the new fabric interconnect pair.

Important 

When configuring two fabric interconnects in the same cluster, make sure to configure the same IP address and domain name.
Step 9

If necessary, upgrade the software bundles so that the fabric interconnects are in sync.

Example:

---- Basic System Configuration Dialog ----

  This setup utility will guide you through the basic configuration of
  the system. Only minimal configuration including IP connectivity to
  the Fabric interconnect and its clustering mode is performed through 
  these steps.

  Type Ctrl-C at any time to abort configuration and reboot system.
  To back track or make modifications to already entered values,
  complete input till end of section and answer no when prompted
  to apply configuration.


  Enter the configuration method. (console/gui) ? console

  Installer has detected the presence of a peer Fabric interconnect. 
  This Fabric interconnect will be added to the cluster. Continue (y/n) ? y

  Enter the admin password of the peer Fabric interconnect:
    Connecting to peer Fabric interconnect... done
    Retrieving config from peer Fabric interconnect... done
   Management Mode is  :  intersight

    Local Fabric Interconnect
      Kickstart version            : 9.3(5)I42(1b)
      Management Plugin version    : 1.0.9-255
      local_model_no               : UCS-FI-6536

    Peer Fabric Interconnect
      Kickstart version            : 9.3(5)I42(1b)
      Management Plugin version    : 1.0.9-281
      local_model_no               : UCS-FI-6536


  Do you wish to update firmware on this Fabric Interconnect to the 
  Peer's version? (y/n): y
  Updating firmware of Fabric Interconnect.......  [ Please don't press Ctrl+c while 
    updating firmware ]

 Updating images
 Please wait for firmware update to complete...
 
                                    <output truncated>
Step 10

Allow the new fabric interconnects to come online.

Step 11

Claim the new fabric interconnect pair to Intersight.

Step 12

In IMM, in the Fabric Interconnects list, find both fabric interconnects that you replaced.

Step 13

Click the checkbox for either of the fabric interconnects (not both) you are replacing in the domain.
Step 14

Click the Replace UCS Domain Interconnect icon.

This step starts the Replace workflow, which guides you through decommissioning the fabric interconnects.

Step 15

Follow the workflow to completion.
Step 16

When the workflow is completed:

  1. Verify that the Domain Profile is deployed for both fabric interconnects.

  2. Check that all discovery and inventory workflows are triggered and successful for both fabric interconnects.

  3. Verify that the Server Profile is deployed for both fabric interconnects.

  4. Verify that the Chassis Profile is deployed for both fabric interconnects.

  5. Ensure that all the correct switch policies are configured for both fabric interconnects.

  6. Verify that all required ports, port channels, virtual Ethernet and virtual Fibre Channel interfaces are configured and up for both fabric interconnects.

  7. Verify that both of the removed fabric interconnects are deleted from the Fabric Interconnects list.

  8. Verify that end-to-end Ethernet connectivity is up for both fabric interconnects.

  9. Verify that end-to-end Fibre Channel connectivity is up for both fabric interconnects.

  10. Verify that you can launch a virtual KVM session on both fabric interconnects.

Removing a Cisco UCS Fabric Interconnect From a Rack


Caution

Support both sides of the fabric interconnect when removing it from the rack. The slider rail and front rack-mount brackets do not have a stop mechanism. If the front of the chassis is unfastened from the rack and the chassis slides forward on the slider rails, it may slip off the end of the rails and fall out of the rack.

Procedure

Step 1

Ensure that the weight of the Cisco UCS Fabric Interconnect is fully supported and that the chassis is being held by another person.

Step 2

Remove the two screws holding the grounding cable to the chassis (if installed).

Step 3

Disconnect the power cords and any console cables.

Step 4

Disconnect all cables that are connected to SFP28 transceivers.

Step 5

Remove the screws fastening the front rack-mount brackets to the mounting rails.

Step 6

Gently slide the Cisco UCS Fabric Interconnect toward you, off of the slider rails and out of the rack.

Repacking the Cisco UCS Fabric Interconnect for Return Shipment

If you need to return the fabric interconnect, remove the fabric interconnect from the rack and repack it for shipment. If possible, use the original packing materials and container to repack the unit. Contact your Cisco customer service representative to arrange for return shipment to Cisco.