Upgrading Cisco HyperFlex Software Components

Overview

This chapter provides detailed instructions to upgrade the supported HyperFlex software components once your HyperFlex cluster has been upgraded.


Note

Cisco HyperFlex users who need to upgrade their environment from a Cisco HyperFlex HX Data Platform software release that is past the last date of support, to the latest suggested release on the Cisco Software Download site. Should use the Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HX Releases guide.

Upgrading Cisco UCS Infrastructure Using Cisco UCS Manager


Note

Ensure that the hx-storage-data and vMotion upstream switches are configured for Jumbo Frames before proceeding forward, otherwise the HyperFlex Cluster will become offline and all datastore will unmount from the ESXi hosts.

Procedure

Step 1

Open the UCS Manager GUI.

Step 2

Select Equipment > Firmware Management > Firmware auto-install.

Step 3

Click Install Infrastructure Firmware.

Step 4

Check Upgrade Now box.

Step 5

Wait for IOM to be upgraded (if the UCS blade server chassis is present).

  1. Select Equipment > Installed Firmware, expand each chassis and check the Update Status of the IO Module.

  2. During upgrade, the Update Status of the IO Modules will be Upgrading.

  3. IOMs will be in a Pending Next Boot for Activate Status once the Update process completes. When IOM upgrade has completed, the Update Status of the IO Modules is set to Ready.

Step 6

Wait for Subordinate FI to be activated.

  1. Select Equipment > Installed Firmware > Fabric Interconnects.

  2. Check the Activate Status of the kernel and switch images. During upgrade, the Activate Status is set to Activating.

Step 7

During FI reboot, all HX traffic will be forwarded to primary FI (based on ESXi vSwitch failover policy). This will cause a brief traffic interruption. This will not cause storage IO failures.

Step 8

Verify subordinate FI has rebooted and joined the UCS cluster.

  1. Select Equipment > Installed Firmware > Fabric Interconnects.

  2. After activation, the Activate Status of the FI is set to Ready.

  3. Check the Overall Status of the FI is operable.

  4. Check the kernel and switch versions of the FI match the desired and updated version.

  5. Check the FI has no fault.

  6. Check the FI cluster membership is Subordinate.

Step 9

If the UCS blade server chassis is present, wait for IOM activation to complete. Only the IOMs connected to the subordinate FI will enter Ready state, IOMs attached to the Primary FI will remain in Pending Next Boot Activate Status.

  1. Select Equipment > Blade Chassis > IO Module.

  2. Wait for the Activate Status of IO module to change to Ready.

Step 10

Wait until HX traffic is re-pinned to both FIs.

Wait for UCS Manager vNIC faults to be cleared. The fault clearing indicates ESXi has loaded the ENIC driver and the interface is up. The traffic is not re-pinned immediately when the network interface goes up because ESXi has a fail back timer. But the Net.teampolicyupdelay timer is very low by default (100ms).

Step 11

Verify the HX Cluster is online, and healthy before rebooting the primary fabric interconnect.

Access summary tab from the vSphere Web Client Navigator. Select Home > vCenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > cluster > Summary.

Step 12

In the UCS manager GUI, on the toolbar, click Pending Activities. Click on Fabric Interconnects tab that display the tasks requiring user acknowledgment before they can complete.

  1. Click Reboot Now for each pending activity that you want to deploy immediately.

  2. Click OK. Cisco UCS Manager immediately reboots the primary FI. This will cause the subordinate FI to become primary (FI failover).

Step 13

During FI reboot, all HX traffic will be forwarded to the new primary FI. This will cause a brief traffic interruption. However, this will not cause storage IO failures.

Step 14

Wait for UCS Manager to be disconnected, then reconnected on the other FI. This is because a UCS Manager fail over occurs due to primary FI reboot.

Step 15

Check subordinate FI has become primary.

Check that the FI cluster membership is Primary.

Step 16

Wait for FI to be activated.

  1. Select Equipment > Installed Firmware > Fabric Interconnects.

  2. Wait for the Activate Status of the FI to be Ready.

  3. Check the Overall Status of FI is operable.

  4. Check the FI has no fault.

Step 17

Verify FI has rebooted and joined the UCS cluster as subordinate.

Check that the FI cluster membership is Subordinate.

Step 18

Wait for IOM activation to complete

  1. Select Equipment > Blade Chassis > IO Module.

  2. Wait for the Activate Status of the IP module to be Ready.

  3. You can monitor the status on the FSM tab.

    Note 

    You will lose connectivity to UCS Manager throughout the entire upgrade. This is a normal behavior.

Step 19

Wait until HX traffic is re-pinned to both FIs.

In the UCS manager GUI, wait until all server vNIC faults have been cleared.

Step 20

Verify the HX Cluster is online, and healthy after rebooting the FI.

Access summary tab from the vSphere Web Client Navigator. Select Home > vCenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > cluster > Summary.

Upgrading Cisco HX Data Platform

Before you begin

  • Complete pre-upgrade validation checks.

  • Download the latest Cisco HX Data Platform Upgrade Bundle for upgrading existing clusters from previous releases from Software Download .

  • Complete steps 1 to 6 in the Online Upgrade Process Work flow.

    • Upgrade Cisco UCS Infrastructure.

    • Bootstrap to upgrade Cisco HX Data Platform plug-in.

    • Disable snapshot schedule, on the bootstrapped storage controller VM.

    • Log in to the Cisco HX Data Platform plug-in with administrator credentials.

  • Pause replication using the run the stcli dp schedule pause command and enable it after upgrade using the stcli dp schedule resume command. For more detailed steps, see Pausing Replication.

  • If DRS is Enabled, the VMs are automatically vMotioned to other hosts.


    Note
    If DRS is Disabled, vMotion the VMs manually to continue the upgrade process. For more information, see VMware Documentation for Migration with vMotion.

Procedure

Step 1

From the vSphere Web Client Navigator, select vCenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > HX-Cluster > Summary.

Step 2

Select Upgrade Cluster.

Step 3

Select only HX Data Platform.

Step 4

Navigate to the .tgz package file on your local PC.

Upload the Cisco HX Data Platform upgrade bundle. This is the same .tgz package file that was used to upgrade the HX Data Platform Plug-in.

Step 5

Enter administrator level vCenter credentials.

(Optional) Enter the MD5 Checksum # information under Advanced Options. The file checksum can be found on the Cisco.com download page by clicking on the download title to reveal the md5 checksum.
This is an optional step that helps you verify the integrity of the uploaded upgrade package bundle.
Step 6

Click Upgrade.

The Validation screen shows the progress of checks performed. Fix validation errors, if any.

The upgrade process proceeds as follows:

  • HyperFlex cluster readiness for upgrade is checked.

  • One by one, the HX nodes enter maintenance mode.

  • HyperFlex vSphere Installation Bundles on Hypervisor are upgraded.

  • Cisco HX Data Platform is upgraded.

  • HX node exits maintenance mode.

  • Cluster will begin to rebuild back to full health.

  • Once the cluster is healthy, upgrade moves on to the next node in the HyperFlex cluster.

During cluster upgrade, if the orchestration node reboots or power cycles due to power issue, the cluster upgrade will be stuck. Once the node is up, restart the cluster upgrade process after cleaning the cluster system using the following command: 

stcli cluster upgrade --components hxdp –clean

If the clean-up command fails, restart the stMgr service on all control VMs (ctrlVM) by running the following command:

#restart stMgr

Then, clean the cluster system by rerunning the following command:

stcli cluster upgrade --components hxdp –clean

Here is a sample code:

root@ucs-stctlvm-385-1:~# stcli cluster upgrade --clean --components hxdp
##Forcefully cleaned up upgrade progress
root@ucs-stctlvm-385-1:~# stcli cluster upgrade --status
##No active upgrade found. Upgrade progress available after triggering an upgrade

Upgrading Cisco UCS Firmware


Important

  • Do not manually acknowledge the pending activities in Cisco UCS Manager.

  • Make sure that the HX Data Platform is already upgraded. When performing a split upgrade, ensure Cisco UCS Firmware is upgraded around the same time as the HX Data Platform upgrade.

Before you begin

  • Complete pre-upgrade validation checks.

  • Complete steps 1 to 3 in the Online Upgrade Process Work flow.

  • Upgrade Cisco UCS Infrastructure.

  • If DRS is Enabled, the VMs are automatically vMotioned to other hosts.


    Note
    If DRS is Disabled, vMotion the VMs manually to continue the upgrade process. For more information, see VMware Documentation for Migration with vMotion.

  • Downgrading the UCSM firmware is not supported.

Procedure

Step 1

From the vSphere Web Client Navigator, select vCenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > HX-Cluster > Summary.

Step 2

Select Upgrade Cluster.

Step 3

Select only, UCS Firmware. Click Next.

Step 4

Enter administrator level UCS Manager credentials.

Field

Data

UCS Manager Host Name

For example: eng-fi12.eng.storvisor.com

User Name

<admin> username

Password

<admin> password

Step 5

Click Discover to view the current firmware package version.

Step 6

Type in the exact, latest version of Cisco UCS firmware in the Target version field.

Step 7

Click Upgrade.

The Cisco UCS servers are now upgraded with the desired firmware packages. The pending activities will be automatically acknowledged in a rolling fashion.

Note 
You can monitor the progress in the Cisco UCS Manager GUI, under the FSM tab for the service profile.
The Validation screen shows the progress of checks performed. Fix validation errors, if any.

The upgrade process proceeds as follows:

  • HyperFlex cluster readiness for upgrade is checked.

  • One by one, the HX nodes enter maintenance mode.

  • The HX Data Platform requests Cisco UCS Manager to begin firmware upgrade. This process can take up to 1 hour.

    Note 
    You can monitor the progress in the Cisco UCS Manager GUI, under the FSM tab for the service profile.

  • HX node exits maintenance mode.

  • Cluster begins to rebuild back to full health.

  • Once the cluster is healthy, upgrade moves on to the next node in the HyperFlex cluster.

What to do next

Confirm that the upgrade is complete. See Post Upgrade Tasks for more details.

Upgrading Cisco UCS Server Firmware Using the HX Connect UI

Before you begin

  • Complete pre-upgrade validation checks.

  • Complete steps 1 to 3 in the Online Upgrade Process Work flow. See Online Upgrade Process Workflow for more details.

  • Upgrade Cisco UCS Infrastructure.

  • If DRS is Enabled, the VMs are automatically vMotioned to other hosts.


    Note

    If DRS is Disabled, vMotion the VMs manually to continue the upgrade process. For more information, see VMware Documentation for Migration with vMotion.

  • Downgrading the UCSM firmware is not supported.

Procedure

Step 1

Log in to HX Connect.

  1. Enter the HX Storage Cluster management IP address in a browser. Navigate to https://<storage-cluster-management-ip>.

  2. Enter the administrative username and password.

  3. Click Login.

Step 2

In the Navigation pane, select Upgrade.

Step 3

On the Select Upgrade Type page, select UCS Server Firmware and complete the following fields:

Field

Essential Information

UCS Manager Hostname field

Enter the Cisco UCS Manager FQDN or IP address. Example: 10.193.211.120.

User Name field

Enter the Cisco UCS Manager <admin> username.

Admin Password field

Enter the Cisco UCS Manager <admin> password.

Discover button

Click Discover to view the current UCS firmware package version, in the Current Version field.

Step 4

Click Upgrade to begin the UCS firmware upgrade process.

Step 5

The Validation Screen on the Upgrade Progress page displays the progress of the checks performed. Fix validation errors, if any. Confirm that the upgrade is complete.

When upgrade is in progress, you may see an error message, 'Websocket connection failed. Automatic refresh disabled'. You can either refresh the page or log out and log back in to clear the error message. You can safely ignore this error message.

Upgrading ESXi


Caution
Using VMware Update Manager (VUM) to upgrade ESXi is discouraged.

If you are using VUM to upgrade ESXi do the following:

  • Use VUM one host at a time.

  • Make sure that the cluster is in healthy state before moving on to the next node.

  • Do not use VUM to upgrade ESXi across a cluster, as there is no guarantee that the cluster will be healthy by the time VUM moves on to the next node.

The ESXi hypervisor version can be upgraded with no disruption to the HyperFlex cluster workload. This is achieved by performing an online rolling upgrade of each node in the HX cluster.


Important

  • ESXi upgrade requires a manual online upgrade.

  • When upgrading VMware ESXi from 5.5 U3b through any version up to 6.0 U2, please contact Cisco TAC.

  • Use the ESXi command line interface esxcli for upgrading or updating ESXi.

  • Replace the build numbers provided in the examples below with the latest version.

Before you begin

  • Complete pre-upgrade validation checks. See upgrade prerequisites.

  • Ensure that you upgrade vCenter to a compatible version before beginning ESXi upgrades on the hosts.

ESXi Patch Only Upgrade

It is not recommended to use VMware vSphere Update Manager to upgrade the ESXi running on a HyperFlex cluster. Always use the HX Connect Upgrade method to upgrade the ESXi (including the ESXi patch builds) running on HyperFlex cluster. This upgrade method requires no manual intervention if the VMware DRS is enabled on the cluster. This automated process puts the nodes in maintenance mode and reboots one at a time.

  1. Check the HyperFlex upgrade guide for currently running HXDP version and confirm the ESXi patch is supported with current HXDP and UCS Manager versions.

  2. Check the VMware vCenter and ESXi compatibility matrix and confirm the vCenter version is supported with the new ESXi patch.

  3. Before starting the ESXi upgrade, upgrade the vCenter to VMware supported version.

  4. If DRS is Enabled, the VMs are automatically vMotioned to other hosts.


Note

If DRS is Disabled, vMotion the VMs manually to continue the upgrade process. For more information, see VMware Documentation for Migration with vMotion.

Procedure

Step 1

Download ESXi upgrade package. When upgrading ESXi from 6.0 Ux to any newer version, use the offline zip file from Download Software.

Example filename: HX-Vmware-ESXi-60U2-4192238-Cisco-Custom-Bundle-6.0.2.3.zip

Attention 

Do not use the HX ISO file or any other VMware ISO to attempt an ESXi upgrade.

Step 2

Select one of the hosts and put it in HX maintenance mode using the vSphere Web Client.. After the host enters maintenance mode, complete the following steps.

Step 3

Remote secure copy the ESXi upgrade bundle to an appropriate folder with sufficient space.

To copy files using SCP, start the SSH service in the destination ESXi hosts as well.

Note 

  • On HX240, you can use the local SpringpathDS datastore or a mounted HX datastore.

  • On HX220, you can use either a mounted HX datastore or create a temporary RAM disk.

 
scp local_filename user@server:/path/where/file/should/go
Step 4

Log in to ESXi, and execute the following command to query the list of available image profiles and for profile name verification.

esxcli software sources profile list -d <location_of_the_esxi_zip_bundle_on_the_datastore>

Attention 

Full path must be used when running the esxcli software command.

Example:

[root@localhost:~] esxcli software sources profile list -d /vmfs/volumes/5d3a21da-7f370812-ca58-0025
b5a5a102/HX-ESXi-6.0U3-13003896-Cisco-Custom-6.0.3.9-upgrade-bundle.zip
Name                                         Vendor  Acceptance Level  Creation Time        Modification Time
-------------------------------------------  ------  ----------------  -------------------  -------------------
HX-ESXi-6.0U3-13003896-Cisco-Custom-6.0.3.9  Cisco   PartnerSupported  2019-04-02T00:14:56  2019-04-02T13:38:34
Step 5

Run the following command to perform the upgrade.

esxcli software profile update -d <path_to_profile_ZIP_file> -p < profile name>

Example:

[root@HX-ESXi-01:/vmfs/volumes/1a234567-89bc1234] esxcli software profile update -d
/vmfs/volumes/1a234567-89bc1234/HX-Vmware-ESXi-60U2-4192238-Cisco-Custom-Bundle-6.0.2.3.zip
-p HX-ESXi-6.0U3-13003896-Cisco-Custom-6.0.3.9
Step 6

Once the upgrade completes, restart the ESXi host. 

esxcli system shutdown reboot -r Update -d 10
Step 7

After the ESXi host comes up, verify that the host has booted up with the correct version. 

vmware -vl
Step 8

Wait for the ESXi host to auto reconnect to vCenter. In some upgrade scenarios it may be necessary to force ESXi to reconnect from vCenter. Right-click on the host and select Connection > Connect.

Step 9

Exit maintenance mode using the vSphere Web Client.

Step 10

Ensure that the cluster becomes healthy between each ESXi upgrade.

stcli cluster storage-summary --detail
Step 11

Repeat this process for all hosts in the cluster in a sequence.

Note 

Make sure that the cluster becomes healthy between each ESXi upgrade.

Upgrading from an Unsupported Cisco HyperFlex HX Data Platform Software Release

Cisco HyperFlex users who need to upgrade their environment from a Cisco HyperFlex HX Data Platform software release that is past the last date of support, to the latest suggested release on the Cisco Software Download site, should follow the upgrade steps for their current release as defined in the Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HX Releases Guide.