Upgrading Cisco HyperFlex 1.8(1a-1e)

Overview: Upgrading Cisco HyperFlex 1.8(1a-1e)

Prerequisites

Before you begin the upgrading process, you should have already completed the following tasks:

Upgrade Path for Cisco HyperFlex Data Platform 1.8(1a-1e)

Upgrading from Cisco HyperFlex Data Platform 1.8(1a-1e) to the Cisco Recommended HXDP release requires that you upgrade to an intermediate release before upgrading to your destination release. You will complete the upgrade workflow twice.

The following outlines your starting release version, the intermediate release you need to upgrade to, and the destination release that completes upgrade to a recommended Cisco HyperFlex version.

Product

Your Current Release

Upgrade to Intermediate Releases

Upgrade to the Destination Release

HyperFlex software

HXDP 1.8(1a-1e)

First: 2.6(1e)

Second: 4.0(2f)

The latest suggested release on the Cisco Software Download site.

UCS firmware

UCS 3.1(2b) or later.

Minimum version 3.2(3d)

The recommended UCS related firmware based on the suggested HXDP release on the Cisco Software Download site.

You are now ready to begin the Cisco HyperFlex Data Center Upgrade process.


Note

The upgrade process consists of the following tasks complete each step in the order presented.

  • Download software bundles.

  • Install software bundles.

  • Upgrade UCS Firmware to a minimum version of 3.2(3d).

  • Upgrade HyperFlex Data Platform Software to Version 2.6(1e).

  • Perform the second upgrade process to the destination version.

Workflow 1: Upgrading to the Intermediate Release

Upgrading to the Intermediate Release

This upgrade requires that you upgrade to an intermediate release before upgrading to your destination release. Perform the following tasks, using the specified software releases to upgrade to your intermediate release:

Software Downloads: Upgrading to the Intermediate Release

Download the software bundles needed for your upgrade from the Cisco Software Downloads:

Software Downloads

Software Download: Upgrade UCS Firmware to 3.2(3d)

UCS Firmware 3.2(3d)

Before starting the upgrade process, download and install the UCS Infra Package that matches your installation as well as the UCS Blade, and the UCS C-series Package.

Download Link: https://software.cisco.com/download/home/283612660/type/283655658/release/3.2(3d)

UCS Infra Package Download

6200 series UCS Infra package: ucs-k9-bundle-infra.3.2.3d.A.bin

6300 series UCS Infra package: ucs-6300-k9-bundle-infra.3.2.3d.A.bin

UCS Blade Package:

ucs-k9-bundle-b-series.3.2.3d.B.bin

UCS C-series Package:

ucs-k9-bundle-c-series.3.2.3d.C.bin

Upgrade the UCS Infrastructure Package

Before you begin:

Configuring UCS Infrastructure

Perform the following steps:

  1. Open the UCS Manager GUI.

  2. Select Equipment > Firmware Management > Firmware auto-install

  3. Click Install Infrastructure Firmware.

    1. The Install Infrastructure Firmware window appears, select ignore all.

  4. Select the desired UCS infrastructure version. Refer the compatibility matrix to identify the version desired for your use case.

    1. Click Next.

  5. Check the Upgrade Now box.

  6. Click Finish.


    Note

    The expected upgrade behavior is for the UCS Manager to stop and then restart with the new version. Wait until the UCS Manager is back online to log back in to UCS Manager and complete the next steps.

    You may check the Ignore All box for warnings are not critical to user environment.

  7. If the UCS blade server chassis is present, wait for IO Modules (IOM) to upgrade.

    1. Select Equipment > Installed Firmware. Expand each chassis and check the Update Status of the IO Module.

      • During upgrade, the IOM status is Upgrading.

      • When the update process completes, the IOMs state is Pending Next Boot for Activate status.

      • After the IOM upgrade is complete, the IOM state is Ready.

  8. Wait for Subordinate Fabric Interconnects (FI) to be activated.

    1. Select Equipment > Installed Firmware > Fabric Interconnects.

    2. Check the Activate Status of the kernel and switch images.


      Note

      During upgrade, the Activate Status is set to Activating.

  9. During FI reboot, all HX traffic is forwarded to the primary FI (based on ESXi vSwitch failover policy).

    • This will cause a brief traffic interruption.

    • This will not cause storage IO failures.

  10. Verify subordinate FI has rebooted and joined the UCS cluster.

    1. Select Equipment > Installed Firmware > Fabric Interconnects.

    2. After activation, the Activate Status of the FI is set to Ready.

    3. Verify that the Overall Status of the FI is operable.

    4. Verify that the kernel and switch versions of the FI match the desired and updated version.

    5. Verify that the FI has no fault.

    6. Verify that the FI cluster membership is Subordinate.

  11. If the UCS blade server chassis is present, wait for IO Modules (IOM) to upgrade.


    Note

    Only the IOMs connected to the subordinate FI will enter Ready state, IOMs attached to the Primary FI will remain in Pending Next Boot Activate Status.

    1. Select Equipment > Blade Chassis > IO Module.

    2. Wait for the Activate Status of the IOM to change to Ready.

  12. Wait until HX traffic is re-pinned to both FIs.

    Wait for UCS Manager vNIC faults to be cleared. The fault clearing indicates ESXi has loaded the ENIC driver and the interface is up. The traffic is not re-pinned immediately when the network interface goes up because ESXi has a fail back timer. But the Net.teampolicyupdelay timer is very low by default (100ms).

  13. Verify the HX Cluster is online, and healthy before rebooting the primary fabric interconnect.

    • Access summary tab from the vSphere Web Client Navigator. Select Home > vCenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > cluster > Summary.

  14. In the UCS manager GUI toolbar, click Pending Activities.

  15. Click on Fabric Interconnects tab that display the tasks requiring user acknowledgment before they can complete.

    1. Click Reboot Now for each pending activity that you want to deploy immediately.

    2. Click OK. Cisco UCS Manager immediately reboots the primary FI and makes the subordinate FI the primary FI (FI failover).

    During FI reboot, all HyperFlex traffic is forwarded to the new primary FI. This will cause a brief traffic interruption, but it will not cause storage IO failures.

  16. Wait for UCS Manager to disconnect, then reconnected the UCS Manager to the other FI. This step is necessary because a UCS fail over occurs during the primary FI reboot (step 15).

  17. Verify the FI settings:

    • Verify that the subordinate FI is primary.

    • Verify that the FI cluster membership is Primary.

  18. Once the FI is activated, Select Equipment > Installed Firmware > Fabric Interconnects.

  19. Wait for the Activate Status of the FI to be Ready and verify the following:

    • Check the Overall Status of FI is operable.

    • Check the FI has no fault.

    • Verify FI has rebooted and joined the UCS cluster as subordinate.

    • Check that the FI cluster membership is Subordinate.

  20. Wait for IOM activation to complete, then select Equipment > Blade Chassis > IO Module.

  21. Wait for the Activate Status of the IP module to be Ready.

    • You can monitor the status on the FSM tab.


      Note

      You will lose connectivity to UCS Manager throughout the entire upgrade. This is normal behavior

  22. Wait for the HX traffic to re-pin to both FIs.

  23. In the UCS manager GUI, wait for all server vNIC faults to clear.

  24. In the vSphere Web Client Navigator. Select Home > vCenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > cluster > Summary. Verify the HX Cluster is online and healthy after rebooting the FI.

Combined HXDP and UCS Firmware Upgrade

Upgrade the Plugin User Interface (UI)

  1. From the vSphere Web Client Navigator, select vCenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > cluster.

  2. Navigate to Actions > Summary and note the Cluster Management IP address.

  3. SSH to the cluster management IP address with root privileges.

  4. Transfer the latest HX Data Platform upgrade bundle to the controller VM's /tmp directory. Depending on your operating system, use you can either use SCP directly or download third-party tools, such as WinSCP or MobaXterm.

  5. From the controller VM shell, change to the /tmp directory.


    Warning

    Do not use any folder other than /tmp and do not create any subfolders.

  6. Uncompress and extract all files to the root using tar –zxvf <storfs package name>.tgz.

    Example:

    storfs-packages-2.6.1e-26812.tgz

  7. To invoke the cluster-bootstrap.sh script to bootstrap packages for upgrade, execute the command ~# ./cluster-bootstrap.sh

    1. Enter the vCenter FQDN or IP address and administrator level username and password.

    2. Wait for the system management service to restart and the bootstrap process to complete. Verify if the HX Data Platform Plug-in is now updated.

  8. Log out from the cluster management IP controller VM.

  9. Log out of vSphere Web Client.


    Note

    Do not merely close the browser.

  10. Log in to vSphere Web Client again to refresh the HX Data Platform Plug-in.

  11. Verify the plugin version in vCenter by navigating to Administration > Client Plug-Ins > Springpath Plugin in the vSphere Web Client. Confirm the current version matches the new version you are upgrading to.

Upgrade the HXDP/UCS Firmware from the Plugin UI


Note

This workflow should only be used if the current HXDP version is before 3.5(1a).

If the current HXDP Release is 3.5(1a) or later, upgrade using HX Connect.

  1. From the vSphere Web Client Navigator, select vCenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > HX-Cluster > Summary.

  2. Select Upgrade Cluster.

  3. Select both, HX Data Platform and UCS Firmware.

  4. Click Next.

  5. Browse and select the HXDP package storfs-packages-<version>.tgz.

  6. To verify the integrity of the uploaded upgrade package bundle, enter administrator level vCenter credentials.

    1. Under Advanced Options, enter the MD5 Checksum # information. The file checksum can be found on the Cisco.com download page by clicking on the download title to reveal the md5 checksum.

  7. Enter administrator level UCS Manager credentials.

  8. To view the current firmware package version, click Discover.

  9. Type the latest version of Cisco UCS firmware in the Target version field.


    Important

    Type the release number exactly as shown, for example 4.0(4k). Confirm the desired version of UCS C-series and B-series firmware bundles are uploaded to the UCSM before you start the upgrade process.

  10. Click Upgrade.

    The Cisco UCS servers are now upgraded with the desired firmware packages. The pending activities are automatically acknowledged in a rolling fashion.


    Note

    The upgrade process performs the rolling reboot of each node in the cluster without any traffic disruptions.

    If the upgrade fails, you can re-try the upgrade or contact Cisco TAC for further assistance. Running a cluster without remediation after an upgrade failure is not recommended. Care should be taken to fully complete the upgrade as soon as possible.

Workflow 2: Upgrading to the Destination Release

Upgrading to the Destination Release

Perform the following tasks, using the specified software releases to upgrade to your destination release:

Software Downloads: From the Intermediate Release to the Destination Release

The second upgrade you need to complete is to upgrade from the intermediate release to the destination release. Download the software bundles needed for your upgrade from the Cisco Software Downloads.

Software Download

Software Download for the Recommended UCS Firmware

Before starting the upgrade process, download and install the UCS Infra Package that matches your installation as well as the UCS Blade, and the UCS C-series Package.


Note

At the time of authoring 4.0(4k) was the compatible UCS server firmware for the suggested HXDP release of 3.5(2h) and 3.5(2i). Before selecting a UCS server firmware to use, start with the current suggested HX release, and choose the corresponding UCS server firmware. For more information about the latest suggested release, see Cisco HyperFlex Recommended Software Release and Requirements Guide.

Download Link: https://software.cisco.com/download/home/286305544/type/286305994/release/3.5(2i)

UCS Infra Package Download

UCS 6200 Fabric Interconnects: ucs-k9-bundle-infra.4.0.4k.A.bin

6300 series UCS Infra package: ucs-6300-k9-bundle-infra.4.0.4k.A.bin

6400 series UCS Infra package: ucs-6400-k9-bundle-infra.4.0.4k.A.bin

UCS Blade Package:

ucs-k9-bundle-b-series.4.0.4k.B.bin

UCS C-series Package:

ucs-k9-bundle-c-series.4.0.4k.C.bin

Software Download for HXDP Release 4.0(2f) or the latest suggested release

Before starting the upgrade process, download and install all software bundles needed to complete this upgrade.


Note

At the time this document was authored, HDPX Release 4.0(2f) was the suggested release. For more information about how to locate the latest suggested release on the Cisco Software Download page, see Navigating the Cisco HyperFlex Data Platform Software Downloads and the Cisco HyperFlex Recommended Software Release and Requirements Guide.

Upgrade the UCS Infrastructure Package

Before you begin:

Configuring UCS Infrastructure

Perform the following steps:

  1. Open the UCS Manager GUI.

  2. Select Equipment > Firmware Management > Firmware auto-install

  3. Click Install Infrastructure Firmware.

    1. The Install Infrastructure Firmware window appears, select ignore all.

  4. Select the desired UCS infrastructure version. Refer the compatibility matrix to identify the version desired for your use case.

    1. Click Next.

  5. Check the Upgrade Now box.

  6. Click Finish.


    Note

    The expected upgrade behavior is for the UCS Manager to stop and then restart with the new version. Wait until the UCS Manager is back online to log back in to UCS Manager and complete the next steps.

    You may check the Ignore All box for warnings are not critical to user environment.

  7. If the UCS blade server chassis is present, wait for IO Modules (IOM) to upgrade.

    1. Select Equipment > Installed Firmware. Expand each chassis and check the Update Status of the IO Module.

      • During upgrade, the IOM status is Upgrading.

      • When the update process completes, the IOMs state is Pending Next Boot for Activate status.

      • After the IOM upgrade is complete, the IOM state is Ready.

  8. Wait for Subordinate Fabric Interconnects (FI) to be activated.

    1. Select Equipment > Installed Firmware > Fabric Interconnects.

    2. Check the Activate Status of the kernel and switch images.


      Note

      During upgrade, the Activate Status is set to Activating.

  9. During FI reboot, all HX traffic is forwarded to the primary FI (based on ESXi vSwitch failover policy).

    • This will cause a brief traffic interruption.

    • This will not cause storage IO failures.

  10. Verify subordinate FI has rebooted and joined the UCS cluster.

    1. Select Equipment > Installed Firmware > Fabric Interconnects.

    2. After activation, the Activate Status of the FI is set to Ready.

    3. Verify that the Overall Status of the FI is operable.

    4. Verify that the kernel and switch versions of the FI match the desired and updated version.

    5. Verify that the FI has no fault.

    6. Verify that the FI cluster membership is Subordinate.

  11. If the UCS blade server chassis is present, wait for IO Modules (IOM) to upgrade.


    Note

    Only the IOMs connected to the subordinate FI will enter Ready state, IOMs attached to the Primary FI will remain in Pending Next Boot Activate Status.

    1. Select Equipment > Blade Chassis > IO Module.

    2. Wait for the Activate Status of the IOM to change to Ready.

  12. Wait until HX traffic is re-pinned to both FIs.

    Wait for UCS Manager vNIC faults to be cleared. The fault clearing indicates ESXi has loaded the ENIC driver and the interface is up. The traffic is not re-pinned immediately when the network interface goes up because ESXi has a fail back timer. But the Net.teampolicyupdelay timer is very low by default (100ms).

  13. Verify the HX Cluster is online, and healthy before rebooting the primary fabric interconnect.

    • Access summary tab from the vSphere Web Client Navigator. Select Home > vCenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > cluster > Summary.

  14. In the UCS manager GUI toolbar, click Pending Activities.

  15. Click on Fabric Interconnects tab that display the tasks requiring user acknowledgment before they can complete.

    1. Click Reboot Now for each pending activity that you want to deploy immediately.

    2. Click OK. Cisco UCS Manager immediately reboots the primary FI and makes the subordinate FI the primary FI (FI failover).

    During FI reboot, all HyperFlex traffic is forwarded to the new primary FI. This will cause a brief traffic interruption, but it will not cause storage IO failures.

  16. Wait for UCS Manager to disconnect, then reconnected the UCS Manager to the other FI. This step is necessary because a UCS fail over occurs during the primary FI reboot (step 15).

  17. Verify the FI settings:

    • Verify that the subordinate FI is primary.

    • Verify that the FI cluster membership is Primary.

  18. Once the FI is activated, Select Equipment > Installed Firmware > Fabric Interconnects.

  19. Wait for the Activate Status of the FI to be Ready and verify the following:

    • Check the Overall Status of FI is operable.

    • Check the FI has no fault.

    • Verify FI has rebooted and joined the UCS cluster as subordinate.

    • Check that the FI cluster membership is Subordinate.

  20. Wait for IOM activation to complete, then select Equipment > Blade Chassis > IO Module.

  21. Wait for the Activate Status of the IP module to be Ready.

    • You can monitor the status on the FSM tab.


      Note

      You will lose connectivity to UCS Manager throughout the entire upgrade. This is normal behavior

  22. Wait for the HX traffic to re-pin to both FIs.

  23. In the UCS manager GUI, wait for all server vNIC faults to clear.

  24. In the vSphere Web Client Navigator. Select Home > vCenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > cluster > Summary. Verify the HX Cluster is online and healthy after rebooting the FI.

Combined HXDP and UCS Firmware Upgrade

Upgrade the Plugin User Interface (UI)

  1. From the vSphere Web Client Navigator, select vCenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > cluster.

  2. Navigate to Actions > Summary and note the Cluster Management IP address.

  3. SSH to the cluster management IP address with root privileges.

  4. Transfer the latest HX Data Platform upgrade bundle to the controller VM's /tmp directory. Depending on your operating system, use you can either use SCP directly or download third-party tools, such as WinSCP or MobaXterm.

  5. From the controller VM shell, change to the /tmp directory.


    Warning

    Do not use any folder other than /tmp and do not create any subfolders.

  6. Uncompress and extract all files to the root using tar –zxvf <storfs package name>.tgz.

    Example:

    storfs-packages-3.5.2h-32139.tgz

  7. To invoke the cluster-bootstrap.sh script to bootstrap packages for upgrade, execute the command ~# ./cluster-bootstrap.sh

    1. Enter the vCenter FQDN or IP address and administrator level username and password.

    2. Wait for the system management service to restart and the bootstrap process to complete. Verify if the HX Data Platform Plug-in is now updated.

  8. Log out from the cluster management IP controller VM.

  9. Log out of vSphere Web Client.


    Note

    Do not merely close the browser.

  10. Log in to vSphere Web Client again to refresh the HX Data Platform Plug-in.

  11. Verify the plugin version in vCenter by navigating to Administration > Client Plug-Ins > Springpath Plugin in the vSphere Web Client. Confirm the current version matches the new version you are upgrading to.

Upgrade HX Data Platform and UCS Firmware

  1. Log in to HX Connect.

    1. Enter the HX Storage Cluster management IP address in a browser.

    2. Navigate to https://<storage-cluster-management-ip>.

    3. Enter the administrative username and password.

    4. Click Login.

  2. Select Upgrade types:

    • HX Data Platform

    • UCS Manager firmware

  3. Click Continue.

  4. Complete the following fields on the Enter Credentials page.

    Table 1. Upgrade HX Data Platform and UCS Manager Firmware

    UI Element

    Essential Information

    Drag the HX file here or click to browse

    Upload the latest Cisco HyperFlex Data Platform Upgrade Bundle for upgrading existing clusters with previous release.tgz package file from Software Download - Hyperflex HX Data Platform

    Example:

    storfs-packages-3.5.2h-32139.tgz

    Current version

    Displays the current HyperFlex Data Platform version

    Current cluster details

    Lists the HyperFlex cluster details like the Hypervisor version and Cluster upgrade state.

    Bundle version

    Displays the HyperFlex Data Platform version of the uploaded bundle.
    Table 2. vCenter Credentials

    UI Element

    Essential Information

    User Name field

    Enter the vCenter <admin> username.

    Admin Password field

    Enter the vCenter <root> password.

    (Optional) Checksum field

    The MD5 checksum can be obtained from the Cisco.com download page by hovering your mouse over the download package.

    This is an optional step that helps you verify the integrity of the uploaded upgrade package bundle.
    Table 3. UCS Manager Credentials

    UI Element

    Essential Information

    UCS Manager Host Name field

    Enter the Cisco UCS Manager host name.

    User Name field

    Enter the Cisco UCS Manager <admin> username.

    Admin Password field

    Enter the Cisco UCS Manager root password.

    Table 4. Discover Current Version

    UI Element

    Essential Information

    Discover button

    Click Discover to view the current UCS firmware package version, in the Current Version field.

    Current Version field

    Displays the current Cisco UCS firmware version.

    Target Version drop-down list

    To upgrade to a higher Cisco UCS firmware version, choose the version of Cisco UCS firmware

    Example: 4.0(4h)


    Note

    If you don't see the desired UCS firmware version in the drop-down, verify that the B-series and C-series firmware bundles are uploaded to UCSM and retry your action.

  5. Click Upgrade.

  6. The Validation Screen on the Upgrade Progress page displays the progress of the checks performed. Fix validation errors, if any. Confirm that the upgrade is complete.


    Note

    When the upgrade is in progress, you may see the error message, 'Websocket connection failed. Automatic refresh disabled'. To clear the error message, you can either refresh the page, or log out and log back in. You can safely ignore this error message.

    The upgrade process performs the rolling reboot of each node in the cluster without any traffic disruptions.

    If the upgrade fails, you can re-try the upgrade or contact Cisco TAC for further assistance. Running a cluster without remediation after an upgrade failure is not recommended. Care should be taken to fully complete the upgrade as soon as possible.