Detailed Pre-Upgrade Procedures

Important


Warning

This chapter contains a list of procedures required by various upgrade workflows. Follow only the procedures that are required for the specific upgrade workflow you intend to use. For step-by-step guidance on online and offline upgrades, refer to the Upgrade Procedures chapter.


Downloading Software

For a successful HyperFlex upgrade, the Cisco HyperFlex System component bundles can be downloaded from the Cisco HyperFlex Download website:

  1. HX Data Platform upgrade bundle (.tgz file)

  2. VMware ESXi Offline Zip bundle

  3. Cisco UCS infrastructure bundle, blade firmware bundle, and rack firmware bundle.

After the Cisco UCS bundles and firmware are downloaded, they need to be copied to Cisco UCS Manager.

To copy UCS software bundles to CIsco UCS Manager

Procedure


Step 1

Log in to the Cisco UCS Manager GUI.

Step 2

Enter the username and password.

Step 3

In the Navigation pane, click the Equipment tab.

Step 4

On the Equipment tab, click the Equipment node.

Step 5

In the Work pane, click Firmware Management > Installed Firmware > Download Firmware.

Step 6

In the Download Firmware dialog box, click the Local File System radio button in the Location of the Image File field and fill in the required fields.

Step 7

In the Filename field, enter the full path and name of the image file.

If you do not know the exact path to the folder where the firmware image file is located, click Browse and navigate to the file.

Step 8

Click OK. The Cisco UCS Manager GUI begins downloading the firmware bundle to the fabric interconnect.

Step 9

Monitor the status of the download on the Download Tasks tab.

Note 

If Cisco UCS Manager reports that the bootflash is out of space, delete obsolete bundles on the Packages tab to free up space. To view the available space in bootflash, navigate to Equipment > Fabric Interconnect > Local Storage Information and look in the Work pane area under the General tab.

Step 10

Repeat this task until all the required firmware and bundles are downloaded to the fabric interconnect.


Test Upstream Network Connectivity

Ensure that the hx-storage-data and vMotion upstream switches are configured for Jumbo Frames. Skipping this step could lead to input/output interruption during Cisco UCS infrastructure upgrade.

Procedure


Step 1

Put a node in Cisco HX Maintenance mode (see Entering Cisco HyperFlex Maintenance Mode).

Step 2

SSH to the ESXi host in step 1.

Step 3

Verify that the ping is working by pinging the corresponding vmk1 IP interface of another host.

If using Jumbo Frames:

vmkping -I vmk1 -d -s 8972 <data IP of address of another host>

If not using Jumbo Frames:

vmkping -I vmk1 -d -s 1472 <data IP of address of another host>

Note 

Early Hyperflex installs used vmnic2 and vmnic3 for hx-storage-data Vswitch. vmnic5 and vmnic1 are supported from HX 3.5(x) and later releases.

Please verify which interfaces are used for testing the hx-storage-data Vswitch failover either through Vcenter GUI, or SSH to the ESXi host using esxcfg-vswitch-l command for your clusters ESXi hosts.

Step 4

Swap the active interfaces in vswitch-hx-storage-data to force traffic upstream.

esxcli network vswitch standard policy failover set -a vmnic2 -s vmnic3 -v vswitch-hx-storage-data

Step 5

Again, verify that the ping is working by pinging the corresponding vmk1 IP interface of another host.

If using Jumbo Frames:

vmkping -I vmk1 -d -s 8972 <data IP of address of another host>

If the ping fails, try again with:

vmkping -I vmk1 -d -s 1472 <data IP of address of another host>

If not using Jumbo Frames:

vmkping -I vmk1 -d -s 1472 <data IP of address of another host>

Note 

If the ping fails, do not proceed to upgrade the Cisco UCS firmware. Investigate the network configuration including upstream switch to identify cause of failure.

Step 6

Swap interface back to defaults even if ping fails.

esxcli network vswitch standard policy failover set -a vmnic3 -s vmnic2 -v vswitch-hx-storage-data

Step 7

Exit the node from Cisco HX Maintenance mode (see Exiting Cisco HyperFlex Maintenance Mode).


Graceful Shutdown of a HX Cluster


Warning

This chapter contains a list of procedures required by various upgrade workflows. Follow only the procedures that are required for the specific upgrade workflow you intend to use. For step-by-step guidance on online and offline upgrades, refer to the Upgrade Procedures chapter.


Procedure


Step 1

SSH to any controller VM in the cluster.

Step 2

Check cluster health ~#stcli cluster info | grep health.

Step 3

If healthy, shutdown the cluster ~#stcli cluster shutdown.

Step 4

Shutdown takes a few minutes, wait for the prompt to return.


Modifying Host Firmware Package Using Cisco UCS Manager

Host Firmware Package is set automatically, during CIsco UCS infrastructure upgrade. To manually stage the correct firmware version before starting the upgrade process use the following procedure.

Procedure


Step 1

Sign in to Cisco UCS Manager.

Step 2

In the Navigation pane, click Servers.

Step 3

Expand Servers > Policies > Sub-Organizations > hx-cluster.

Step 4

Expand Host Firmware Packages and choose the policy you want to update.

Step 5

In the Work pane, click the General tab.

Step 6

To modify the components in the host firmware package, click Modify Package Versions.

Step 7

Modify value for Blade Package and Rack Package, to the latest firmware version. See Recommended FI/Server Firmware - 4.0(x) Releases for the list of recommended UCS FI firmware.

Step 8

In the Excluded Components area, check the boxes corresponding to the components that you want to exclude from this host firmware package.

Step 9

Click OK. Click yes for all the warnings.


What to do next

Verify pending activities.

User acknowledgement of each server is performed automatically during upgrade. Optionally, you can manually acknowledge pending activities on the HyperFlex node.

On the Cisco UCS Manager toolbar, click Pending Activities.

The User Acknowledged Activities tab lists the HyperFlex nodes that require user acknowledgment within the selected sub-org. They are in Pending Reboot status.


Note

Do not acknowledge service profiles.


Auto Bootstrap Upgrade Process from HX Connect UI

Before you begin

If you are running HyperFlex release 3.5(1a) or later, you can upgrade the Cisco HX Data Platform by performing the auto-bootstrap process from the HX Connect UI. However, if you are running a HyperFlex release that is earlier than release 3.5(1a) you must run the manual bootstrap process to upgrade the Cisco HX Data Platform as outlined in Manual Bootstrap Upgrade Process.

Procedure


Step 1

Log in to HX Connect.

  1. Enter the HX Storage Cluster management IP address in a browser. Navigate to https://<storage-cluster-management-ip>.

  2. Enter the administrative username and password.

  3. Click Login.

Step 2

In the Navigation pane, select Upgrade. The Select Upgrade Type page appears.

Figure 1. Select Upgrade Type Page
Step 3

Upload the new Cisco HX Data Platform upgrade package, specify vCenter credentials, and then click Upgrade.

Figure 2. Uploading a Cisco HX Data Platform Upgrade File
Step 4

Click Confirm in the Initiating Pre-Upgrade message box. The Pre-Upgrade process begins and will be performed on all nodes.

Figure 3. Initiating the Pre-Upgrade Process
Figure 4. Pre-Upgrade Progress Screen
Step 5

When the Pre-Upgrade upgrade (management packages upgrade) process is completed, the HX Connect UI prompts you so with an onscreen message. The software will again ask you which component you want to upgrade. Click Upgrade again the complete the HX data platform part of the upgrade.

Note 

The cluster upgrade is not complete until the full upgrade is initiated. The upgrade is only partially complete after the pre-upgrade process (as outlined in step 4).

Figure 5. Completing the Pre-Upgrade Process

Manual Bootstrap Upgrade Process

The manual bootstrap process enables you to upgrade the Cisco HX Data Platform and the Cisco HX Data Platform Plug-in.


Note

Perform this procedure on the node that has the Cluster Management IP address.


Procedure


Step 1

From the vSphere Web Client Navigator, select vCenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > cluster.

Step 2

Navigate to Actions > Summary and note the Cluster Management IP address.

Step 3

SSH to the cluster management IP address with root privileges.

Step 4

Transfer the latest HX Data Platform upgrade bundle to the controller VM's /tmp directory.

Depending on your operating system, use you can either use SCP directly or download third-party tools, such as WinSCP or MobaXterm.

Step 5

From the controller VM shell, change to the /tmp directory.

Warning 

Do not use any folder other than /tmp and do not create any subfolders.

Step 6

Un-compress the package using tar –zxvf <storfs package name>.tgz.

Example:

This un-compresses and extracts all files to the root of the /tmp folder.

Step 7

Invoke the cluster-bootstrap.sh script to bootstrap packages for upgrade. Execute the command

~# ./cluster-bootstrap.sh

Enter the vCenter FQDN or IP address and administrator level username and password.

Wait for the system management service to restart and the bootstrap process to complete. Verify if the HX Data Platform Plug-in is now updated.


Verify vMotion Configuration for HX Cluster

Before you perform maintenance operations on the Cisco HyperFlex (HX) cluster, verify all nodes in the HX cluster are configured for vMotion. Confirm the following from your vSphere Web Client:

  1. Verify that the vMotion port group is configured with vmnic3 and vmnic7 in an active/standby configuration across all of the ESXi hosts in the cluster.

  2. Verify that a port group is configured for vMotion, and that the naming convention is EXACTLY the same across all ESXi hosts in the cluster.


    Note

    The name is case-sensitive.


  3. Verify that you have assigned a static IP to each vMotion port group, and that the static IPs for each vMotion port group are in the same subnet.


    Note

    The static IP address is defined as a VMKernel interface.


  4. Verify that the vMotion port group has the vMotion option checked in the properties, and that no other port groups (such as management) have this option checked, on each ESXi host in the cluster.

  5. Verify in the settings that the vMotion port group is set to 9000 MTU, (if you are using jumbo frames), and the VLAN ID matches the network configuration for the vMotion subnet.

  6. Verify you can ping from the vMotion port group on one ESXi host to the vMotion IP on the other host.

    Type vmkping -I vmk2 -d -s 8972 <vMotion IP address of neighboring server>

Entering Cisco HyperFlex Maintenance Mode

Using the Cisco HyperFlex (HX) Connect User Interface


Note

Maintenance Mode was introduced in Cisco HyperFlex Release 2.5(1a) and 2.5(1b).


  1. Log in to Cisco HX Connect: https://<cluster management ip>.

  2. In the menu, click System Information.

  3. Click Nodes, and then click the row of the node you want to put in to maintenance mode.

  4. Click Enter HX Maintenance Mode.

  5. In the Confirm HX Maintenance Mode dialog box, click Enter HX Maintenance Mode.


    Note

    After you complete any maintenance tasks, you must manually exit HX maintenance mode.


Using the vSphere Web Client

  1. Log in to the vSphere web client.

  2. Go to Home > Hosts and Clusters.

  3. Expand the Datacenter that contains the HX Cluster.

  4. Expand the HX Cluster and select the node.

  5. Right-click the node and select Cisco HX Maintenance Mode > Enter HX Maintenance Mode.

Using the Command-Line Interface

  1. Log in to the storage controller cluster command line as a user with root privileges.

  2. Move the node into HX Maintenance Mode.

    1. Identify the node ID and IP address.

      # stcli node list --summary

    2. Enter the node into HX Maintenance Mode.

      # stcli node maintenanceMode (--id ID | --ip IP Address) --mode enter

      (see also stcli node maintenanceMode --help)

  3. Log in to the ESXi command line of this node as a user with root privileges.

  4. Verify that the node has entered HX Maintenance Mode.

    # esxcli system maintenanceMode get

You can monitor the progress of the Enter Maintenance Mode task in vSphere Web Client, under the Monitor > Tasks tab.

If the operation fails, an error message displays. Try to fix the underlying problem and attempt to enter maintenance mode again.

Exiting Cisco HyperFlex Maintenance Mode

Using the Cisco HyperFlex (HX) Connect User Interface


Note

Maintenance Mode was introduced in Cisco HyperFlex Release 2.5(1a) and 2.5(1b).


  1. Log in to HX Connect: https://<cluster management ip>.

  2. In the menu, click System Information.

  3. Click Nodes, and then click the row of the node you want to remove from maintenance mode.

  4. Click Exit HX Maintenance Mode.

Using the vSphere Web Client

  1. Log in to the vSphere web client.

  2. Go to Home > Hosts and Clusters.

  3. Expand the Datacenter that contains the HX Cluster.

  4. Expand the HX Cluster and select the node.

  5. Right-click the node and select Cisco HX Maintenance Mode > Exit HX Maintenance Mode.

Using the Command-Line Interface

  1. Log in to the storage controller cluster command line as a user with root privileges.

  2. Exit the node out of HX Maintenance Mode.

    1. Identify the node ID and IP address.

      # stcli node list --summary

    2. Exit the node out of HX Maintenance Mode.

      # stcli node maintenanceMode (--id ID | --ip IP Address) --mode exit

      (see also stcli node maintenanceMode --help)

  3. Log in to the ESXi command line of this node as a user with root privileges.

  4. Verify that the node has exited HX Maintenance Mode.

    # esxcli system maintenanceMode get

You can monitor the progress of the Exit Maintenance Mode task in vSphere Web Client, under the Monitor > Tasks tab.

If the operation fails, an error message displays. Try to fix the underlying problem and attempt to exit maintenance mode again.