Chapter 1. Overview

- Overview

Overview

Cisco HX Data Platform upgrade for clusters running on Microsoft Hyper-V is completed one node at a time in HX Connect UI. The following table provides a summary of the automated steps that DO NOT require manual intervention.

#

Step

Result

1

Pre-upgrade validations

- Windows Failover Cluster is configured.

- All non-clustered VMs are powered off. These VMs are either shutdown or manually migrated to another node.

- HX Cluster is healthy.

Run the Hypercheck: HyperFlex Health & Pre-upgrade Check Tool - HyperV

- Live migration is configured.

2

Enter HX Connect Maintenance Mode

- Pause the Failover Cluster node that in addition triggers Live migration of clustered VMs.

- Stop the storfs service on the controller VM.

3

Upgrade

- Upgrade storfs packages in the controller VM.

- Reboot the controller VM.

4

Exit Maintenance Mode

- Start the storfs service on the controller VM.

- Resume Failover Cluster Node that will trigger failback for the clustered VMs in Step #2.

Chapter 2. Pre-Upgrade Procedures

- Test Upgrade Eligibility

- Hypercheck: HyperFlex and Pre-Upgrade Check Tool - Hyper-V

- Upgrade Recommendations

- Bootstrap Process

Test Upgrade Eligibility

Beginning with Cisco HyperFlex Release 4.0(2a), the Upgrade page displays the last cluster upgrade eligibility test result and last tested version of UCS server, HX data platform, and/or ESXi.

Before upgrading the HyperFlex Data Platform, perform upgrade eligibility test in the Upgrade page to validate and check the cluster readiness and the infrastructure compatibility for an upgrade.

To perform upgrade eligibility test:

1. Select Upgrade > Test Upgrade Eligibility.

2. Select the HX Data Platform check box to test upgrade eligibility of HyperFlex Data Platform.

Upload the Cisco HyperFlex Data Platform Upgrade Bundle that need to be validated before upgrade.

3. Click Validate.

The progress of the upgrade eligibility test is displayed.

Hypercheck: HyperFlex and Pre-Upgrade Check Tool - Hyper-V

The Hypercheck : Hyperflex Health & Pre-Upgrade Check Tool - HyperV are automated health and pre-upgrade checks that are designed to ensure your clusters are healthy before you upgrade. It is imperative that this health check is not just performed, but that you take corrective action on any cluster that is found to be unhealthy. Correct all issues reported by the Hypercheck health check before continuing.

Hypercheck : HyperFlex Health & Pre-Upgrade Check Tool - HyperV (HyperFlex Releases 3.5 and later) https://www.cisco.com/c/en/us/support/docs/hyperconverged-infrastructure/hyperflex-hx-data-platform/216027-hypercheck-hyperflex-health-pre-upgr.html

Upgrade Recommendations

For upgrading supported releases, see the Recommended Cisco HyperFlex HX Data Platform Software Releases - for Cisco HyperFlex HX-Series Systems.

If you want to upgrade from a release that is no longer supported, see the Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HX Releases.

Bootstrap Process

Bootstrap process enables you to upgrade Cisco HX Data Platform.

Important

This procedure is required if you are upgrading from HXDP release 3.0(1x) to 3.5(1a) and later.

Procedure

Step 1

SSH to the cluster management IP address with root privileges.

Step 2

Transfer the latest HX Data Platform upgrade bundle to the controller VM's /tmp directory. Depending on your operating system, use you can either use SCP directly or download third-party tools, such as WinSCP or MobaXterm.

Step 3

From the controller VM shell, change to the /tmp directory

Caution

Do not use any folder other than /tmp and do not create any subfolders.

Step 4

Un-compress the package using tar –zxvf <storfs package name>.tgz.

tar –zxvf storfs-packages-3.0.1a-26263.tgz

This un-compresses and extracts all files to the root of the /tmp folder.

Step 5

Invoke the cluster-bootstrap.sh script to bootstrap packages for upgrade. Execute the following command:

~# ./cluster-bootstrap.sh

Important

Wait for the system management service to restart and the bootstrap process to complete.

Step 6

Log out from the cluster management IP controller VM.

Chapter 3. Upgrading Cisco HX Data Platform using HX Connect

- Upgrading Cisco HX Data Platform using HX Connect UI

- Upgrading UCS Firmware

Upgrading Cisco HX Data Platform using HX Connect UI

Procedure

Step 1

Log into HX Connect.

1. Enter the HX Storage Cluster management IP address in a browser. Navigate to https://<storage-cluster-management-ip>.

2. Enter the administrative username and password.

3. Click Login.

Step 2

On the Select Upgrade Type page, select the HX Data Platform check box.

Step 3

In the Drag the HX file here or click to browse area, upload the latest Cisco HyperFlex Data Platform Upgrade Bundle for upgrading existing clusters with previous release.tgz package file from Software Download - HyperFlex HX Data Platform.

For example:

storfs-packages-4.0.2c-35590.tgz

Step 4

Click Upgrade.

Step 5

The Validation Screen on the Upgrade Progress page displays the progress of the checks performed. Fix validation errors, if any. Confirm that the upgrade is complete.

Note

As of HX 5.0(1b), an upgrade status appears providing the result of the last upgrade along with the versions that were upgraded (source and target versions for each component selected in the upgrade). You can dismiss this status only if it is successful. If the last upgrade fails, you will need to fix the issue. This banner is a reminder to take action to correct the upgrade.

Upgrading UCS Firmware

After you have successfully upgraded the Cisco HX Data Platform, change the UCS host firmware policy accordingly with the recommended UCS firmware. For more information, see the HyperFlex release notes.

Chapter 4. Troubleshooting Information

- Troubleshooting Information

Troubleshooting Information

Symptom

Resolution

Connection to HX Connect lost after pre-upgrade step from HX 3.5(2g) to HX 4.0(2a). During the upgrade, if there is an expired certificate in the upgrade source version, the browser will log user out after pre-upgrade step. This is accepted secure behavior since the certificate of the server has changed after pre-upgrade.

Refresh the browser and login again.

Validation fails when the non-clustered VMs are running on the Hyper-V host that is being placed in HX Maintenance Mode.

Power off the non-clustered VMs or manually migrate them to another node.

VM Live migration can fail while pausing the Failover cluster node.

This can be due to several underlying issues related to resources on the Windows host or networking failure between the nodes within the failover cluster.

Check the Failover cluster event log to debug the issue.

Retry placing the node in HX Maintenance Mode after resolving these issues.

When the cluster is configured with rf=2 + strict access policy, the cluster resiliency state remains in warning state and the online upgrade fails due to the resiliency state.

Change strict mode to lenient mode temporarily before upgrading and then change back after the upgrade is complete:

sysmtool --ns cluster --cmd accesspolicy --accesspolicy=lenient

After a cluster is upgraded to HX 4.0(2a) from any previous version, the user may see a logout screen with a message indicating:

Invalid CSRF token

Re-log back in again. This will generate CSRF tokens to validate all future requests between the client and server.

Option to upgrade UCS does not appear in HX Connect.

Verify that all backend services are up and running:

1. Verify that stNodeMgr is running on Hyper-V clusters.

2. Verify that stUpgradeSvc is running on Hyper-V clusters.

3. If any of the services are stopped, start them by running start <service-name>, where <service-name> is stNodeMgr or stMgr or stUpgradeSvc.

During UCS Server Firmware Upgrade, when the node that is not part of the Fail Over Cluster (FOC) being used, then upgrade fails with Pause HyperV Host: Suspend-ClusterNode -Drain. An error occurred pausing node 'node not in FOC'.

Remove one of the FOCs from the node which shows as failed WFC and make sure that the working FOC has all the nodes of the cluster.

Perform the following on all stCtlVms of every node:

Run

1) /bin/hxdpservices stop

2) /bin/hxdpservices start

Wait for 5 minutes

Start UCS Server Firmware upgrade again

Upgrade process already in progress even after failure in a Hyper-V setup.

Fetching upgrade status from ZK("i32":1)

[zk: localhost:2181(CONNECTED) 0] get
/hxUpgrade/clusterUpgradeStatus
/2802157519622062146:7345585794604318257
{"1":{"rec":{"1":{"str":"2802157519622062146:
7345585794604318257"},
"2":{"i32":4},"3":{"str":"ucs1796smb"}}},
"2":{"i32":1},"4":
{"str":"5.0.2a-41393"}}

Updating upgrade status in ZK. ("i32":0)

[zk: localhost:2181(CONNECTED) 3]
set /hxUpgrade/clusterUpgradeStatus/
2802157519622062146:
7345585794604318257 '{"1":{"rec":
{"1":{"str":"2802157519622062146:7
345585794604318257"},
"2":{"i32":4},"3":{"str":"ucs1796smb"}}},"
2":{"i32":0},"4":
{"str":"5.0.2a-41393"}}'-

Chapter 5. Applying Windows Patches

- Applying Windows Patch Updates

Applying Windows Patch Updates

Windows Service Pack updates should be completed on one Hyper-V Server at a time to prevent the HyperFlex cluster from experiencing an outage. Multiple Hyper-V hosts booting at the same will cause their associated controller VMs to be offline and a HyperFlex cluster with 5 or more nodes can only tolerate 2 controller VMs being offline at the same time.

Note

Clusters comprised of 3 or 4 nodes can only tolerate one node to be down at a time.

Perform the following procedure for updating Windows Service Pack that includes cluster health checks.

Procedure

Step 1

Log into a controller VM to verify that the cluster is healthy. Check for The Resiliency Health and “# of node failures tolerable” lines. Make a note of how many node failures are tolerable by your HX cluster.

Step 2

Run the hxcli cluster info command.

Sample Output:
Cluster Name : SUP61-SMB
Cluster UUID : 3789536550516014784:5974749742096294602
Cluster State : ONLINE
Cluster Access Policy : Lenient
Space Status : NORMAL
Raw Capacity : 92.4 TB
Total Capacity : 30.8 TB
Used Capacity : 269.8 GB
Free Capacity : 30.5 TB
Compression Savings : 82.61%
Deduplication Savings : 0.00%
Total Savings : 82.61%
# of Nodes Configured : 4
# of Nodes Online : 4
Data IP Address : SUP61-CIP.hx.local
Resiliency Health : HEALTHY
Policy Compliance : COMPLIANT
Data Replication Factor : 2 Copies
# of node failures tolerable : 1
# of persistent device failures tolerable : 2
# of cache device failures tolerable : 2
Zone Type : Unknown
All Flash : No

Step 3

Select a Hyper-V host to be updated.

Step 4

Log in to HX Connect with the HyperFlex Admin Service account.

Step 5

Navigate to System Information > Nodes.

Step 6

Click on the Hyper-V host to be updated. The host entry is highlighted in a light blue color.

Step 7

In the Nodes tab, Click Enter HX Maintenance Mode.

Step 8

An HX Maintenance Mode screen displays with an alert for maintenance mode operations.

Step 9

In Failover Cluster Manager, verify in the Roles screen that all VMs have been evacuated from the selected Hyper-V Host with the exception of the Controller VM (StCtlVM). The controller VM cannot be reallocated.

Step 10

Perform the Window Update on the selected Hyper-V host.

Step 11

After all the reboots and software updates have been completed in the HX Connect UI, System Information > Nodes, select the updated Hyper-V host and click Exit HX Maintenance Mode.

Step 12

Log into a controller VM and use the hxcli cluster info check command to verify that the cluster is healthy and can tolerate the same number of node failures in Step 1.

Note

This operation may take a few minutes before the cluster returns to a healthy state.

Step 13

Once the cluster is healthy and can tolerate the same number of node failures in Step 1 then repeat Steps 2 – 12 until all Hyper-V hosts have been updated.

Communications, Services, Bias-free Language, and Additional Information

- To receive timely, relevant information from Cisco, sign up at Cisco Profile Manager.

- To get the business impact you’re looking for with the technologies that matter, visit Cisco Services.

- To submit a service request, visit Cisco Support.

- To discover and browse secure, validated enterprise-class apps, products, solutions and services, visit Cisco Marketplace.

- To obtain general networking, training, and certification titles, visit Cisco Press.

- To find warranty information for a specific product or product family, access Cisco Warranty Finder.

Documentation Feedback

To provide feedback about Cisco technical documentation, use the feedback form available in the right pane of every online document.

Cisco Bug Search Tool

Cisco Bug Search Tool (BST) is a web-based tool that acts as a gateway to the Cisco bug tracking system that maintains a comprehensive list of defects and vulnerabilities in Cisco products and software. BST provides you with detailed defect information about your products and software.

Bias-Free Language

The documentation set for this product strives to use bias-free language. For purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on standards documentation, or language that is used by a referenced third-party product.

Full Cisco Trademarks with Software License

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS" WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.

All printed copies and duplicate soft copies of this document are considered uncontrolled. See the current online version for the latest version.

Cisco has more than 200 offices worldwide. Addresses and phone numbers are listed on the Cisco website at www.cisco.com/go/offices.

The documentation set for this product strives to use bias-free language. For purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on standards documentation, or language that is used by a referenced third-party product.

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: https://www.cisco.com/c/en/us/about/legal/trademarks.html. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1721R)