Chapter 1. Overview

About This Guide

Upgrade Modes

Cisco HyperFlex HX-Series System Compatibility Matrix

Supported Upgrade Paths

Chapter 2. Upgrade Scenarios and Worflow

Upgrade Scenarios

Single Cluster Environment

Multi Cluster Environment

Addition of a New Cluster on the Same HX Domain

Addition of a New HX Node to an Existing Cluster

Chapter 3. Prerequisites and Guidelines

Overview

Prerequisites

Cautions and Guidelines

HyperFlex Node Upgrade Validations

Viewing HyperFlex Cluster Health

Verifying If DRS Is Enabled

Viewing ESX Agent Manager

Verify Health of HyperFlex Cluster In Cisco UCS Manager

Configuring vMotion Interfaces

Configure Lenient Mode

Chapter 4. Preparing For Upgrade

Downloading Software

Test Upstream Network Connectivity

Graceful Shutdown of a HX Cluster

Modifying Host Firmware Package Using Cisco UCS Manager

Bootstrap Process

Entering Cisco HyperFlex Maintenance Mode

Exiting Cisco HyperFlex Maintenance Mode

Chapter 5. Upgrade Procedures

Online Upgrade Process Workflow

Offline Upgrade Process Workflow

Offline Upgrade Guidelines

Combined Upgrade - Offline Mode

Split Upgrade - Offline Mode

Start Cluster and Power On VMs

Chapter 6. Upgrading Cisco HyperFlex Software Components

Upgrading Cisco UCS Firmware Using Cisco UCS Manager

Upgrading Cisco HX Data Platform Software

Upgrading Cisco UCS Firmware Using vSphere Web Client

Combined Upgrade of Cisco HX Data Platform and Cisco UCS Firmware

Upgrading ESXi

Upgrade of Compute-only Nodes

Upgrading Cisco UCS Firmware on Compute-only Node from version 2.2(6f) to 3.1(2b)

Chapter 7. Post Upgrade Tasks

Confirm Upgrade is Complete

Check Firmware Versions in UCSM

Upgrading VMware Tools in Storage Controller VMs

Chapter 8. Known Issues

Overview

Upgrade Button Missing in Cisco HX Data Platform

Cluster Start Command Error: Node Not Available

A Node Fails to Upgrade due to vCenter Issues

Copyright Page

Chapter 1. Overview

- About This Guide

- Upgrade Modes

- Cisco HyperFlex HX-Series System Compatibility Matrix

- Supported Upgrade Paths

About This Guide

This document describes how to upgrade an existing installation of Cisco HX Data Platform .

Cisco HyperFlex Systems has several components that may be upgraded depending on your environment. HyperFlex Systems core components are:

- Cisco UCS server firmware

- Cisco HX Data Platform software

- VMware ESXi software

The upgrade order for the endpoints depends on the upgrade path. See Upgrade Scenarios to determine the appropriate upgrade path.

Upgrade Modes

Cisco HyperFlex Systems upgrade deployment can be performed in either online or offline mode.

Online Upgrade Mode

Use the online upgrade mode if you cannot tolerate downtime and the maintenance window is permissible.

During an online upgrade, all virtual machines remain in a running state without service disruption. User intervention is not required after the upgrade starts, or when DRS is licensed, enabled, and set to automatic mode. Cisco recommends performing the upgrade during a maintenance window when one HX node is already offline. See Online Upgrade Process Workflow for more details.

Note

To upgrade a nested VMware vCenter (a vCenter on an HX datastore) use the online upgrade mode.

Offline Upgrade Mode

Use the offline upgrade mode if you desire a shorter time window and workload downtime is acceptable.

Offline upgrade mode means the Cisco HX Data Platform is offline. However, storage controller VMs are online and HX Data Platform management is viewable through both the stcli command-line access and the VMware vSphere Web Client. The vSphere Web Client reports on the storage I/O layer. The stcli cluster command returns that overall storage cluster status is offline. Refer to Offline Upgrade Process Workflow.

Cisco HyperFlex HX-Series System Compatibility Matrix

See resolved and open caveats before upgrading. Review the new features, refer to the Cisco HX Data Platform Release Notes.

For a complete list of hardware and software interdependencies, refer to respective UCSM release version of Hardware and Software Interoperability for Cisco HyperFlex HX-Series.

Supported HyperFlex Software Versions

HyperFlex Release

HX Data Platform Installer

HX Data Platform

Recommended UCS FI Firmware

2.1(1b)

2.1(1b)

2.1(1b)

3.1(2g)

2.0(1a)

2.0(1a)

2.0(1a)

3.1(2f)

1.8(1f)

1.8(1f)

1.8(1f)

3.1(2b)

1.8(1e)

1.8(1e)

1.8(1e)

3.1(2b)

1.8(1c)

1.8(1c)

1.8(1c)

3.1(2b)

1.8(1b)

1.8(1b)

1.8(1b)

3.1(2b)

1.8(1a)

1.8(1a)

1.8(1a)

3.1(2b)

1.7.3

1.7.3

1.7.3

2.2(7c)

1.7.1

1.7.1-14835

1.7

2.2(7c)

1.7.1

1.7.1

1.7

2.2(6f)

Supported VMware vSphere Versions and Editions

HyperFlex Version

vSphere Versions

vSphere Editions

2.1(1b)

6.0 U1b, 6.0 U2, 6.0 U2 Patch 3, 6.0 U2 Patch 4, 6.0 U3.

Enterprise, Enterprise Plus, Standard, Essentials Plus, ROBO

2.0(1a)

6.0 U1b, 6.0 U2, 6.0 U2 Patch 3, 6.0 U2 Patch 4

Enterprise, Enterprise Plus, Standard, Essentials Plus, ROBO

1.8(1f)

6.0 U1b, 6.0 U2, 6.0 U2 Patch 3

Enterprise, Enterprise Plus, Standard, Essentials Plus, ROBO

1.8(1e)

6.0 U1b, 6.0 U2, 6.0 U2 Patch 3

Enterprise, Enterprise Plus, Standard, Essentials Plus, ROBO

1.8(1c)

6.0 U1b, 6.0 U2, 6.0 U2 Patch 3

Enterprise, Enterprise Plus, Standard, Essentials Plus, ROBO

1.8(1c)

5.5 U3

Enterprise, Enterprise Plus

1.8(1b)

6.0 U1b, 6.0 U2, 6.0 U2 Patch 3

Enterprise, Enterprise Plus, Standard, Essentials Plus, ROBO

1.8(1a)

6.0 U1b, 6.0 U2, 6.0 U2 Patch 3

Enterprise, Enterprise Plus, Standard, Essentials Plus, ROBO

1.7.3

6.0 U1b, 6.0 U2

Enterprise, Enterprise Plus, Standard, Essentials Plus, ROBO

1.7.3

5.5 U3

Enterprise, Enterprise Plus

1.7.1

6.0 U1b

Enterprise, Enterprise Plus,Standard, Essentials Plus, ROBO

1.7.1

5.5 U3

Enterprise, Enterprise Plus

Supported Upgrade Paths

The following four table matrices map the supported upgrade paths for Cisco HyperFlex Systems.

Cisco HX Data Platform Matrix

Cisco HX Data Platform Version

Supported Upgrade Paths

2.1(1b)

2.0(1a)

1.8(1f)

1.8.(1f)

Yes

Yes

—

1.8(1e)

Yes

Yes

Yes

1.8(1c)

Yes

Yes

Yes

1.8(1b)

Yes

Yes

Yes

1.8(1a)

Yes

Yes

Yes

1.7.x

Yes

Yes

Yes

Prior to upgrading to HXDP 2.1(1b), ensure that you upgrade UCS server firmware to 3.1(2f) or 3.1(2g). Either a combined or split upgrade can be selected through the vSphere Web Client plugin. Direct firmware upgrade through Cisco UCS Manager should not be attempted.

Cisco UCS Manager Matrix

Cisco UCS Manager Version

Supported Upgrade Paths

3.1(2g)

3.1(2f)

3.1(2e)

3.1(2c)

3.1(2b)

2.2(7c)

3.1(2f)

Yes

—

—

—

—

—

3.1(2e)

Yes

Yes

—

—

—

—

3.1(2c)

Yes

Yes

Yes

—

—

—

3.1(2b)

Yes

Yes

Yes

Yes

—

—

2.2(7c)

Yes

Yes

Yes

Yes

Yes

—

2.2(6f)

Yes

Yes

Yes

Yes

Yes

Yes

VMware ESXi Matrix

- If you have the ESXi 5.5 U3b version, we recommend an ESXi upgrade. Contact Cisco TAC for additional information.

- If you have the ESXi 6.0 U1 version, we recommend an ESXi upgrade. There is a known VMware issue where the node becomes unresponsive due to a PSOD and OS crash. See VMware KB article, VMware ESXi 6.0, Patch ESXi600-201608401-BG: Updates esx-base, vsanhealth, vsan VIBs (2145664).

VMware ESXi Version

Supported Upgrade Paths

6.0 U3

6.0 U2 (All Patch Levels)

6.0 U1b

6.0 U2 (All Patch Levels)

Yes

—

—

6.0 U1b

Yes

Yes

—

5.5 U3b (Only on HX240c)

Yes

Yes

Yes

Cisco HX Data Platform Versions Supported by Cisco UCS Manager

Cisco UCS Manager Version

Cisco HX Data Platform

2.1(1b)

2.0(1a)

1.8.x

1.7.x

3.1(2g)

Yes

Supports hybrid and All Flash

Yes

Supports hybrid and All Flash

—

—

3.1(2f)

Yes

Supports hybrid and All Flash

Yes

Supports hybrid and All Flash

—

—

3.1(2e)

—

Yes

Supports hybrid

—

—

3.1(2c)

—

Yes

Supports hybrid

Yes

—

3.1(2b)

—

Yes

Supports hybrid

Yes

—

2.2(7c)

—

—

—

Yes

2.2(6f)

—

—

—

Yes

Chapter 2. Upgrade Scenarios and Worflow

- Upgrade Scenarios

- Single Cluster Environment

- Multi Cluster Environment

- Addition of a New Cluster on the Same HX Domain

- Addition of a New HX Node to an Existing Cluster

Upgrade Scenarios

The following table lists the different upgrade scenarios:

Scenario

See

An existing single cluster

Single Cluster Environment

An existing multi-cluster

Multi Cluster Environment

Addition of a new cluster on the same HX cluster

Addition of a New Cluster on the Same HX Domain

Addition of a new HX node to the existing HX cluster

Addition of a New HX Node to an Existing Cluster

Considerations: - A new HX node may have a CPU that is different from the existing HX nodes.

Upgrades apply to entire HX clusters, fabric interconnects, and HX cluster components. Based on your requirement, you may require an upgrade of:

1. Cisco HX Data Platform only

2. Cisco UCS firmware only

3. Combined upgrade of Cisco UCS firmware and HX Data Platform

4. VMware ESXi

Single Cluster Environment

Before You Begin

- VMware ESXi is upgraded separately.

- You could do a split upgrade if there is a small maintenance window, by upgrading Cisco HX Data Platform first and then upgrading Cisco UCS firmware later.

Single Cluster Upgrade

Current State

Component

From

To

Single HX cluster - Upgrade all components

Cisco HX Data Platform

1.7.x

1.8.x

2.0(1a)

Cisco UCS firmware

2.2(6f)/2.2(7c)/3.1(2x)

3.1(2f)

VMware ESXi

5.5 U3b

Note:
This version is supported only on HX240c nodes.

6.0 U2 patch 4

Note:
Call Cisco TAC for this upgrade.

6.0 U1

No upgrade is required. However, Cisco recommends upgrade to the latest version of 6.0 U2 patch 4.

Multi Cluster Environment

Before You Begin

All the HyperFlex clusters must be upgraded after the first cluster upgrade, because in case of an unexpected power outage or a reboot, the rest of the clusters will be in an unsupported configuration. However, the upgrade can be spread across different maintenance windows.

Important:
If the current version of your Cisco HX Data Platform(HXDP) is at 1.8(1a) or 1.8(1b), upgrade only the HXDP component to 1.8(1e). Do not upgrade the Cisco UCS Firmware.

Multi Cluster Environment - Partial Upgrade

Action

From

To

Upgrade ONLY existing Cisco UCS infrastructure

2.2(6f)/2.2(7c)/3.1(2b)

3.1(2f)

Choose ONE cluster to do combined upgrade of HXDP and Cisco UCS firmware

HXDP 1.7.x

HXDP 1.8.x

2.0(1a)

2.2(6f)/2.2(7c)/3.1(2b)

3.1(2f)

ESXi

ESXi is upgraded separately.

5.5 U3b

Note:
Note: This version is supported only on HX240c nodes.

6.0 U2 patch 4

Call Cisco TAC for this upgrade.

6.0 U1

Option 1 - No upgrade required

Option 2 - Upgrade to the latest version of 6.0 U2 patch 4

Multi Cluster Environment - Full upgrade

For a full upgrade in a multi cluster environment, follow Offline Upgrade Process Workflow or Online Upgrade Process Workflow to upgrade all clusters.

Addition of a New Cluster on the Same HX Domain

Adding a new cluster

Options

Action

Component

Currently @

Upgrade To

Add a new 1.8 cluster to Existing cluster - Upgrade Cisco UCS firmware only

Step 1 - Upgrade only.

Cisco UCS firmware

2.2(6f)/2.2(7c)/3.1(2b)

3.1(2f)

Step 2 - Add the new HX cluster.

See Setting up Multiple Clusters chapter in Getting Started Guide

Note: When adding a new cluster, place it in a new Sub-org. This will ensure both clusters can be upgraded independently in the future.

Complete Upgrade

Add a new 1.8 cluster to Existing 1.7 - Upgrade existing cluster.

Step 1 - Upgrade the entire existing cluster.

HX Data Platform

1.7.1/1.7.3/1.8.x

2.0(1a)

Upgrade to the latest patch

Cisco UCS Firmware

2.2(6f)/2.2(7c)/3.1(2b)

3.1(2f)

Step 2 - Add the new HX cluster.

See Setting up Multiple Clusters chapter in Getting Started Guide

Note: When adding a new cluster, place it in a new Sub-org. This will ensure both clusters can be upgraded independently in the future.

Optional

ESXi

5.5 U3b

Note:
Note: This version is supported only on HX240c nodes.

6.0 U2 patch 4

Call Cisco TAC for this upgrade.

6.0 U1

No upgrade is required. However, Cisco recommends upgrade to the latest version of 6.0 U2 patch 4.

Addition of a New HX Node to an Existing Cluster

Adding a new HX Node

Options

Action

Component

Currently @

Upgrade To

Complete Upgrade

Add a new HX node to Existing cluster - Upgrade existing cluster.

Step 1 - Upgrade the entire existing cluster.

HX Data Platform

1.7.x/1.8.x

2.0(1a)

Cisco UCS Firmware

2.2(6f)/2.2(7c)/3.1(2.x)

3.1(2f)

Step 2 - Enable EVC. See Setting Up Mixed Cluster section in Getting Started Guide for detailed steps.

Step 3 - Add the HyperFlex node. See Expanding HyperFlex Clusters chapter in Getting Started Guide for detailed steps

Optional

ESXi

5.5 U3b

Note:
Note: This version is supported only on HX240c nodes.

6.0 U2 patch 4

Call Cisco TAC for this upgrade.

6.0 U1

Option 1 - No upgrade required

Option 2 - Upgrade to the latest version of 6.0 U2 patch 4

Chapter 3. Prerequisites and Guidelines

- Overview

- Prerequisites

- Cautions and Guidelines

- HyperFlex Node Upgrade Validations

- Viewing HyperFlex Cluster Health

- Verifying If DRS Is Enabled

- Viewing ESX Agent Manager

- Verify Health of HyperFlex Cluster In Cisco UCS Manager

- Configuring vMotion Interfaces

- Configure Lenient Mode

Overview

Before you upgrade the Cisco HX Data Platform and the Cisco UCS server firmware in your Cisco HyperFlex System, consider the guidelines, best practices, and recommendations listed in this chapter.

Prerequisites

1. See resolved and open caveats before upgrading. Review the new features, refer to the Cisco HX Data Platform Release Notes.

2. Review supported versions and system requirements. See Hardware and Software Interoperability for Cisco HyperFlex HX-Series for more details.

Important:

- Verify that you have the latest software bundle versions. Review the software versions, Cisco HX Data Platform Release Notes

- Ensure that the operating systems on all servers have the right driver levels for the release of Cisco UCS to which you plan to upgrade. See Cisco UCS Driver Installation Guide for identifying the server hardware.

3. Back up the configuration into an All Configuration backup file, refer Cisco UCS Manager Backing Up and Restoring the Configuration Guide for detailed steps.

4. Before you perform firmware updates, use the Cisco UCS Manager image management interfaces to download relevant images to the fabric interconnect. Images are stored in bootflash partition in the fabric interconnect. See Downloading Software for more details.

5. An ESXi upgrade is required from ESX 5.5 U3b and ESXi 6.0 U1. Only VMware vSphere versions ESXi 6.0 U1b, 6.0U2, and 6.0U2 patch 3 are supported. For ESXi upgrade, refer Upgrading ESXi.

6. Keep SSH enabled on all ESXi Hosts.

7. Enable vMotion so that the VMs can be moved automatically during the upgrade and MTUs are set as Jumbo. See Configuring vMotion Interfaces for details on adding VMkernel interface.

8. Verify the HyperFlex cluster is healthy. See HyperFlex Node Upgrade Validations for more details.

9. Verify that the cluster is in lenient mode. If not, set the cluster to lenient mode, refer Configure Lenient Mode.

Cautions and Guidelines

Before you begin upgrade of a Cisco HyperFlex System, consider the following cautions, guidelines, and limitations.

- Optimizations in Capacity Tier—Backend access is optimized to significantly reduce the magnitude and frequency of high latency spikes.

Important Upgrade Guidelines

- This upgrade is recommended for only those customers who have been identified having this problem.

- For hybrid clusters—The default upgrade process will not enable this optimization. Contact Cisco TAC to enable this performance enhancement during the upgrade process. Enabling this optimization will require a longer maintenance window.

- For All Flash clusters—The upgrade times will not be significantly affected and the default upgrade path will enable this performance enhancement.

- Upgrade VMware ESXi before starting the upgrade process.

Important:
If you have to upgrade from VMware ESXi version 5.5 U3, contact Cisco TAC for assistance.

- The Cisco HX Data Platform and Cisco UCS firmware bundles must be compatible. Refer UCS Hardware and Software Compatibility Matrix for more details.

- For a split upgrade, Cisco HX Data Platform must be updated first, before updating the Cisco UCS firmware to avoid no connection error message.

- During online upgrade, as one node is being upgraded (put into maintenance mode), the number of tolerated node failures is reduced based on the Data Replication Factor and Access Policy settings.

- All endpoints in a Cisco HyperFlex domain must be fully functional and all processes must be complete before you begin a firmware upgrade on those endpoints. For example, the firmware on a server that has not been discovered cannot be upgraded or downgraded. Each endpoint is a component in the Cisco HyperFlex domain that requires firmware to function.

- In a three node cluster, if you shut down one node or put into maintenance mode it makes the cluster unhealthy, but the cluster is still online. During the upgrade process, put the host in maintenance mode one at a time and move to the next host, after the cluster becomes healthy.

Note

You cannot remove a node from 3 node cluster by doing stcli node remove operation. To replace a node on a 3 node cluster, please contact Cisco TAC for assistance with the node replacement procedure.

- Firefox browser is not supported due to an outdated version of flash that is bundled with the browser. Manual update of flash within Firefox is possible, but the recommendation is to use either Chrome or Internet Explorer with a modern version of flash.

HyperFlex Node Upgrade Validations

Perform the following validations on each HyperFlex node before moving on to upgrade the next node in the cluster.

- Verify that the HyperFlex cluster is healthy and online. Verify all HyperFlex cluster nodes are connected to the vCenter and are online.

- SSH into each controller VM in the HX cluster and make sure that there is space in /var/stv for the upgrade using the command df -h /var/stv.

Output example
/dev/sdb1 63G 2.7G 58G 5% /var/stv

Important:
If there is not enough space (usage is over 50%), contact Cisco TAC for assistance.

- Verify that no major Alarms are reported for the HyperFlex cluster in the vSphere Web Client.

- Verify that DRS is enabled and set to fully automated.

- Verify that vSphere services are running and ESXi Agent Manager (EAM) health is normal.

- Verify the health of the cluster in Cisco UCS Manager.

- Viewing HyperFlex Cluster Health

- Verifying If DRS Is Enabled

- Viewing ESX Agent Manager

- Verify Health of HyperFlex Cluster In Cisco UCS Manager

Viewing HyperFlex Cluster Health

Using GUI

- From the vSphere Web Client Navigator, select vCenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > cluster > Summary. View the cluster widget to verify if the HyperFlex cluster is healthy and online.

- From the vSphere Web Client Navigator, select vCenter Inventory Lists > Clusters > cluster > Summary. Verify if all HX Cluster nodes are connected to the vCenter and they are online.

Using CLI

Log in to any controller VM in the storage cluster. Run the command stcli cluster storage-summary –-detail.

Sample response that indicates the HyperFlex storage cluster is online and healthy.

address: 192.168.100.82
name: HX-Cluster01
state: online
uptime: 0 days 12 hours 16 minutes 44 seconds
activeNodes: 5 of 5
compressionSavings: 78.1228617455
deduplicationSavings: 0.0
freeCapacity: 38.1T
healingInfo:
inProgress: False
resiliencyDetails:
current ensemble size:5
# of ssd failures before cluster shuts down:3
minimum cache copies remaining:3
minimum data copies available for some user data:3
minimum metadata copies available for cluster metadata:3
# of unavailable nodes:0
# of nodes failure tolerable for cluster to be available:2
health state reason:storage cluster is healthy.
# of node failures before cluster shuts down:3
# of node failures before cluster goes into readonly:3
# of hdd failures tolerable for cluster to be available:2
# of node failures before cluster goes to enospace warn trying to move the existing data:na
# of hdd failures before cluster shuts down:3
# of hdd failures before cluster goes into readonly:3
# of ssd failures before cluster goes into readonly:na
# of ssd failures tolerable for cluster to be available:2
resiliencyInfo:
messages:
Storage cluster is healthy.
state: healthy
hddFailuresTolerable: 2
nodeFailuresTolerable: 1
ssdFailuresTolerable: 2
spaceStatus: normal
totalCapacity: 38.5T
totalSavings: 78.1228617455
usedCapacity: 373.3G
clusterAccessPolicy: lenient
dataReplicationCompliance: compliant
dataReplicationFactor: 3

Verifying If DRS Is Enabled

Step 1

1. From the vSphere Web Client Navigator, select vCenter Inventory Lists > Clusters > cluster > Summary.

Verify that DRS is Enabled.

Step 2

Click the vSphere DRS tab.
Check if Migration Automation Level is set to Fully Automated.

Viewing ESX Agent Manager

From the vSphere Web Client Navigator, select Administration > vCenter Server Extensions > vSphere ESX Agent Manager > Summary.
Verify that vSphere services are running and ESX Agent Manager (EAM) health is normal.

Verify Health of HyperFlex Cluster In Cisco UCS Manager

Step 1

Verify if the high availability status of the fabric interconnects shows that both the fabric interconnects are up and running.

Step 2

Verify that the data path is up and running. See the Cisco UCS Manager Firmware Management Guide for more information.

Step 3

Verify that the HyperFlex servers have no faults.

Step 4

Verify that vNIC faults are cleared to ensure VMware ESXi vSwitch uplinks are up and operational.

Step 5

Verify if all servers have been discovered.

Configuring vMotion Interfaces

The following steps add the VMkernel interface necessary for vMotion to work:

Before You Begin

With the new release, vMotion networking is already been pre-defined with a vSwitch created, and the vNICs and VLANs are already defined in UCS Manager.

Step 1

Click on the Host in, Inventory > Manage > Networking > VMkernel adapters.

Step 2

Click Add Host Networking.

Step 3

Select VMkernel Network Adapter.

Step 4

Select the existing vmotion vSwitch by selecting browse.

Step 5

Provide a name, and refer to table below to enter the appropriate VLAN ID.

Cluster Installation Version

VLAN ID

1.7.x

0 (default)

1.8.x

same as vMotion network

Step 6

Provide a Static IP Address and complete the wizard.

Step 7

(Optional) To use jumbo frames, edit the vmk2 and set the MTU to 9000. Your upstream switch must be configured to pass jumbo frames on the vMotion VLAN.

Step 8

Repeat steps 1 to 5 for all hosts in the cluster.

Configure Lenient Mode

Cluster access policy is set by default to lenient mode. To manually set the cluster access policy to lenient, use the following procedure.

Command or Action

Purpose

Step 1

SSH to any one of the controller VMs and login as root.

Step 2

#stcli cluster get-cluster-access-policy

Check if lenient mode is already configured.

Step 3

~/#stcli cluster set-cluster-access-policy --name lenient

If set to strict, change to lenient.

If already set to lenient, no further action is required.

Step 4

stcli cluster info | grep -i policy

Confirm the change.

The following example checks how lenient mode is currently configured. If the lenient mode is set to strict, it sets the mode to lenient, and confirms the change made to lenient mode.

~/#stcli cluster get-cluster-access-policy
strict
~/#stcli cluster set-cluster-access-policy --name lenient
stcli cluster info | grep -i policy

Chapter 4. Preparing For Upgrade

- Downloading Software

- Test Upstream Network Connectivity

- Graceful Shutdown of a HX Cluster

- Modifying Host Firmware Package Using Cisco UCS Manager

- Bootstrap Process

- Entering Cisco HyperFlex Maintenance Mode

- Exiting Cisco HyperFlex Maintenance Mode

Downloading Software

For a successful HyperFlex upgrade, the HyperFlex System component bundles can be downloaded from Cisco website.

1. HX Data Platform upgrade bundle (.tgz file)

2. VMware ESXi

3. Cisco UCS Infra (A) bundle, B and C bundles, and Firmware

All the components can be downloaded from HyperFlex Download site.

After the Cisco UCS bundles and firmware have been downloaded, they need to be copied to Cisco UCS Manager

Copy UCS bundles to UCS Manager.

Step 1

Login to Cisco UCS Manager GUI.

Step 2

Type in the username and password.

Step 3

In the Navigation pane, click the Equipment tab.

Step 4

On the Equipment tab, click the Equipment node.

Step 5

In the Work pane, click the Firmware Management > Installed Firmware > Download Firmware.

Step 6

In the Download Firmware dialog box, click the Local File System radio button in the Location of the Image File field and fill in the required fields.

Step 7

In the Filename field, type the full path and name of the image file.
If you do not know the exact path to the folder where the firmware image file is located, click Browse and navigate to the file.

Step 8

Click OK. Cisco UCS Manager GUI begins downloading the firmware bundle to the fabric interconnect.

Step 9

Monitor the status of the download on the Download Tasks tab.
Note:
If Cisco UCS Manager reports that the bootflash is out of space, delete obsolete bundles on the Packages tab to free up space. To view the available space in bootflash, navigate to Equipment > Fabric Interconnect > Local Storage Information in the work pane area under the General tab.

Step 10

Repeat this task until all the required firmware and bundles have been downloaded to the fabric interconnect.
This includes infrastructure (A) bundle, blade bundle (B), and rack bundle (C).

Test Upstream Network Connectivity

Ensure that the hx-storage-data and vMotion upstream switches are configured for Jumbo Frames. Skipping this step could lead to input/output interruption during Cisco UCS infrastructure upgrade.

Step 1

Put the node in Cisco HX Maintenance mode, refer Entering Cisco HyperFlex Maintenance Mode.

Step 2

SSH to one of the ESX hosts.

Step 3

Verify that ping is working.
vmkping -I vmk1 -d -s 8972 <data IP of address of another host>

Step 4

Swap active interfaces in vswitch-hx-storage-data to force traffic upstream.
esxcli network vswitch standard policy failover set -a vmnic2 -s vminc3 -v vswitch-hx-storage-data

Step 5

Verify that ping is working.
vmkping -I vmk1 -d -s 8972 <data IP of address of another host>

If the ping fails, try again with:

vmkping -I vmk1 -d -s 1472 <data IP of address of another host>

Attention:
If the ping fails, do not proceed to upgrade the Cisco UCS firmware. Please contact Cisco TAC for assistance.

Step 6

Swap interface back to defaults even if ping fails.
esxcli network vswitch standard policy failover set -a vmnic3 -s vminc2 -v vswitch-hx-storage-data

Step 7

Exit the node from Cisco HX Maintenance mode, refer Exiting Cisco HyperFlex Maintenance Mode.

Graceful Shutdown of a HX Cluster

Step 1

SSH to any controller VM in the cluster.

Step 2

Check cluster health ~#stcli cluster info | grep health.

Step 3

If healthy, shutdown the cluster ~#stcli cluster shutdown.

Step 4

Shutdown takes a few minutes, wait for the prompt to return.

Modifying Host Firmware Package Using Cisco UCS Manager

Host Firmware Package is set automatically, during Cisco UCS infrastructure upgrade. To manually stage the correct firmware version before starting the upgrade process use the following procedure.

Step 1

Sign in to Cisco UCS Manager.

Step 2

In the Navigation pane, click Servers.

Step 3

Expand Servers > Policies > Sub-Organizations > hx-cluster.

Step 4

Expand Host Firmware Packages and choose the policy you want to update.

Step 5

In the Work pane, click the General tab.

Step 6

To modify the components in the host firmware package, click Modify Package Versions.

Step 7

Modify value for Blade Package and Rack Package, to the latest firmware version. See HyperFlex Software Versions for the list of recommended UCS FI firmware.

Step 8

In the Excluded Components area, check the boxes corresponding to the components that you want to exclude from this host firmware package.

Step 9

Click OK. Click yes for all the warnings.

What to Do Next
Verify pending activities.
User acknowledgement of each server is performed automatically during upgrade. Optionally, you can manually acknowledge pending activities on the HyperFlex node.

On the Cisco UCS Manager toolbar, click Pending Activities.

The User Acknowledged Activities tab lists the HyperFlex nodes that require user acknowledgment within the selected sub-org. They are in Pending Reboot status.

Note

Do not acknowledge service profiles.

Bootstrap Process

Bootstrap process enables you to upgrade the Cisco HX Data Platform Plug-in.

Note

Perform this procedure on the node that has the Cluster Management IP address.

Step 1

From the vSphere Web Client Navigator, select vCenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > cluster.

Step 2

Navigate to Actions > Summary and note the Cluster Management IP address.

Step 3

SSH to the cluster management IP address with root privileges.

Step 4

Transfer the latest HX Data Platform upgrade bundle to this controller VM.
Depending on your operating system, use you can either use SCP directly or download third-party tools, such as WinSCP or MobaXterm.

Step 5

From the controller VM shell, change to the /tmp directory.
Warning:
Do not use any folder other than /tmp and do not create any subfolders.

Step 6

Un-compress the package using tar –zxvf <storfs package name>.tgz.

Example:

tar –zxvf storfs-packages-2.0.1a-20704.tgz

This un-compresses and extracts all files to the root of the /tmp folder.

Step 7

Invoke the cluster-bootstrap.sh script to bootstrap packages for upgrade. Execute the command

:~# ./cluster-bootstrap.sh --vc-url <vCenter IP or FQDN> --vcenter-user administrator@vsphere.local

Enter the vCenter FQDN or IP address and administrator level username and password.

Example:

root@ucs-stctlvm-101-1:/tmp# ./cluster-bootstrap.sh
Preparing...
Enter vCenter Host: 1.1.1.1
Enter vCenter User: admin
Enter vCenter password:2222
Cluster upgrade bootstrap is required. Updating now...
Installing HyperFlex Software Package
--------------------------------------
--------------------------------------
BUILD_ID:19694
Release: 1.8.1c
Tag:
Date: 20161023
Branch: 1.8.1
Install Date: Mon Oct 24 18:52:26 UTC 2016
------------------------------------------
Applying changes to HX controller
Cron stop/waiting
Cron start/running, process 23868
HX controller changes done
Migrating instll directory to front SSD
Installing python module simplejson
Installing python module vcversioner
Installing python module functools32...

wait for system management service to restart...
[1]Waiting for services to start...

Updating vCenter plugin ...
Updating plugin version for Springpath Storage Manager from 1.8.1.19547 (GA build)
to 1.8.1.19694
Updated plugin version to 1.8.1.19694
Bootstrap completed.

Wait for the system management service to restart and the bootstrap process to complete. Verify if the HX Data Platform Plug-in is now updated.

Step 8

Log out from the cluster management IP controller VM. Make sure to log out of vSphere Web Client.
Note:
Do not merely close the browser.

Step 9

Log in to vSphere Web Client again to refresh the HX Data Platform Plug-in.

Entering Cisco HyperFlex Maintenance Mode

Using the vSphere Web Client

1. Log in to the vSphere web client.

2. Go to Home > Hosts and Clusters.

3. Expand the Datacenter that contains the HX Cluster.

4. Expand the HX Cluster and select the node.

5. Right-click the node and select Cisco HX Maintenance Mode > Enter HX Maintenance Mode.

Using the Command-Line Interface

1. Log in to the storage controller cluster command line as a user with root privileges.

2. Move the node into HX Maintenance Mode.

1. Identify the node ID and IP address.

# stcli node list --summary

2. Enter the node into HX Maintenance Mode.

# stcli node maintenanceMode (--id ID | --ip IP Address) --mode enter

(see alsostcli node maintenanceMode --help)

3. Log in to the ESXi command line of this node as a user with root privileges.

4. Verify that the node has entered HX Maintenance Mode.

# esxcli system maintenanceMode get

You can monitor the progress of the Enter Maintenance Mode task in vSphere Web Client, under the Monitor > Tasks tab.

If the operation fails, an error message is displayed. Try to fix the underlying problem and attempt to enter the maintenance mode again. If you are unable to resolve the problem, please contact Cisco TAC.

Exiting Cisco HyperFlex Maintenance Mode

Using the vSphere Web Client

1. Log in to the vSphere web client.

2. Go to Home > Hosts and Clusters.

3. Expand the Datacenter that contains the HX Cluster.

4. Expand the HX Cluster and select the node.

5. Right-click the node and select Cisco HX Maintenance Mode > Exit HX Maintenance Mode.

Using the Command-Line Interface

1. Log in to the storage controller cluster command line as a user with root privileges.

2. Exit the node out of HX Maintenance Mode.

1. Identify the node ID and IP address.

# stcli node list --summary

2. Exit the node out of HX Maintenance Mode.

# stcli node maintenanceMode (--id ID | --ip IP Address) --mode exit

(see alsostcli node maintenanceMode --help)

3. Log in to the ESXi command line of this node as a user with root privileges.

4. Verify that the node has exited HX Maintenance Mode.

# esxcli system maintenanceMode get

You can monitor the progress of the Exit Maintenance Mode task in vSphere Web Client, under the Monitor > Tasks tab.

If the operation fails, an error message is displayed. Try to fix the underlying problem and attempt to enter the maintenance mode again. If you are unable to resolve the problem, please contact Cisco TAC.

Chapter 5. Upgrade Procedures

- Online Upgrade Process Workflow

- Offline Upgrade Process Workflow

- Start Cluster and Power On VMs

Online Upgrade Process Workflow

First upgrade Cisco UCS infrastructure to the latest version and then use the automated upgrade workflow for a combined upgrade of Cisco UCS firmware and Cisco HX Data Platform. Online upgrade uses host firmware packages to upgrade all server endpoints.

During online upgrade, as one node is being upgraded (placed into maintenance mode), the number of tolerated node failures is reduced based on the Data Replication Factor and Access Policy settings.

Prior to upgrading to HXDP 2.1(1b), ensure that you upgrade UCS server firmware to 3.1(2f) or 3.1(2g). Either a combined or split upgrade can be selected through the vSphere Web Client plugin. Direct firmware upgrade through Cisco UCS Manager should not be attempted.

Caution

Do not use Firefox browser. It is not supported due to an outdated version of flash that is bundled with the browser.

1. Download Cisco UCS Infrastructure Software A, B, and C bundles. See Downloading Software.

2. Ensure that the hx-storage-data and vMotion upstream switches are configured for Jumbo Frames before proceeding forward. Otherwise the HyperFlex Cluster becomes offline and all datastores unmount from the ESXi hosts, refer Test Upstream Network Connectivity.

3. Upgrade Cisco UCS Infrastructure. See Upgrading Cisco UCS Firmware Using Cisco UCS Manager.

4. Bootstrap to upgrade Cisco HX Data Platform Plug-in. See Bootstrap Process.

Important:
Be sure to copy the bootstrap file to the controller VM /tmp directory.

5. To disable snapshot schedule, on the bootstrapped storage controller VM, run the command stcli snapshot-schedule --disable.

It is enough to run this script on one of the controller nodes.

6. Log in to the HX Data Platform Plug-in with administrator credentials.

7. Start combined upgrade of HX Data Platform and Cisco UCS firmware. See Combined Upgrade of Cisco HX Data Platform and Cisco UCS Firmware.

To upgrade HX Data Platform only or Cisco UCS firmware only. See Upgrading Cisco HX Data Platform Software and Upgrading Cisco UCS Firmware Using vSphere Web Client for more details.

Note

When only Cisco UCS firmware is being upgraded, you may see the process pause at the validation screen, after the fabric interconnect discovery. Most likely this occurs due to network connectivity failure. Please contact Cisco TAC for help.

8. Confirm that upgrade is complete. See Post Upgrade Tasks.

9. On the same controller VM, to enable snapshot schedule, run the command stcli snapshot-schedule -–enable.

Offline Upgrade Process Workflow

1. Download Cisco UCS Infrastructure Software A, B, and C bundles. See Downloading Software.

2. Ensure that the hx-storage-data and vMotion upstream switches are configured for Jumbo Frames before proceeding forward. Otherwise the HyperFlex Cluster becomes offline and all datastores unmount from the ESXi hosts, refer Test Upstream Network Connectivity.

3. Upgrade Cisco UCS Infrastructure. See Upgrading Cisco UCS Firmware Using Cisco UCS Manager.

4. Launch the vSphere Web Client and power down all VMs. After the VMs have been shut down, verify the health state of the cluster and perform a graceful shutdown, refer Graceful Shutdown of a HX Cluster.

Important:
HyperFlex controller VMs (stCtlVMs) must remain powered on.

5. Host Firmware Package is set automatically, during Cisco UCS infrastructure upgrade. To manually stage the correct firmware version before starting the upgrade process, refer Modifying Host Firmware Package Using Cisco UCS Manager.

6. Acknowledge the pending reboot on the servers that comprise your HX cluster nodes. Include both converged nodes and compute-only nodes connected to the cluster. Wait until all nodes are upgraded. Confirm that correct firmware packages have been installed before proceeding.

Important:

- If you are upgrading from Cisco UCS firmware version 2.2(7c):

Acknowledge the pending reboot on the HX nodes that you are upgrading. Wait until all nodes are upgraded to the new version before continuing further. This is applicable only to an Offline upgrade.

- If you are upgrading from UCS Firmware version 2.2(6f):

Reacknowledge the pending reboot on the HX nodes to do a deep discovery.

In Cisco UCS Manager, navigate to Equipment > Rack Mount > Servers > the server in 2.2.6 > right click Server Maintenance > re-acknowledge.

7. Bootstrap to upgrade the Cisco HX Data Platform Plug-in. See Bootstrap Process.

Important:
Be sure to copy the bootstrap file to the controller VM /tmp directory.

8. To disable snapshot schedule, on the bootstrapped storage controller VM, run the command stcli snapshot-schedule --disable.

It is enough to run this script on one of the controller nodes.

9. From the same controller VM, begin the upgrade. See Combined Upgrade - Offline Mode or Split Upgrade - Offline Mode for more details.

Attention:
Combined upgrade is highly recommended to have the most efficient offline upgrade experience.

10. Confirm that upgrade is complete. See Post Upgrade Tasks.

11. After the upgrade is complete, start the cluster and power on VMs. See Start Cluster and Power On VMs.

12. On the same controller VM, to enable snapshot schedule, run the command stcli snapshot-schedule -–enable.

- Offline Upgrade Guidelines

- Combined Upgrade - Offline Mode

- Split Upgrade - Offline Mode

Offline Upgrade Guidelines

Important:

- --ucsm-host and --ucsm-user parameters are required when you are upgrading from 1.7x to 1.8x. These parameters must not be used when moving up from 1.8(1a)/1.8(1b) to 2.0(1a) as we are not changing the Cisco UCS server firmware version.

Cisco recommends using GUI upgrade over CLI for ease of use and better reporting.

Before you proceed, with either combined or split upgrade consider the following guidelines:

- The package name must match the file you uploaded to the controller VM and it may differ from the examples shown.

- Enter passwords when prompted.

- Nodes are upgraded with the new version of the Cisco HX Data Platform software and rebooted one at a time.

Combined Upgrade - Offline Mode

Combined Offline Upgrade Using GUI

Start combined upgrade of HX Data Platform and Cisco UCS firmware. See Combined Upgrade of Cisco HX Data Platform and Cisco UCS Firmware.

Combined Offline Upgrade Using CLI

From the same controller VM, use the command:

# stcli cluster upgrade --components hxdp,ucs-fw --location/tmp/
<storfs package name> --vcenter-user <vcuser> --ucsm-host <IP/FQDN of UCSM>
--ucsm-user <UCSM User> --ucsfw-version <UCSM Firmware Version>

Example:

root@ucs-stctlvm-357-1:~# stcli cluster upgrade --components hxdp,ucs-fw --location
/tmp/storfs-packages-1.8.1c-19712.tgz --vcenter-user administrator@vsphere.local
--ucsm-host eng-fi16.eng.storvisor.com --ucsm-user admin --ucsfw-version '3.1(2b)'

Split Upgrade - Offline Mode

Upgrade Cisco UCS Firmware Only Using GUI

Start upgrade of Cisco UCS firmware only. See Upgrading Cisco UCS Firmware Using vSphere Web Client.

Upgrade Cisco HX Data Platform Only Using GUI

Start upgrade of Cisco HX Data Platform only. See Upgrading Cisco HX Data Platform Software.

Upgrade Cisco UCS Firmware Only Using CLI

From the same controller VM, use the command:

stcli cluster upgrade --components ucs-fw --ucsm-host eng-fi1
--ucsm-user <UCSM User> --ucsm-pwd <UCSM Password> --ucsfw-version <UCSM Firmware Version>

Example:

stcli cluster upgrade --components ucs-fw --ucsm-host eng-fi1
--ucsm-user admin --ucsm-pwd admin --ucsfw-version '3.1(2b)'

Upgrade Cisco HX Data Platform Only Using CLI

stcli cluster upgrade --components hxdp --location
/tmp/<storfs package name> --vcenter-user <vcuser>

Example:

stcli cluster upgrade --components hxdp --location
/tmp/storfs-packages-1.8.1c-19694.tgz --vcenter-user administrator@vsphere.local

Start Cluster and Power On VMs

After the upgrade is complete and the cluster has been upgraded, log out and log back in to vCenter to see upgrade changes.

Step 1

After the upgrade is complete, start your cluster.

Step 2

Login to any controller VM through SSH: .
# stcli cluster start

Example:

HyperFlex StorageController 1.8(1c)
Last login: Wed Sept 21 23:54:23 2016 from pguo-dev.eng.storvisor.com
root@ucs-stclivm - 384 -1;~# stcli cluster upgrade-status
Cluster upgrade succeeded. Cluster version: 1.8(1c)
root@ucs-stctlvm-384;~# stcli cluster start
waiting for Cluster to start on nodes: [ucs-383, ucs-384, ucs-385, ucs-386]

This will start the cluster and mount the HX datastores. Wait for cluster to come online. You will see the prompt:

Started cluster on nodes; [ucs-383, ucs-384, ucs-385, ucs-386]
Cluster is online
root@ucs-stctlvm-384-1;~#

Step 3

Wait for cluster to become healthy before starting the VMs. Run command:
~# stcli clustr info|grep health

Example:

root@SpringpathControllerZRVF040451;~# stcli cluster info | grep health
healthState: healthy
state: healthy
storage cluster is healthy

Step 4

After the cluster is healthy, launch vSphere Web Client or Thick Client, navigate to Hosts and Cluster > Datacenter > Cluster > . Right click, select Power> Power On to start the VMs.

Chapter 6. Upgrading Cisco HyperFlex Software Components

- Upgrading Cisco UCS Firmware Using Cisco UCS Manager

- Upgrading Cisco HX Data Platform Software

- Upgrading Cisco UCS Firmware Using vSphere Web Client

- Combined Upgrade of Cisco HX Data Platform and Cisco UCS Firmware

- Upgrading ESXi

- Upgrade of Compute-only Nodes

- Upgrading Cisco UCS Firmware on Compute-only Node from version 2.2(6f) to 3.1(2b)

Upgrading Cisco UCS Firmware Using Cisco UCS Manager

The UCS infrastructure firmware includes UCS Manager, switch firmware, blade chassis IO module firmware and rack server FEX firmware. The UCS infrastructure firmware can be upgraded with no disruption to the HyperFlex workloads. This is achieved by doing a rolling upgrade of the UCS fabric interconnects. The HX nodes are configured to failover all Ethernet traffic when a UCS fabric interconnect is rebooted to upgrade the firmware.

Note

Ensure that the hx-storage-data and vMotion upstream switches are configured for Jumbo Frames before proceeding forward, otherwise the HyperFlex Cluster will become offline and all datastore will unmount from the ESXi hosts.

Step 1

Open the UCS Manager GUI.

Step 2

Select Equipment > Firmware Management > Firmware auto-install.

Step 3

Click Install Infrastructure Firmware.

Step 4

Select the desired UCS infrastructure version. Refer Supported Upgrade Paths to identify the version desired for your use case. Click Next.
A pre-requisite dialog box will pop-up. You will see a list of warnings. Fix them first before proceeding further.

1. Cisco UCS Manager upgrade validation fails.

In that case, stop the upgrade process. Determine the reason for upgrade validation failure. Take corrective action and restart the upgrade process.

2. Cisco UCS Manager GUI is disconnected.

This is expected during UCS Manager upgrade as UCS Manager is stopped, then restarted at the new version. Wait until UCS Manager goes back online. Log back in to UCS Manager to complete the next steps.

You may check Ignore All box if the warnings are not critical to user environment.

Step 5

Check Upgrade Now box.

Step 6

Click Finish.

Step 7

Wait for IOM to be upgraded (if blade chassis are present)

1. Select Equipment > Installed Firmware, expand each chassis and check the Update Status of the IO Module.

2. During upgrade, the Update Status of the IO Modules will be Upgrading.

3. When IOM upgrade has completed, the Update Status of the IO Modules is set to Ready.

Step 8

Wait for Subordinate FI to be activated.

1. Select Equipment > Installed Firmware > Fabric Interconnects.

2. Check the Activate Status of the kernel and switch images. During upgrade, the Activate Status is set to Activating.

Step 9

During FI reboot, all HX traffic will be forwarded to primary FI (based on ESXi vSwitch failover policy). This will cause a brief traffic interruption. This will not cause storage IO failures.

Step 10

Verify subordinate FI has rebooted and joined the UCS cluster.

1. Select Equipment > Installed Firmware > Fabric Interconnects.

2. After activation, the Activate Status of the FI is set to Ready.

3. Check the Overall Status of the FI is operable.

4. Check the kernel and switch versions of the FI match the desired and updated version.

5. Check the FI has no fault.

6. Check the FI cluster membership is Subordinate.

Step 11

Wait for IOM activation to complete.

1. Select Equipment > Blade Chassis > IO Module.

2. Wait for the Activate Status of IO module to change to Ready.

Step 12

Wait until HX traffic is re-pinned to both FIs.
Wait for UCS Manager vNIC faults to be cleared. The fault clearing indicates ESXi has loaded the ENIC driver and the interface is up. The traffic is not re-pinned immediately when the network interface goes up because ESXi has a fail back timer. But the Net.teampolicyupdelay timer is very low by default (100ms).

Step 13

Verify the HX Cluster is online, and healthy before rebooting the primary fabric interconnect.

1. Access summary tab from the vSphere Web Client Navigator. Select Home > vCenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > cluster > Summary .

Step 14

In the UCS manager GUI, on the toolbar, click Pending Activities. Click on Fabric Interconnects tab that display the tasks requiring user acknowledgment before they can complete.

1. Check the box in the Reboot Now for each pending activity that you want to deploy immediately.

2. Click OK. Cisco UCS Manager immediately reboots the primary FI. This will cause the subordinate FI to become primary (FI failover).

Step 15

During FI reboot, all HX traffic will be forwarded to the new primary FI. This will cause a brief traffic interruption. However, this will not cause storage IO failures.

Step 16

Wait for UCS Manager to be disconnected, then reconnected on the other FI. This is because a UCS Manager fail over occurs due to primary FI reboot.
Check subordinate FI has become primary.

Step 17

Check subordinate FI has become primary.
Check that the FI cluster membership is Primary.

Step 18

Wait for FI to be activated.

1. Select Equipment > Installed Firmware > Fabric Interconnects.

2. Wait for the Activate Status of the FI to be Ready.

3. Check the Overall Status of FI is operable.

4. Check the FI has no fault.

Step 19

Verify FI has rebooted and joined the UCS cluster as subordinate.
Check that the FI cluster membership is Subordinate.

Step 20

Wait for IOM activation to complete

1. Select Equipment > Blade Chassis > IO Module.

2. Wait for the Activate Status of the IP module to be Ready.

3. You can monitor the status on the FSM tab.
Note:
You will lose connectivity to UCS Manager throughout the entire upgrade. This is a normal behavior.

Step 21

Wait until HX traffic is re-pinned to both FIs.
In the UCS manager GUI, wait until all server vNIC faults have been cleared.

Step 22

Verify the HX Cluster is online, and healthy after rebooting the FI.
Access summary tab from the vSphere Web Client Navigator. Select Home > vCenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > cluster > Summary .

Upgrading Cisco HX Data Platform Software

The HX Data Platform software can be upgraded with no disruption to the HyperFlex workloads. This is achieved by performing a rolling upgrade of each node in the HX cluster.

Step 1

Complete pre-upgrade validation checks.

Step 2

Download HX Upgrade package. See Download section.

Step 3

Bootstrap to upgrade HX Data Platform Plug-in. See Bootstrap Process

Step 4

Next, login HX Data Platform Plug-in with administrator credentials.

Step 5

Start upgrade of HX Data Platform.

1. Launch the Upgrade UI from vSphere web client by navigating to Home > Inventory > Host and Clusters > HX cluster. Right click, scroll down to HX Data Platform, and select Upgrade.
It is recommended to not use the cluster Summary tab for starting the upgrade due to issues with web client flash.

2. Select only HX Data Platform from the pop-up window. Click Next.

3. You will be prompted to load the file for starting the upgrade of HX Data platform. Navigate to the file on your local PC and make sure to select the correct version of the .tgz package file.

4. Enter administrator level vCenter credentials.

5. There is an advanced option to enter the Checksum # information. This is an optional step that helps you verify the integrity of the uploaded upgrade package bundle.
The number can be found on the CCO download page by hovering over the download file.

6. Click Upgrade to start the rolling upgrade.

Step 6

Complete post-upgrade validations.

Upgrading Cisco UCS Firmware Using vSphere Web Client

The new release allows for an independent upgrade of Cisco UCS Firmware. This procedure is applicable for upgrading UCS Server firmware from 2.2(6f)/2.2(7c) to 3.1(2b).

The upgrade will not cause any disruption to the workloads running on the HX Cluster.

Considerations

- Make sure that the HX Data Platform software has been upgraded first.

- In case of a split upgrade, UCS Firmware should be upgraded around the same time as the HX Data Platform upgrade itself.

Prerequisites

Before starting make sure the following steps have been completed.

1. Complete pre-upgrade validation checks.

2. Download HX Upgrade package. See Download section.

3. Upgrade the UCS Infrastructure.

4. Bootstrap to upgrade HX Data Platform Plug-in.

5. Upgrade HX Data Platform.

6. Next, login HX Data Platform Plug-in with administrator credentials.

Step 1

Launch the Upgrade UI from vSphere web client by navigating to Home > Inventory > Host and Clusters > HX cluster. Right click, scroll down to HX Data Platform, and select Upgrade.
Note: It is recommended to not use the cluster Summary tab for starting the upgrade due to issues with web client flash.

Step 2

Select only UCS Firmware the pop-up window. Click Next.

Step 3

Enter administrator level UCS Manager credentials.

Field

Data

UCS Manager Host Name

For example: eng-fi12.eng.storvisor.com

UCS Manager User Name

UCS Manager Password

Discover

Lists the current version

Target Version

Enter the latest UCS Firmware version. For example: 3.1(2b).

Note - the version number has to be exact otherwise the process will fail and the firmware will not upgrade.

Step 4

Click Upgrade to start the rolling upgrade.
This process installs the UCS firmware B and C bundles only as part of the upgrade. It acknowledges the Pending Activities of the host by doing the reboot and completes the upgrade.

The Validation screen will show the progress of checks performed.

- UCS Manager connectivity.

- Check that the target UCS server firmware version is valid.

- Shows Upgrade progress.

Note: This can also be monitored from the Cisco UCS Manager FSM tab.

Combined Upgrade of Cisco HX Data Platform and Cisco UCS Firmware

Combined upgrade will not cause any disruption to the workloads running on the HX Cluster.

The automated combined upgrade of Cisco HX Data Platform and Cisco UCS Server firmware achieves the following:

- Checks Cluster readiness.

- Puts the HX node in maintenance mode, one at a time.

- If DRS is enabled, make sure that it is set to automatic mode before starting the upgrade. This moves the VMs automatically to other hosts using vMotion.

Note

If the DRS functionality is not available, the user must manually vMotion the VMs to continue the process. For more information, see VMware Documentation for Migration with vMotion.

- Upgrades HX VIBs on Hypervisor.

- HX Data Platform will be upgraded followed by a reboot of the host server.

- HX will request UCS Manager to begin firmware upgrade. This process can take up to one hour.

Note: You can monitor the progress in UCS Manager GUI, under FSM tab for the service profile.

- HX Node will exit maintenance mode.

- Cluster will take a few minutes to rebuild back to full health.

- Upgrade will then move on to the next node in the cluster.

Prerequisites

Before starting make sure the following steps have been completed.

1. Complete pre-upgrade validation checks.

2. Download HX Upgrade package.

3. Upgrade the UCS Infrastructure.

4. Bootstrap to upgrade HX Data Platform Plug-in.

5. Next, login HX Data Platform Plug-in with administrator credentials.

Step 1

Launch the Upgrade UI from vSphere web client by navigating to Home > Inventory > Host and Clusters > HX cluster. Right click, scroll down to HX Data Platform, and select Upgrade.
Note: It is recommended to not use the cluster Summary tab for starting the upgrade due to issues with web client flash.

Step 2

You will be prompted to load the file for starting the upgrade of HX Data platform. Navigate to the file on your local PC and make sure to select the correct version of the .tgz package file.
This is the same file that was used in the prior step for updating the HX Plug-in. The wizard shows that the file was uploaded successfully.

Step 3

Enter administrator level vCenter credentials.

Step 4

Enter the Checksum # information under Advanced Options. This is an optional step that helps you verify the integrity of the uploaded upgrade package bundle.
The number is stored in a separate text file that can be found at the same /tmp directory where the upgrade package was downloaded.

Step 5

Enter administrator level UCS Manager credentials.

Step 6

Click Discover

Step 7

Type in the exact Target version of the UCS Firmware.

Step 8

Click Upgrade to start the rolling upgrade.
This installs the UCS firmware B and C bundles only as part of the upgrade. And it acknowledges the previous Pending Activities of the host by doing the reboot and completes the upgrade.

The Validation screen will show the progress of checks performed.

- Get release metadata from package.

- Check if vMotion is enabled.

- Check if all nodes are online and connected to vCenter.

- Check cluster connectivity.

- Check cluster state.

Upgrading ESXi

Caution

Using VMware Update Manager (VUM) to upgrade ESXi is discouraged. If you are using VUM to upgrade ESXi, use VUM one host at a time. Make sure that the cluster is in healthy state before moving on to the next node. Do not use VUM to upgrade ESXi across a cluster, as there is no guarantee that the cluster will be healthy by the time VUM moves on to the next node.

The ESXi hypervisor version can be upgraded with no disruption to the HyperFlex cluster workload. This is achieved by performing an online rolling upgrade of each node in the HX cluster.

Important:

- ESXi upgrade requires a manual online upgrade.

- When upgrading VMware ESXi from 5.5 U3b through any version up to 6.0 U2, please contact Cisco TAC.

- Use the ESXi command line interface esxcli for upgrade or updates.

- Replace the build numbers provided in the examples with the latest version.

Before You Begin

- Complete pre-upgrade validation checks. See upgrade prerequisites in Chapter 3.

- Ensure that you upgrade vCenter to a compatible version before beginning ESXi upgrades on the hosts.

Step 1

Download ESXi upgrade package. When upgrading ESXi from 6.0 Ux to any versions up to 6.0U2, use the offline zip file from the Cisco Website.
Example filename: HX-Vmware-ESXi-60U2-4192238-Cisco-Custom-Bundle-6.0.2.3.zip

Attention:
Do not use the HX ISO file or any other VMware ISO to attempt an ESXi upgrade.

Step 2

Select one of the hosts and put it in HX maintenance mode using the vSphere Web Client, refer Entering Cisco HyperFlex Maintenance Mode. After the host enters maintenance mode, complete the following steps.

Step 3

Remote secure copy the ESXi upgrade bundle to an appropriate folder with sufficient space.
To copy files using SCP, start the SSH service in the destination ESXi hosts as well.

Note:

- On HX240, you can use the local SpringpathDS datastore or a mounted HX datastore.

- On HX220, you can use either a mounted HX datastore or create a temporary RAM disk.

scp local_filename user@server:/path/where/file/should/go

Step 4

Log in to ESXi, and execute the command to query the list of available image profiles. For profile name verification, use the command:
esxcli software sources profile list -d <location_of_the_esxi_zip_bundle_on_the_datastore>

Example:

esxcli software sources profile list -d /patch/
HX-Vmware-ESXi-60U2-4192238-Cisco-Custom-Bundle-6.0.2.3.zip

Name                                    Vendor       Acceptance Level
----------------------------------------       ---------    -------------------
Vmware-ESXi-60U2-4192238-Cisco-Custom-HX         Cisco       PartnerSupported

Step 5

Run the following command to perform the upgrade.

esxcli software profile update -d <path_to_profile_ZIP_file> -p < profile name>

Example:

esxcli software profile update -d <path_to_profile_ZIP_file>/
HX-Vmware-ESXi-60U2-4192238-Cisco-Custom-Bundle-6.0.2.3.zip -p
Vmware-ESXi-60U2-4192238-Cisco-Custom-HX

Step 6

Once the upgrade completes, reboot the ESXi host.

Step 7

After the ESXi host comes up, verify that the host has booted up with the correct profile or path.

vmware -vl

Step 8

Exit maintenance mode using the vSphere Web Client, refer Exiting Cisco HyperFlex Maintenance Mode.

Step 9

Ensure that the cluster becomes healthy between each ESXi upgrade using stcli cluster storage-summary --detail.

Step 10

Repeat this process for all hosts in the cluster in a sequence.
Note:
Make sure that the cluster becomes healthy between each ESXi upgrade, refer HX Upgrade Node Validations.

Upgrade of Compute-only Nodes

The upgrade deployment can be done either in online of offline mode. For a successful upgrade, you must understand the upgrade options and the sequence of tasks that impact the upgrade process.

Compute-only Node Upgrade from version 2.2(7c) to 3.1(2b)

If you are upgrading your compute-only node from 2.2(7c)/1.7x to 3.1(2b)/1.8(1c), follow the non-disruptive online rolling upgrade process as described in the Online Upgrade Process Workflow.

The new HX Data Platform Upgrade workflow will automatically upgrade both compute-only node firmware and HX Data Platform components as part of the full upgrade.

For offline upgrade, refer Offline Upgrade Process Workflow.

Upgrading Cisco UCS Firmware on Compute-only Node from version 2.2(6f) to 3.1(2b)

If you are upgrading your compute-only node from 2.2(6f)/1.7x to 3.1(2b)/1.8(1c), the new HX Data Platform Upgrade workflow will automatically upgrade the HX Data Platform components.

However, after the HX Data Platform upgrade completes, you need to manually complete the upgrade of the compute-only node server firmware.

Step 1

Ensure Distributed Resource Scheduler (DRS) is enabled and set to automatic mode.
This allows the HX node to be vacated automatically. Otherwise, use vMotion to manually migrate all VMs off the host. Migration with vMotion allows you to move a virtual machine to a new host without any interruption in the availability of the virtual machine. For detailed steps, see VMware KB article.

Step 2

Place each compute-only node into maintenance mode.

Step 3

In the UCS manager GUI, Servers > Service Profiles, click Pending Activities to display the tasks requiring user acknowledgment before they can complete. Manually Acknowledge the pending reboot for each of the compute-only server. This will trigger a reboot and subsequent installation of new version of UCS firmware.
PLEASE CHECK THIS STEP

Step 4

Wait for the compute-only node to upgrade and fully reboot. After the compute-only node is up, verify the correct UCS firmware is present in UCS Manager. See Check Firmware Versions in UCSM.

Step 5

Repeat this process for all compute-only nodes in the cluster.

Chapter 7. Post Upgrade Tasks

- Confirm Upgrade is Complete

- Check Firmware Versions in UCSM

- Upgrading VMware Tools in Storage Controller VMs

Confirm Upgrade is Complete

After the upgrade is complete and the cluster has been upgraded, log out and log back in to vCenter to see upgrade changes.

Step 1

Login to Cisco UCS Manager to ensure that the HX nodes have no pending server activities.
From Server > Pending Activities tab check for all server activities.

Step 2

Confirm that HX nodes match the expected firmware versions. For details refer to the Check Firmware Versions in UCSM.

Step 3

Login to any controller VM through SSH.
# ssh root@controller_vm_ip

Step 4

Confirm product version.
# stcli cluster version

Cluster version: 2.0(1a)
Node HX-03 version: 2.0(1a)
Node HX-04 version: 2.0(1a)
Node HX-01 version: 2.0(1a)
Node HX-02 version: 2.0(1a)

Step 5

Verify the HX storage cluster is online, healthy, and displaying correct storage space.
# stcli cluster info|less

Step 6

Verify the datastores are up.

- From HX controller VMs
# stcli datastore list

- Or From ESXi host
# esxcfg-nas -l

Shows the datastore is mounted properly on the ESXi host.

Check Firmware Versions in UCSM

In Cisco UCS Manager, from Equipment > Firmware Management > Installed Firmware tab, verify for correct firmware versions.

Refer to the HyperFlex Interoperability Matrix for specific versions.

The table below is an example of 3.1(2b) firmware on HX220c server.

System Components Firmware version

Components

Version

UCS Manager

3.1(2b)

UCS Fabric Interconnect A and B

Kernel

5.0(3)N2(3.12b)

System

5.0(3)N2(3.12b)

Server Components

Adapters

4.1(2d)

Bios

C220M4.2.0.13d.0.0812161113

Board Controller

32.0

CIMC Controller

2.0(13e)

Flash Controller 1

1.3.2 build 165

Storage Controller SAS

11.00.00.07

Important: removed old notes about SAS version and re-ack

Upgrading VMware Tools in Storage Controller VMs

To apply VMware Tools to your VMs see the VMware Guest OS guides for instructions on installing VMware Tools specific to the guest operating system on your VMs. This includes your working VMs and the storage controller VMs. For the storage controller, see the VMware KB article, Installing VMware Tools in an Ubuntu virtual machine (1022525).

VMware recommends that if you upgraded VMware Tools as part of a vSphere upgrade, upgrade to the latest virtual hardware version available for the virtual machine. For more information, see Upgrading hardware of a virtual machine to the latest version (1010675).

Procedure

After all ESXi hosts are updated with new tools, when you power on a virtual machine, if a new version of VMware Tools is available, you see a notification in the status bar VMware Tools is outdated on this virtual machine.

1. Click on Upgrade VMware tools in the alert.

2. Automatic upgrade is preselected. Click Upgrade.

Note: This completes the upgrade of VMware tools. It may or may not reboot the VM.

3. Wait for cluster to be healthy before upgrading VMware tools on the next controller VM.

Chapter 8. Known Issues

- Overview

- Upgrade Button Missing in Cisco HX Data Platform

- Cluster Start Command Error: Node Not Available

- A Node Fails to Upgrade due to vCenter Issues

Overview

This chapter provides information to help you troubleshoot common problems that may occur during the Cisco HyperFlexupgrade process.

Upgrade Button Missing in Cisco HX Data Platform

If the upgrade button is missing in the Cisco HX Data Platform Objects tab, under the Actions drop-down list:

- Try using a different browser with the latest version of flash.

Firefox browser is not supported due to an outdated version of flash that is bundled with the browser. Manual update of flash within Firefox is possible, but the recommendation is to use either Chrome or Internet Explorer with a modern version of flash.

- Try restarting VMware vCenter.

Cluster Start Command Error: Node Not Available

Description

After an offline upgrade, due to a VMware EAM issue, sometimes all the controller VMs do not restart. The stcli start cluster command returns an error: Node not available.

Action: Manually power on the controller VMs, then start the storage cluster.

Step 1

Manually power on the controller VMs.

1. Login to the vSphere Web Client.

2. Locate the controller VMs that are not powered on.
From the Navigator select, vCenter Inventory Lists > Virtual Machines > vm.

Storage controller VMs, have the prefix, stCtlVM.

3. From the right-click or Actions menu select, Power > Power On.

4. Repeat until all the storage controller VMs are powered on.

Step 2

Restart the storage cluster.

1. Login to the command line of any controller VM.

2. Run the command.
# stcli cluster start

A Node Fails to Upgrade due to vCenter Issues

Description

Sometimes during an online upgrade the vCenter daemon crashes on a node. When this happens, the node cannot enter HX maintenance mode. Without entering HX maintenance mode, the node cannot complete the upgrade. All other nodes, with properly functioning vCenter, complete the upgrade.

Action: Re-run the Upgrade on the Affected Node

1. Correct the vCenter issue.

2. Re-run the upgrade on the affected node.

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS" WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: Click Here . Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R)