Troubleshoot Hyperflex Maintenance Mode Failures

Introduction

This document describes the most common issues when you enter a host in Maintenance Mode in HyperFlex.

Prerequisites

Requirements

• Basic understanding of VMware vSphere
• Basic understanding of UCS Manager(UCSM)
• Basic understanding of Networking

Components Used

The information in this document is based on these software and hardware versions:

• HyperFlex StorageController 5.0.2d
• VMware ESXi, 7.0.3, 21930508
• vCenter Version: 8.0.2 Build-22617221

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

When you encounter issues entering a host into Maintenance Mode in Cisco HyperFlex, there are several troubleshooting steps you can take to identify and resolve the issue.

Configure

If initiating Maintenance Mode for a host via HX Connect is unsuccessful, consider using the ESXi UI to enter Maintenance Mode, as this can help identify and eliminate typical vCenter-related complications.

Tip: If Distributed Resource Scheduler (DRS) is enabled in the cluster, it can manage the vMotion process automatically. Ensure that DRS is set to an appropriate automation level, or if DRS is disabled, you can need to manually migrate or power off other VMs on the source host to balance the load.

1. Use an SSH client to connect to the ESXi using root login.

2. Check whether the host is in maintenance mode. (The specific commands can vary across different versions.)

esxcli system maintenanceMode get

3. Place the host in maintenance mode.

esxcli system maintenanceMode set -e true

4. Verify that the host is in maintenance mode.

esxcli system maintenanceMode get

5. Exit from maintenance mode.

esxcli system maintenanceMode set -e false

Common Errors

Configure the VMkernel Adapter

1. Log into the vCenter Server using vSphere Client.
2. Click to select the host.
3. Click the Configuration tab.
4. Click VMkernel Adapter under Networking.
5. Click Add Networking.
6. Select the VMkernel Adapter and click Next.
7. Select the Existing Standard vSwitch, select the vMotion vSwitch, and click Next.
8. Enter a name in the Network Label to identify the network that vMotion uses.
9. Select or Enter a VLAN ID from the VLAN ID.
10. Check the vMotion service checkbox and click Next.
11. Enter the IP address and Subnet Mask of the host's vMotion Interface.
12. Click Next, then click Finish.

Verify that vSwitch is Using the Appropriate Uplinks

1. Log into the vCenter Server using vSphere Client.
2. Click to select the host.
3. Click Virtual Switches under Networking.
4. Select Standard Switch: vMotion.
5. Click Manage Physical Network Adapters.
6. Active adapters: vmnic3
7. Standby adapters vmnic7
8. Click Ok to Finish.

Note: For Hyperflex Edge Cluster - The vMotion VMkernel port (vmk2) is configured using the post_install script. The failover order is set for active/standby.

ESXi-Firewall Settings

From ESXi, verify that any firewalls between the hosts are not blocking vMotion traffic. vMotion uses TCP port 8000 by default, but other ports can also be involved depending on your configuration.

Source: remote (on-premises) ESXi vMotion VMkernel networks

Destination: ESXi

Port: TCP 8000

[root@Monterrey-server-2:~] nc -zv 10.31.123.195 8000
Connection to 10.31.123.195 8000 port [tcp/*] succeeded!

Note: If this is a new deployment, run the hx_post_install script to configure vMotion interfaces per best practice.

Run the hx_post_install Script

Use an SSH client to connect to cluster virtual IP using admin log in.

Type hx_post_install and hit Enter.

admin@SpringpathController:~$ hx_post_install

Select post_install workflow-

1. New/Existing Cluster
2. Expanded Cluster (for non-edge clusters)
3. Generate Certificate

Note: Workflow No.3 is mandatory to have unique SSL certificate in the cluster. By Generating this certificate, it will replace your current certificate. If you're performing cluster expansion, then this option is not required.

Selection: 1
Logging in to controller HX-01-cmip.example.com
HX CVM admin password:
Getting ESX hosts from HX cluster...
vCenter URL: 192.168.202.35
Enter vCenter username (user@domain): administrator@vsphere.local
vCenter Password:
Found datacenter HX-Clusters
Found cluster HX-01

post_install to be run for the following hosts:
HX-01-esxi-01.example.com
HX-01-esxi-02.example.com
HX-01-esxi-03.example.com

Enter ESX root password:

Enter vSphere license key? (y/n) n

Enable HA/DRS on cluster? (y/n) y
Successfully completed configuring cluster HA.

Disable SSH warning? (y/n) y

Add vmotion interfaces? (y/n) y
Netmask for vMotion: 255.255.254.0
VLAN ID: (0-4096) 208
vMotion MTU is set to use jumbo frames (9000 bytes). Do you want to change to 1500 bytes? (y/n) y
vMotion IP for HX-01-esxi-01.example.com: 192.168.208.17
Adding vmotion-208 to HX-01-esxi-01.example.com
Adding vmkernel to HX-01-esxi-01.example.com
vMotion IP for HX-01-esxi-02.example.com: 192.168.208.18
Adding vmotion-208 to HX-01-esxi-02.example.com
Adding vmkernel to HX-01-esxi-02.example.com
vMotion IP for HX-01-esxi-03.example.com: 192.168.208.19
Adding vmotion-208 to HX-01-esxi-03.example.com
Adding vmkernel to HX-01-esxi-03.example.com

Add VM network VLANs? (y/n) y
 Attempting to find UCSM IP
 Found UCSM 10.75.61.254, logging with username admin.  Org is HX-Cluster
 UCSM Password:
 Port Group Name to add (VLAN ID will be appended to the name): USERS
 VLAN ID: (0-4096) 1219
 Adding VLAN 1219 to FI
 Adding VLAN 1219 to vm-network-a VNIC template
 Adding USERS-1219 to HX-01-esxi-01.example.com
 Adding USERS-1219 to HX-01-esxi-02.example.com
 Adding USERS-1219 to HX-01-esxi-03.example.com
Add additional VM network VLANs? (y/n) n

Run health check? (y/n) y

Validating cluster health and configuration...

Cluster Summary:
Version - 3.5(2i)
Model - HXAF220C-M5SX
Health - HEALTHY
ASUP enabled - False

admin@SpringpathController:~$

The Operation Could not Proceed Because vCenter is not Reachable

Solution: Re-register vCenter

1. Use an SSH client to connect to cluster virtual IP using admin login.
2. Run this command:

stcli cluster reregister --vcenter-datacenter ...

• --vcenter-datacenter <datacenter name>
• --vcenter-cluster <cluster name>
• --vcenter-url <vcenter url ip or hostname>
• --vcenter-sso-url <sso url or use vcenter ip otherwise use vcenter name/ip>
• --vcenter-user <vcenter username>Example

stcli cluster reregister --vcenter-datacenter "HX-DC-test" --vcenter-cluster "HX-Cluster-test" --vcenter-url "test.vsphere.lab" --vcenter-user "administrator@vsphere.local"

Firewall Settings

Ensure that ports 80 and 443 are open for incoming traffic to maintain proper server functionality.

nc -v <vcenter ip> 443
nc -v <vcenter ip> 80

hxshell:~$ nc -v 10.31.123.186 80
Connection to 10.31.123.186 80 port [tcp/http] succeeded!

hxshell:~$ nc -v 10.31.123.186 443
Connection to 10.31.123.186 443 port [tcp/https] succeeded!

Caution: If you have recently replaced the motherboard, you can reach out to the Technical Assistance Center (TAC) to assist you in running the Motherboard Replacement Script.

If the problem persists after performing these checks, consider reaching out to Cisco support for HyperFlex or VMware support for assistance with the vMotion issue. They can provide specific guidance based on your environment and the exact error messages you are encountering.