Cisco HyperFlex Systems Troubleshooting Reference Guide, 4.5

Understanding System Events

HX Data Platform messages include the error, warning, and informational messages that the system displays during various events. These include HX storage cluster-wide events and events due to changes in HX storage cluster components.

Messages are initiated by activities in the HX storage cluster. They are distributed to assorted locations, including:

VMware vCenter Events or Alarms pages—Some messages are directed to the vCenter Events and Alarms pages by the HX Data Platform. Some messages that are stored in HyperFlex log files are queried by vCenter and added to the vCenter Events and Alarm pages.
HX Data Platform Plug-in Monitor > Events tab. See View HX Data Platform Plug-in Events.
HX Data Platform Auto Support (ASUP) system—Auto Support must be enabled to send email notifications. Typically, Auto Support is configured during HX storage cluster creation by configuring the SMTP mail server and adding email recipients.
Smart Call Home (SCH) notifications—SCH is an automated support capability that offers around-the-clock device monitoring, proactive diagnostics, real-time email alerts, service ticket notifications, and remediation recommendations for critical system events on your HX storage clusters.
HX Connect user interface—In the header, the bell icon () displays an alarm count of your current errors or warnings. If there are both errors and warnings, the count shows the number of errors. For more detailed information, see the Alarms page or the Events page in the HX Connect user interface.

View HX Data Platform Plug-in Events

The Monitor Events tab displays information about the state changes of the HX storage cluster. Events include user actions and system actions that occur on the HX storage cluster, hosts, or datastores. For example, adding a node to the HX storage cluster, removing a node from the HX storage cluster, or reconfiguring a VM resource.

You can perform the following tasks in the Events tab:

Select an event to display the event details at the bottom of the tab.
Use the filter controls above the list to filter the list. For example, type memory to display a subset of events.
Click a column heading to sort the list.

Procedure

From the vSphere Web Client Navigator, select vCenter Inventory Lists > Cisco HyperFlex > Systems > Cisco HX Data Platform > cluster > Monitor > Events. You have the following fields:


Field	Description
Description	Event message content. See the section for each event type.
Type	Type of message.
Date Time	Timestamp when the event occurred. Time is displayed in local browser time, calculated from UTC.
Target	Name of the target. Target type options include: storage cluster, host, datastore, or disk.
User	Consumer of the resource for the event.
VC Cluster Events	Link to vSphere storage cluster Events.
Event Detail	Same content for the event as the Event table. Target link - The Target object in the Event detail links to the vSphere target Summary page. For example, the HX storage cluster Summary page or node Summary page.

Automatically Acknowledged Events

Storage cluster-wide events have an auto-acknowledge feature. When an event occurs, a message is published. When the condition for the message is corrected, for example, a node returns to healthy, the original alarm is acknowledged. Alarms can also be manually acknowledged.

The following events are automatically acknowledged when the corresponding event occurs.


Triggering Event	Resetting Event	Alarm Reset to
ClusterHealthCriticalEvent	ClusterHealthNormalEvent	Green
ClusterPolicyComplianceDegradedEvent	ClusterPolicyComplianceSatisfiedEvent	Green
ClusterPolicyComplianceFailedEvent	ClusterPolicyComplianceSatisfiedEvent	Green
ClusterPolicyComplianceImprovedEvent	ClusterPolicyComplianceSatisfiedEvent	Green
ClusterReadonlyEvent	ClusterOnlineEvent	Green
ClusterUnhealthyEvent	ClusterHealthNormalEvent	Green
CriticalInfraComponentEvent	ClusterHealthNormalEvent	Green
FileSystemUsageAlertEvent	FileSystemUsageNormalEvent	Green
HighMemoryUsageAlertEvent	NormalMemoryUsageEvent	Green
HXHostUnreachable	HXHostReachable	Green
HXSshAccessToggleActionFailed	HXSshAccessToggleActionSuccess	Green
HXSyslogServerNotReachableWarningEvent	HXSyslogServerReachableEvent	Green
LowSystemMemoryAlertEvent	SystemMemoryNormalEvent	Green
NtpServerOfflineEvent	NtpServerOnlineEvent	Green
SmartLicenseEvalExpiringEvent	SmartLicenseEvalExpiringEvent	Green
SmartLicenseFeatureNotInLicenseEvent	SmartLicenseEvalExpiringEvent	Green
SpaceAlertEvent	SpaceRecoveredEvent	Green
SpaceCriticalEvent	SpaceRecoveredEvent	Green
SpaceWarningEvent	SpaceRecoveredEvent	Green
VCConnectionDown	VCConnectionUp	Green

Manually Acknowledging Alarms Using HX Connect

The Alarms page displays a list of HX storage cluster and ESXi alarms raised by the HyperFlex Data Platform. Acknowledging an alarm lets other users know that you are taking ownership of the issue. After an alarm is acknowledged, its alarm actions are discontinued. However, the alarm is still visible in the system. Alarms are neither cleared, nor reset when acknowledged.

Note

Starting with HX 4.0(2a), you must acknowledge and reset alarms from the HX Connect user interface and in vCenter. Resetting and acknowledging alarms from vCenter will not reflect in HX Connect and vice versa. For full context about the alarm, it is recommended that you view in HX Connect.

Procedure

Step 1

Log in to HX Connect.

Step 2

In the menu, click Alarms.

Step 3

Click the alarm you want to acknowledge, and then click Acknowledge.

vCenter recognizes a session with HX Connect, therefore system messages that originate with vCenter might indicate the session user; for example, Acknowledged By might list com.springpath.sysmgmt.domain-c7.

Cluster Access Policy Compliance Events

ClusterPolicyComplianceDegradedEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Cluster access compliance has degraded.		Event and Alarm			No	No	See the Cisco HyperFlex Data Platform Administration Guide for information about managing the HX storage cluster.

ClusterPolicyComplianceFailedEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Cluster access compliance has failed.		Event and Alarm		Yes	No	No	See Cisco HyperFlex Data Platform Administration Guide for information about managing the HX storage cluster.

ClusterPolicyComplianceImprovedEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Cluster access compliance has improved.		Event and Alarm			No	No	None.

ClusterPolicyComplianceSatisfiedEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Cluster access compliance is satisfied.					No	No	None.

Cluster Events

ClusterAddedEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
New storage cluster has been added.	Online	Event			No	No	None

ClusterCapacityChangedEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Effective physical capacity was changed with addition or removal of disks or nodes.	Online	Event		Yes	Yes	No	See Cisco HyperFlex Data Platform Administration Guide for information about managing the HX storage cluster.

ClusterHealthCriticalEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
There are three simultaneous non-collocated HDD or SSD, or node failures.	Offline	Event and Alarm		Yes	Yes	Yes	See Cisco HyperFlex Data Platform Administration Guide for information about managing the HX storage cluster.

ClusterHealthNormalEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Cluster returned to normal health state. The system recovers from HDD or SSD, or node failure. Or You replaced failed resources.	Online	Event		Yes	No	No	None

ClusterOnlineEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
You created the storage cluster successfully.	Online	Event		Yes	No	No	See Cisco HyperFlex Data Platform Administration Guide for information about managing the HX storage cluster.

ClusterReadOnlyEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Cluster transitions to ReadOnly mode.	ReadOnly	Event and Alarm		Yes	Yes	Yes	See Cisco HyperFlex Data Platform Administration Guide for information about managing the HX storage cluster. If ASUP enabled, TAC ticket created.

ClusterReadyEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Cluster is online and serving I/Os. NFS datastores are reachable.	Online	Event			No	No	None

ClusterShutdownEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
The system triggered the Cluster Health Critical Event. The storage cluster is shutdown. All VMs are inaccessible.	Offline	Event and Alarm	Yes	Yes	Yes	Yes	See Cisco HyperFlex Data Platform Administration Guide for information about managing the HX storage cluster.

ClusterUnhealthyEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Cluster is unhealthy.	Online	Event and Alarm		Yes	Yes	Yes	See Cisco HyperFlex Data Platform Administration Guide for information about managing the HX storage cluster.

VmMetadataNotSyncedWithIntersight


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Cluster {ClusterName} metadata is not synced with Intersight.	Online	Event and Alarm		Yes	Yes	Yes	See Cisco HyperFlex Data Platform Administration Guide for information about managing the HX storage cluster.

VmMetadataSyncedWithIntersight


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Cluster {ClusterName} metadata is synced with Intersight.	Online	Event and Alarm		Yes	Yes	Yes	See Cisco HyperFlex Data Platform Administration Guide for information about managing the HX storage cluster.

Critical Infrastructure Events

CriticalInfraComponentEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH report results in an SR	Action
Information message when controller VM fails or is shutdown.	Offline	Event and Alarm		Yes	Yes	Yes	See TAC.

Disk Events

DiskAddedEvent


Description Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
You added a new disk.		Event		Event	No	No	None.

DiskFailedEvent


Description Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
A disk failed.		Event and Alarm		Yes	Yes	No	Replace the disk.

DiskHardBlacklistedEvent


Description Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
A disk has failed permanently.		Event and Alarm		Yes	Yes	Yes

DiskHealthEvent


Description Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Disk health deteriorated.		Event	Yes	No	No	No	Customer to review the remaining life value or the disk and decide if it needs replacment. See the following Disk Life Values table for more details.

Table 1. Disk Life Values

Life Remaining %

Comments

0%

May be raised for any drive type - SSDs and HDDs.
The event happens when the internal SMART Health status of the drive reports a predictive failure or the drive fails to power up.
Drive needs Return Merchandise Authorization (RMA).

1% to 14%

Applies to SSDs only.
SSD Wear Level reporting 14% or less life remaining.
Replacement drive needs to be ordered by Customer.

Purchase a replacement SSD(s) to replace any drive(s) past the wear condition not covered by HW replacement contracts.

Note

It is possible for a SSD to wear out eventually and report 0% life remaining. However, the wearout may be identified by reviewing the nightly wear-level check logs; Look for logs and events from 14% remaining life onwards.

DiskPhysicalAddedEvent


Description Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
A disk was added to the storage cluster.		Event and Alarm		Yes	No	No	None.

DiskPhysicalRemovedEvent


Description Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
A disk was removed from the storage cluster.		Event and Alarm		Yes	No	No	Replace the disk.

DiskRepairingEvent


Description Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
A disk is being repaired.		Event		Yes	No	No

DiskRepairSucceededEvent


Description Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
A disk was successfully repaired.		Event		Yes	No	No

DiskRemovalCompletedEvent


Description Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
A disk removal succeeded.		Event		Yes	No	No

DiskRemovalFailedEvent


Description Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
A disk removal failed.		Event		Yes	No	No

DiskRemovalInProgressEvent


Description Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
A disk removal is in progress.		Event		Yes	No	No

DiskRemovedEvent


Description Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
You either physically removed a disk or it became offline.		Event and Alarm		Yes	No	No	Replace the disk.

Host Events

HXHostUnreachable


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported in HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Triggered when HX HyperV host is unreachable.	Online	Event		Yes		No	No	None

HXHostOffline


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported in HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Triggered when Get-ClusterNode command fails when FOC is down	Online	Event		Yes		No	No	None

HXUBRNotSupported


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported in HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Triggered when UBR is unsupported version or quorum configuration for the fail over cluster is using SMB access point.	Online	Event		Yes		No	No	None

Node Events

NodeDecommissionedEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Node removed from storage cluster.		Events		Yes	No	No

NodeJoinedEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
A new node joined the storage cluster.		Event		Yes	No	No

NodeMaintenanceEnteredEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Node entered Maintenance Mode.		Events			No	No

NodeMaintenanceExitedEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Node exited Maintenance Mode.		Events		Yes	No	No

NodePoweredDownEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
You powered down a storage controller VM node. The storage cluster operates with reduced capacity or might be in a Degraded state.		Event		Yes	No	No

NodeReadyForIOEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Node is ready for I/O operations.		Event		Yes	No	No

NodeReadyForShutdownEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Node prepared for shutdown.				Yes	No	No

NodeCriticalEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Cannot access the node due to I/O failure to the node.	Offline	Event and Alarm		Yes	No	No

ComputeNodeAddedEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Compute node added to HX cluster successfully				Yes	No	No

Encryption Events

Cluster Level SED Events

EncryptionLocalConfigDisableCompletedEvent
EncryptionRemoteConfigDisableCompletedEvent
EncryptionLocalConfigEnableCompletedEvent
EncryptionRemoteConfigEnableCompletedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Disabling or enabling encryption configuration on the local or remote cluster successful.	Online	Events				No	No	None

EncryptionLocalConfigDisableFailedEvent
EncryptionRemoteConfigDisableFailedEvent
EncryptionLocalConfigEnableFailedEvent
EncryptionRemoteConfigEnableFailedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Disabling or enabling encryption configuration on the local or remote cluster failed.	Online	Events		Yes		No	No	Identify and correct the issue and retry the task.

EncryptionLocalConfigDisableStartedEvent
EncryptionRemoteConfigDisableStartedEvent
EncryptionLocalConfigEnableStartedEvent
EncryptionRemoteConfigEnableStartedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Disabling or enabling encryption configuration on the local or remote cluster started.	Online	Events		Yes		No	No	None.

EncryptionLocalConfigRekeyCompletedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Configuring Rekey of a disk on the local cluster successful. Message for ConfigRekey task precedes the ReKey task for local cluster encryption.	Online	Events				No	No	None

EncryptionLocalConfigRekeyFailedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Configuring ReKey of a disk on the local cluster failed. Message for ConfigRekey task precedes the ReKey task for local cluster encryption.	Online	Events		Yes		No	No	Identify and correct the issue and retry the task.

EncryptionLocalConfigRekeyStartedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Configuring ReKey of a disk on the local cluster started. Message for ConfigRekey task precedes the ReKey task for local cluster encryption.	Online	Events		Yes		No	No	None.

EncryptionLocalRekeyCompletedEvent
EncryptionRemoteRekeyCompletedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Configuring Rekey of a disk on the local or remote cluster successful.	Online	Events				No	No	None

EncryptionLocalRekeyFailedEvent
EncryptionRemoteRekeyFailedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Configuring ReKey of a disk on the local or remote cluster failed.	Online	Events		Yes		No	No	Identify and correct the issue and retry the task.

EncryptionLocalRekeyStartedEvent
EncryptionRemoteRekeyStartedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Configuring ReKey of a disk on the local or remote cluster started.	Online	Events		Yes		No	No	None.

EncryptionLocalDisableCompletedEvent
EncryptionLocalEnableCompletedEvent
EncryptionRemoteDisableCompletedEvent
EncryptionRemoteEnableCompletedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Disabling or enabling encryption on the local or remote cluster successful.	Online	Events				No	No	None

EncryptionLocalDisableFailedEvent
EncryptionLocalEnableFailedEvent
EncryptionRemoteDisableFailedEvent
EncryptionRemoteEnableFailedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Disabling or enabling encryption on the local or remote cluster failed.	Online	Events		Yes		No	No	Identify and correct the issue and retry the task.

EncryptionLocalDisableStartedEvent
EncryptionLocalEnableStartedEvent
EncryptionRemoteDisableStartedEvent
EncryptionRemoteEnableStartedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Disabling or enabling encryption on the local or remote cluster started.	Online	Events		Yes		No	No	None.

EncryptionLocalRemoteCompletedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Encryption task on the local or remote cluster successful.	Online	Events				No	No	None

EncryptionLocalRemoteFailedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Encryption task on the local or remote cluster failed.	Online	Events		Yes		No	No	Identify and correct the issue and retry the task.

EncryptionLocalRemoteStartedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Encryption task on the local or remote cluster started.	Online	Events		Yes		No	No	None.

EncryptionLocalRemoteConfigCompletedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Encryption configuration on the local or remote cluster successful.	Online	Events				No	No	None

EncryptionLocalRemoteConfigFailedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Encryption configuration task on the local or remote cluster failed.	Online	Events		Yes		No	No	Identify and correct the issue and retry the task.

EncryptionLocalRemoteConfigStartedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Encryption configuration task on the local or remote cluster started.	Online	Events		Yes		No	No	None.

EncryptionOperationTimeoutEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Encryption task on the local or remote cluster timed out.	Online	Events		Yes		No	No	Identify and correct the issue and retry the task.

Node Level SED Events

EncryptionNodeDisableStartEvent
EncryptionNodeEnableStartEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Disable or enable encryption on a node in the HX cluster started.	Online	Events				No	No	None.

EncryptionNodeDisableSuccessEvent
EncryptionNodeEnableSuccessEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Disable or enable encryption on a node in the HX cluster successful.	Online	Events				No	No	None

EncryptionNodeRekeyStartEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
ReKey a node in the HX cluster started.	Online	Events				No	No	None.

EncryptionNodeRekeySuccessEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
ReKey a node in the HX cluster successful.	Online	Events				No	No	None

Disk Level SED Events

EncryptionDiskDisableEvent
EncryptionDiskEnableEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Disable or enable encryption on a disk on a node in the HX cluster task.	Online	Events				No	No	None

EncryptionDiskDisableFailedEvent
EncryptionDiskEnableFailedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Disable or enable encryption on a disk on a node in the HX cluster failed.	Online	Events				No	No	None

EncryptionDiskEraseEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Erase an encrypted disk on a node in the HX cluster task.	Online	Events				No	No	None

EncryptionDiskEraseFailedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Erasing an encrypted disk on a node in the HX cluster failed.	Online	Events				No	No	None

EncryptionDiskRekeyEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
ReKey an encrypted disk on a node in the HX cluster task.	Online	Events				No	No	None

EncryptionDiskRekeyFailedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
ReKey an encrypted disk on a node in the HX cluster failed.	Online	Events				No	No	None

EncryptionDiskSecureDriveEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Secure an encrypted disk on a node in the HX cluster task.	Online	Events				No	No	None

EncryptionDiskSecureDriveFailedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
An encryption secured disk on a node in the HX cluster failed.	Online	Events				No	No	None

EncryptionDiskSecureDriveUnlockFailedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Unlocking a failed encryption secured disk on a node in the HX cluster failed.	Online	Events				No	No	None

EncryptionDiskUnlockForeignEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Unlocking an unknown disk on a node in the HX cluster task.	Online	Events				No	No	None

EncryptionDiskUnlockForeignFailedEvent


Description	Cluster State	Reported in vCenter	Reported in HXDP Plug-in	Reported HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Unlocking an unknown disk on a node in the HX cluster failed.	Online	Events				No	No	None

Replication and Recovery Events

Note

Events that are reported in vCenter are listed in the respective object > Events page. For example, virtual machine events or resource pool events.

AddedDpGroup


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported in HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Virtual Machine protection group created.	Online	Event		Yes		No	No	None

AddedDpVm


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported in HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Virtual Machine protection initiated.	Online	Event		Yes		No	No	None

CreateDpVmSnapshotFailed


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported in HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Virtual Machine snapshot creation failed.	Online	Event		Yes		No	No	None

FailoverFailed


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported in HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Triggered when virtual machine recovery fails.	Online	Event		Yes		No	No	None

RecoveryFailed


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported in HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Virtual Machine recovery failed.	Online	Event		Yes		No	No	None

RecoveryInitiated


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported in HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Initiated recovery of Virtual Machine.	Online	Event		Yes		No	No	None

RecoverySucceeded


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported in HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Successfully recovered Virtual Machine.	Online	Event		Yes		No	No	None

ReplicationFailed


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported in HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Virtual Machine replication failed.	Online	Event		Yes		No	No	None

ReplicationPeriodExceeded


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported in HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Virtual Machine replication did not complete in the specified interval.	Online	Event		Yes		No	No	None

TestFailoverFailed


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported in HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Triggered when virtual machine test recovery fails.	Online	Event		Yes		No	No	None

Security Events

LockdownModeEnabledAlarm


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported in HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Triggered when lock down mode is enabled on one or more nodes.	Online	Event		Yes		No	No	None

HXSyslogServerNotReachableWarningEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported in HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Triggered when syslog server is not reachable."	Online	Event		Yes		No	No	None

v


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported in HX Connect	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Triggered when controller VM SSH access toggle action failed.	Online	Event		Yes		No	No	None

Resource Monitoring Events and Alarms

This table describes the possible event severities in increasing order of severity:


Severity Level	Description
info	A basic notification or informational message that is possibly independently insignificant.
warning	A potential or impending service-affecting fault that currently has no significant effects in the system. An action should be taken to further diagnose, if necessary, and correct the problem to prevent it from becoming a more serious service-affecting fault.
error	A nonservice-affecting fault condition that requires corrective action to prevent a more serious fault from occurring. If the issue can not be resolved, contact support at http://www.cisco.com/techsupport.
critical	A service-affecting condition that requires immediate corrective action. If the issue can not be resolved, contact support at http://www.cisco.com/techsupport.

Name/Description

Cluster State

Reported to vCenter

Shown on HX Connect UI as Event

Send ASUP

Send SCH

SCH report results in an SR

Action

Info Severity Level

CoreFileInfoEvent-HX Controller VM has one or more core files

Yes

No

No user action required

Warning Severity Level

FileSystemUsageAlertEvent-Not enough free capacity on one or more partitions of HX Controller VM

Yes

No

Login to the Controller VM and check the File System usage.

NtpClockDriftEvent-HX Controller VM clock not in sync with configured NTP servers

Yes

No

Confirm whether your NTP source is reachable from the Controller VM or from the Storage cluster. Run the show ntp command to determine the issue.

NtpClockOfflineEvent-One or more NTP servers configured on HX controller VM are not responding

Yes

No

Confirm whether your NTP source is reachable from the Controller VM or from the Storage cluster. Run the show ntp command to determine the issue.

Note

If this alarm persists for more than 24 hours then contact support.

VCConnectionDown-Node has lost connection to vCenter

Yes

No

Confirm whether the vCenter is reachable from the Controller VM.

Note

If this alarm persists for more than 24 hours then contact support.

Error Severity Level

ClusterUnhealthyEvent-Cluster is unhealthy

Yes

See list of possible alarms.

Critical Severity Level

NodeCriticalEvent-Cannot access the node due to I/O failure to the node

Yes

No

Requires immediate corrective action. Contact support for assistance to resolve the issue.

HX Data Platform Alarms


Name/Description	Alarm Code	Message	Severity	Send SCH	SCH Generates an SR
DiskRemovedEvent-Triggered when a disk is removed.	HXA-DSK-0001	A disk {UUID} was removed. {DISKDETAILS}	Warning	No	No
DiskFailedEvent-Triggered when a disk fails in a host.	HXA-DSK-0002	A disk failed. {DISKDETAILS}	Warning	Yes	No
DiskPhysicalRemovedEvent-Triggered when a physical disk is removed in a host.	HXA-DSK-0003	Physical disk {UUID} ({DISKPATH}) on controller host {HOSTNAME} at slot {DISKSLOT} was removed. {DISKDETAILS}	Warning	No	No
DiskHardBlacklistedEvent-Triggered when disk in a HX cluster is in a hard black list and cannot be recovered.	HXA-DSK-0004	{DISKMEDIUM} Disk {DISKPATH} on node {HOSTNAME} has failed permanently. {DISKDETAILS}	Warning	Yes	Yes
DiskHealthEvent-Triggered when a disk deteriorates in a cluster.	HXA-DSK-0005	Disk health has deteriorated. SSD with UUID: {UUID} and S/N: {SN} on host {HOSTNAME} ({IPADDRESS}) has {LIFE_LEFT}% remaining life. SSD must be replaced. {DISKDETAILS}	Warning	No	No
VCConnectionDown-Triggered when node has lost connection to VCenter.	HXA-NOD-0012	{NAME} has lost connection to VCenter.	Warning	No	No
ClusterUnhealthyEvent-Triggered when the health status of the HX cluster is unhealthy. This alarm is reset to green when the cluster health status is back to normal.	HXA-CLU-0002	Cluster [{NAME}] is unhealthy.	Critical	Yes	Yes
NodeCriticalEvent-Triggered due to I/O access failure to the node.	HXA-NOD-0001	Cannot access the node due to I/O failure to the node.	Critical	No	No

Smart Call Home Events

CallhomeEndpointConnectionFailedEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Node is unable to reach Smart Call Home endpoint.	Online	Event			No	No	None

CallhomeEndpointConnectionOKEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Node is able to reach Smart Call Home endpoint.	Online	Event			No	No	None

Smart License Events

SmartLicenseEvalEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Smart License for the cluster is in Evaluation Mode.	Online	Event			No	No	None

SmartLicenseEvalExpiringEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Smart License Evaluation period is expiring soon	Online	Event		Yes	No	No	Check the system information page in HX Connect for details on license expiry register the cluster with Cisco Smart Software Manager by going to Cisco Software Central and clicking “Smart Software Licensing”

SmartLicenseEvalExpiredEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Smart License Evaluation period for the cluster has expired.	Online	Event		Yes	No	No	Register the cluster with Cisco Smart Software Manager by going to Cisco Software Central and clicking “Smart Software Licensing”

SmartLicenseFeatureNotInLicenseEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
The feature used in the cluster isn't avaliable in the HX license tier.	Online	Event		Yes	No	No	The feature will reset when the license meets compliance.

SmartLicenseInComplianceEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
The number of licenses allocated to the cluster is compliant with Smart Licensing.	Online	Event			No	No	None.

SmartLicenseOutOfComplianceEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Insufficient number of Smart Licenses in the Virtual Account makes the cluster non compliant.	Online	Event		Yes	No	No	Purchase or transfer applicable licenses to your Cisco Smart Software Manager Virtual Account by going to Cisco Software Central and clicking “Smart Software Licensing”

Snapshot Events

ScheduledSnapshotRPReachedMaxLimitEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
A VM resource pool scheduled snapshot failed. Scheduled snapshot failed since the virtual machine has reached maximum number of supported snapshots.		Event and Alarm			No	No	Adjust the retention policy on scheduled snapshot or delete user created snapshots.

ScheduledSnapshotVMFolderReachedMaxLimitEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
A VM folder scheduled snapshot failed. Scheduled snapshot failed since the Virtual Machine has reached maximum snapshots supported.		Event and Alarm			No	No	Adjust the retention policy of scheduled snapshot or delete user created snapshots.

ScheduledSnapshotVMReachedMaxLimitEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
A VM scheduled snapshot failed. Scheduled snapshot failed since the virtual machine has reached maximum number of supported snapshots.		Event and Alarm			No	No	Adjust the retention policy on scheduled snapshot or delete user created snapshots.

Space Utilization Events

This section describes the messages that appear related to the space used by the storage cluster.

Note

When you add more nodes to the storage cluster, the HX Data Platform plug-in does not immediately reflect the storage cluster capacity.

SpaceAlertEvent


Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH report results in an SR	Action
Space capacity usage remains at error level. This alert is issued after storage capacity has been reduced, but is still above the warning threshold. The overall cluster storage capacity consumed is beyond the healthy threshold. The storage cluster is online and can perform write operations. HX Summary Health bar: Orange	Online	Event and Alarm	Event	Yes	Yes	Yes	Add storage or delete files. Continue to reduce the amount of storage capacity used, until it is below the warning threshold.

SpaceCriticalEvent


Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH report results in an SR	Action
Space capacity usage is at error level. The cluster storage is full and offline. Your system has exceeded the functional amount of allocated capacity. The storage cluster transitions to read-only mode. It does not accept any write operations. This is a critical threshold. The system triggers a Critical alarm in vCenter. HX Summary Health bar: Red	ReadOnly	Event and Alarm	Event	Yes	Yes	No	Add storage or delete files. Reduce the amount of storage capacity used to below the warning threshold. Reclaimed space released after deleting files within few hours of deletion, based on system workload. The storage controller VM monitors and reclaims space until the system becomes operational (transitions out of read-only state).

SpaceRecoveredEvent


Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH report results in an SR	Action
Used capacity is below the warning threshold. Usage is back to normal. HX Summary Health bar: Blue	Online		Event	Yes	No	No	None.

SpaceWarningEvent


Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH report results in an SR	Action
Space capacity usage is at error level. This is the first level warning you are approaching a critical state. The overall cluster storage capacity consumed is beyond the healthy threshold. This is a warning threshold. The system triggers an Error alarm in vCenter. The storage cluster can perform write operations. HX Summary Health bar: Yellow	Online	Alarm	Event	Yes	Yes	No	Add storage or delete files. Do not continue the storage cluster operations until you reduce the amount of storage capacity used to below this warning threshold.

VirtualSpaceWarnEvent


Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH report results in an SR	Action
Cluster virtual usage is above warning threshold.		Event and Alarm		Yes	Yes	No

VirtualSpaceWarnClearEvent


Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH report results in an SR	Action
Cluster virtual space usage is back to normal.		Event		Yes	No	No

Storage Events

StorageApdTimeoutEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
On an ESX server and APD state was entered and was not exited within defined number of seconds.				Yes	Yes	Yes

Upgrade Events

ClusterUpgradeCompletedEvent


Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
The storage cluster upgraded successfully.	Online	Event		Yes	No	No

ClusterUpgradeFailedEvent


Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
One or more nodes in the storage cluster failed to upgrade.	Offline	Event		Yes	No	No	See TAC.

ClusterUpgradeStartedEvent


Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Cluster upgrade to version `<version>` started.	Online	Event		Yes	No	No

NodeUpgradeCompletedEvent


Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Node upgrade to version `<version>` for `node_ID` completed successfully. `node_ID` is the ID of the node that appears in the message.	Online	Event		Yes	No	No

NodeUpgradeFailedEvent


Trigger	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Node upgrade for `node_ID` failed. `node_ID` is the ID of the node that appears in the message.	Degraded	Event and Alarm		Yes	No	No	See TAC.

NodeUpgradeStartedEvent


Description	Cluster State	Reported vCenter	Reported HXDP Plug-in	Reported ASUP	Reported SCH	SCH Generates an SR	Action
Node upgrade to version `<version>` started.	Degraded	Event		Yes	No	No

Bias-Free Language

Results

Chapter: HX Data Platform Events

HX Data Platform Events

Understanding System Events

View HX Data Platform Plug-in Events

Procedure

Automatically Acknowledged Events

Manually Acknowledging Alarms Using HX Connect

Procedure

Cluster Access Policy Compliance Events

Cluster Events

Critical Infrastructure Events

Disk Events

DiskAddedEvent

DiskFailedEvent

DiskHardBlacklistedEvent

DiskHealthEvent

DiskPhysicalAddedEvent

DiskPhysicalRemovedEvent

DiskRepairingEvent

DiskRepairSucceededEvent

DiskRemovalCompletedEvent

DiskRemovalFailedEvent

DiskRemovalInProgressEvent

DiskRemovedEvent

Host Events

Node Events

NodeDecommissionedEvent

NodeJoinedEvent

NodeMaintenanceEnteredEvent

NodeMaintenanceExitedEvent

NodePoweredDownEvent

NodeReadyForIOEvent

NodeReadyForShutdownEvent

NodeCriticalEvent

ComputeNodeAddedEvent

Encryption Events

Cluster Level SED Events

Node Level SED Events

Disk Level SED Events

Replication and Recovery Events

Security Events

Resource Monitoring Events and Alarms

HX Data Platform Alarms

Smart Call Home Events

Smart License Events

Snapshot Events

Space Utilization Events

Storage Events

Upgrade Events

Was this Document Helpful?

Contact Cisco