Managing HX Native Snapshots

HX Native Snapshots Overview

HX native snapshots are a backup feature that saves versions (states) of VMs. VMs can be reverted back to a prior saved version using an HX native snapshot. A native snapshot is a reproduction of a VM that includes the state of the data on all VM disks and the VM powerstate (on, off, or suspended) at the time the native snapshot is taken. Taking a native snapshot to save the current state of a VM gives you the ability to revert back to the saved state.

The following methodologies are used in the administration of HX native Snapshots:

  • The vSphere “Manage Snapshots” function can revert to a specific HX native snapshot, or delete all snapshots.

  • Cisco HyperFlex Connect can create on-demand and schedule HX native snapshots.

  • The HyperFlex command line user interface can create HX native snapshots.

  • HX REST APIs can create and delete HX native snapshots.

For additional information about VMware snapshots, see the VMware KB, Understanding virtual machine snapshots in VMware ESXi and ESX (1015180) at, http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1015180

Benefits of HX Native Snapshots

HX native Snapshots provide the following benefits:

  • Revert Registered VMs - If a VM is registered, whether powered-on or powered-off, HX native snapshots and VM snapshots, can be used to revert to an earlier point in time at which the snapshot was created.

  • High Performance -The HX native snapshot process is fast because it does not incur I/O overhead.

  • VM Performance - HX native snapshots do not degrade VM performance.

  • Crash-Consistent - HX native snapshots are crash-consistent by default. I/O crash consistency is defined as maintaining the correct order of write operations to enable an application to restart properly from a crash.

  • Quiescence -HX native snapshots can be created with the guest file system quiesced. The quiesce option is available when using Cisco HyperFlex Connect, the HyperFlex command line user interface, and HX REST APIs. VMware tools should be installed in the guest VM when creating HX native snapshots using the quiesce option.

    Improved performance and reliability of Quiesced Snapshot beginning with HyperFlex Release 4.5(2a) and VMware ESXi 7.0 U2.

    Quiescing a file system is a process of bringing the on-disk data of a physical or virtual computer into a state suitable for backups. This process might include operations such as flushing dirty buffers from the operating system's in-memory cache to disk, or other higher-level application-specific tasks.

    Quiesce based snapshot is not supported on the Windows2008R2 operating system. The operating system is End-of-Life. Please review the VMware documentation for the current list of supported operation systems. VMware Compatibility Guide

    If your system displays quiesce errors, review the following VMware KB article Troubleshooting Volume Shadow Copy (VSS) quiesce related issues (1007696).

  • Scheduled Snapshots are Tolerant to Node Failures - Scheduled snapshots are tolerant to administrative operations that require a node shutdown, such as HX maintenance mode and HX online upgrade.

    Scheduled Snapshots are tolerant to failures in other HX clusters in multi-cluster environments.

  • Granular Progress and Error Reporting - Monitoring is performed at the task level for VM level HX native snapshots.

  • Instantaneous Snapshot Delete - Deletion of an HX native snapshot and consolidation is always instantaneous.

  • VDI Deployment Support - Scheduled HX native snapshots are supported for desktop VMs on VDI deployments which are using VMware native technology.

  • Datastore Access - Snapshots work on partially mounted or accessible datastores as long as the VM being snapshotted is on an accessible mountpoint.

HX Native Snapshot Considerations

Snapshot Parameters


Attention

Beginning with HX Release 4.5(2a) and VMware ESXi 7.0 U2 Sentinel snapshots are not applicable.

  • HX native snapshots - When creating the first HX native snapshot an HX SENTINEL snapshot is created prior to the HX native snapshot. The SENTINEL snapshot is a baseline snapshot which ensures that subsequent snapshots are HX native snapshots. For additional information about SENTINEL snapshots, see Understanding SENTINEL Snapshots. When a SENTINEL snapshot is present, creating additional snapshots using vSphere results in the creation of HX native snapshots.


    Note
    There should be no VMware snapshots (non-native) present when creating native snapshots.

  • HX Snapshots compatibility with VMware VAIO - Creating HX Snapshots with VMWare VAIO configured is not supported. Attempting to create HX snapshots will power the VM off. HX Snapshots cannot coexist with Virtual Machines enabled with vSphere APIs for IO Filtering (VAIO). The VAIO framework might be used by backup solutions to enable Continuous Data Protection (CDP) for Virtual Machines. To use the backup solutions with CDP, delete any existing HX snapshots before enabling the CDP functionality.

    To determine if your product uses VMware VAIO framework, review the list of qualified vendors: https://www.vmware.com/resources/compatibility/search.php?deviceCategory=vaio

  • Maximum Number of Stored Snapshots - VMware has a limitation of 31 snapshots per VM. The limitation total is equal to the sum of all VMware created snapshots, the HX SENTINEL snapshot, and HX native snapshots.

    Beginning with HX Release 4.0 Snapshot operations beyond the number set in the snapshot.maxSnapshots property of the VM fail, with the following error message: Snapshot operation cannot be performed.

  • Scheduled Snapshots - Do not have overlapping snapshots scheduled on VMs and their resource pools.

Performance

  • VMware vSphere Storage APIs Array Integration (VAAI) - To attain the best HX snapshot performance and functionality, upgrade to the ESXi 7.0 U2.

Snapshots During Upgrade Processes

  • HX native snapshots are not supported while an upgrade of HX Data Platform, ESXi, or UCS is in progress.

VMs

Table 1. Release Specific VM Considerations

Release

Consideration

HX Release 3.5 deployments

Ensure that the VM is not attached to Graphic Processing Unit (GPU).

HX Release 3.5.2(a) and earlier

All powered on VMs use synchronous consolidation (asynConsolidate = false) when taking HX native snapshots.

Beginning with HX Release 3.5.(2b)

All powered on VMs use asynchronous consolidation (asyncConsolidate = true) when taking HX native snapshots. If the VM is powered off, the settings remain unchanged.

Beginning with HX Release 4.5(2a) using VMware ESXi 7.0 U2 and later

Consolidation time is no longer proportional to the I/O load on the Virtual Machine.

Sentinel snapshots are no longer created.

The following considerations apply to all supported releases:

  • VM Hardware Version - HX native Snapshots requires the use of VM hardware Version 9 or later. Using the most recent version is recommended.

  • Deleted VMs - The life cycle of HX native snapshots, similar to VM snapshots, is tied to the virtual machine. If the VM is deleted (accidentally or intentionally), all associated HX native snapshots are deleted. HX native snapshots do not provide a mechanism to recover from a deleted VM. Use a backup solution to protect against VM deletion.

  • HX Data Platform storage controller VMs -You cannot schedule HX native snapshots for storage controller VMs.

  • Non-HX Data Platform VMs - HX native snapshots will fail for any VM that is not on a HX datastore. To create an HX native snapshot, the VM must reside on a single HX datastore.

  • Suspended VMs - Creating the first HX native snapshot and the HX SENTINEL snapshot on VMs in a suspended state is not supported.

  • VM Name - The VM name must be unique per vCenter for taking an HX native snapshot.

  • Ready storage cluster - To enable the creation of an HX native snapshot, the storage cluster must be healthy, have sufficient space, and be online. The datastore in which the VM resides must be accessible. The VM must be valid and not in a transient state, such as vMotioning.

Cluster with only 1 on-line Node Remaining

  • HX native Snapshot is not supported on single (1-node) on-line node on a CBT Enabled VM in a powered-on state. Power-off the VM and take the SENTINEL snapshot, subsequent snapshots on powered-on VMs are supported.

vCenter

  • vMotion - vMotion is supported on VMs with HX native snapshots.

  • Storage vMotion - Storage vMotion is not supported on VMs with HX native snapshots. If the VM needs to be moved to a different datastore, delete the snapshots before running storage vMotion.

Naming

  • Duplicate names - VMs or Resource Pools with duplicate names within the HX Data Platform vCenter are not supported and cause HX native snapshots to fail. This includes parents and children within nested resource pools and resource pools within different vCenter clusters.

  • Characters in Names - Special characters are not supported. Using special characters in names results in names appearing different than specified.

  • Maximum Snapshot Name Length - characters.

Disks and Datastores

  • VM Datastores - Ensure that all the VMDKs belonging to a VM are on the same datastore prior to creating HX native snapshots.

    • HX native snapshots are not supported with multiple datastores.

  • Thick Disks - If the source disk is thick, then the HX native snapshot of the VM will also be thick. Increase the datastore size to accommodate the snapshot if required.


    Note
    When creating a new virtual machine disk on the HyperFlex datastore and you want to enable the creation of thick-provisioned disks, there is no option to create a thick-provisioned disk. This is a known issue in VMware. For more information, see Creating VMDK with NFS-backed storage does not allow thick-provisioning with vendor plugin.

    Note
    ESXi cannot distinguish between thick provision lazy zeroed and thick provision eager zeroed virtual disks on NFS datastores. When you use NFS datastores, the vSphere client allows you to create virtual disks in Thick Provision Lazy Zeroed (zeroedthick) or Thick Provision Eager Zeroed (eagerzeroedthick) format. However, when you check the disk type on the Virtual Machine Properties dialog box, the Disk Provisioning section always shows Thick Provision Eager Zeroed as the disk format (no matter which format you selected during the disk creation).

  • Virtual Disk Types - VMware supports a variety of virtual disk backing types. The most common is the FlatVer2 format. HX native snapshots are supported for this format.

    There are other virtual disk formats such as Raw Device Mapping (RDM), SeSparse, VmfsSparse (Redlog format). VMs containing virtual disks of these formats are not supported with HX native snapshots.

Login Access

  • SSH - Ensure that SSH is enabled in ESX on all the nodes in the storage cluster.

Limitations

Object

Maximum Number

HX native snapshots

30 per VM

VMware limit is 31. One snapshot is consumed by the HX SENTINEL snapshot.

Datastores

48 per storage cluster

Maximum VMDK size

3 TB

HX Native Snapshots Best Practices

  • When creating large numbers of HX native snapshots consider the following:

    • Schedule HX native snapshots at a time the expected data traffic is low.

    • Stagger HX native snapshot schedules such that a large numbers of VMs are not scheduled to be snapshotted at the same time.

  • If vCenter running on a VM in the storage cluster, do not take an HX native snapshot of the vCenter VM. This is related to VMware KB, VMware VirtualCenter Server service fails due to a quiesced snapshot operation on the vCenter Server database virtual machine (2003674), at https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2003674

HX Native Snapshots Best Practices HX Release 4.5(1x) and Earlier

Significant updates were introduced in HX Release 4.5(2a) and ESXi 7.0 U2. The following recommendations only apply to users using a release that was introduced before this release.


Important

Always use the HX native snapshots feature to create the first snapshot of a VM. This ensures that all subsequent snapshots are in native format.

  • Do not use the VMware snapshot feature to create the first snapshot of a VM. VMware snapshots use redo log technology that result in degraded performance of the original VM. This performance degrades further with each additional snapshot.

    • If there are redo log snapshots that should be deleted, edit the /etc/vmware/config file and set snapshot.asyncConsolidate="TRUE" on the ESXi host(s) where the redo log snapshots reside.

  • HX native snapshots do not impact VM performance after the initial HX SENTINEL and HX native snapshot are created.

  • Add all the VMDKs to the VM prior to creating the first HX native snapshot.

    When VMDKs are added to a VM, additional HX SENTINEL snapshots are taken. Each additional HX SENTINEL consumes additional space.

    For example, when adding 2 new VMDKs to an existing VM that has HX native snapshots, at the next scheduled HX native snapshot, 1 new HX SENTINEL is created. If it is necessary to add one or more additional VMDKs, review any existing HX native snapshot schedule retention plans and make sure that the total number of retained HX native snapshots plus any HX SENTINEL snapshots will not attempt to exceed a total value of 31.

Understanding SENTINEL Snapshots


Important

This section is for use with HX Release 4.5(1x) and earlier. SENTINEL Snapshots are not used beginning with HX Release 4.5(2x) and ESXi 7.0 U2.

When creating the first HX native snapshot of a VM, a baseline snapshot called an HX SENTINEL snapshot is created as well as the requested HX native snapshot. The HX SENTINEL snapshot ensures that all subsequent of the VM snapshots are HX native snapshots

HX SENTINEL snapshots prevent reverted VMs from having VMware redo log-based virtual disks. Redo log-based virtual disks occur when an original snapshot is deleted and the VM is reverted to the second oldest snapshot.

An HX SENTINEL snapshot consumes one snapshot of the 31 total available per VMware limitation.

Using HX SENTINEL snapshots

  • Do not delete the HX SENTINEL snapshot unless it is a required operation.

  • When using VMware ESXi 7.0 U2, the create snapshot action deletes HX SENTINEL shapshots when the VM only contains HX SENTINEL shapshots. If the VM has user snapshots and HX SENTINEL snapshots, the HX SENTINEL snapshots are not deleted.

  • Do not revert a VM to the HX SENTINEL snapshot.

  • If the HX SENTINEL snapshot needs to be deleted for any reason, all snapshots should be deleted.

HX Native Snapshot Time Zones

There are three objects that display and affect the timestamp and schedule of snapshots:

  • vSphere and vCenter use UTC time.

  • vSphere client (HTML5) uses the browser timezone.

  • The HX vSphere client (HTML5) plug-in, HX storage cluster, and HX storage controller VM use the same time zone. This is enforced across the HX storage cluster. The time zone used by these entities is configurable. The default is UTC.

The HX storage controller VM time is used to set the schedule. The vSphere UTC time is used to create the HX native snapshots. The logs and timestamps vary depending upon the method used to view them.

When a schedule is created using the HX vSphere client (HTML5) plug-in, the scheduled times are converted to UTC from the HX storage controller VM time zone. When you view the schedule through the vSphere client (HTML5) Scheduled Tasks it displays the tasks in browser time zone.

When converted to the same timezone, they translate to the same time. For example: 5:30pm PST, 8:30PM EST, 1:30AM UTC are all the same time.

To have vSphere Scheduled Tasks tab display the same time for a scheduled snapshot that you create in the HX vSphere client (HTML5) plug-in, set the storage controller VM to UTC.

To have scheduled snapshots run based on local time zone settings, set that time zone for the storage cluster. By default, the storage controller VM uses the UTC time zone set during HX Data Platform installation.

If the vSphere and the storage controller VM are not using the same time zone, the vSphere scheduled tasks tab might display a different time than the schedule in the HX vSphere client (HTML5) plug-in schedule snapshot dialog.

When you configure an hourly snapshot, the snapshot schedule runs between a specific start time and end time. The vSphere Task window might display a status that a scheduled snapshot was completed outside the hourly end time based on the timezone

Identify and set the time zone used by the storage controller VM

  1. From the storage controller VM command line, view the configured time zone.

    $ stcli services timezone show

  2. Change the storage cluster time zone.

    $ stcli services timezone set --timezone timezone_code

    See a time zone reference for time zone codes, such as https://en.wikipedia.org/wiki/List_of_tz_database_time_zones

Creating HX Native Snapshots

To create HX native snapshots, perform the following steps:

Before you begin

Remove any redolog snapshots for VMs in the HX storage cluster. If this step is not completed, VMs might be stunned during snapshot consolidation.

Redo log snapshots are snapshots that are created through the VMware snapshot feature and not through the HX native snapshot feature. To edit the ESXi host configuration where the redo log snapshots reside,

  1. Login to the ESXi host command line.

  2. Locate and open the /etc/vmware/config file for editing.

  3. Set the snapshot.asyncConsolidate parameter to TRUE. 

    snapshot.asyncConsolidate="TRUE"

Procedure

Step 1

From the vSphere client (HTML5) Navigator display the list of VMs in vCenter at the VM level. Display the VM list with one of the following methods, Hosts and Clusters, VMs and Templates , Storage, Networking, or Global Inventory Lists

Example:

Global Inventory Lists > VMs
Step 2

Select a storage cluster VM and open the Actions menu. Either right-click the VM or click the Actions menu in the VM information portlet.

Note 

Ensure there are no non-HX Data Platform datastores on the storage cluster resource pool or the snapshot will fail.

Step 3

Select Cisco HX Data Platform > Snapshot Now to display the Snapshot dialog box.

Step 4

Complete the dialog box

Table 2. Take Snapshot Dialog Box

Field

Description and Usage Notes

Name

Type the Snapshot name. Maximum snapshot name length: characters.

Description

Type a description of the snapshot.

Snapshot option check box

Use the check boxes to select Snapshot the virtual machine's memory or Quiesce guest file system (Needs VMware Tools installed)
Step 5

Click OK to create an HX native snapshot.

The Recent Tasks tab displays the status message: 

Create virtual machine native snapshot.
 The first snapshot

HX Native Snapshots using ESXi 7.0 U2

Creating snapshots using ESXi 7.0 U2 introduces the following enhancements:

  • Sentinel snapshots are not created.

  • Supports VAAI offload for all snapshots of a VM by automatically configuring VM attributesnapshot.alwaysAllowNative=TRUE.

  • Improved performance, reliability and functionality.

  • Supports snapshot of VM on spanned datastores.

  • Automatically identifies and deletes sentinels when no longer needed.

Scheduling HX Native Snapshots Overview

Apply HX native snapshot schedules to storage cluster objects, such as VMs or resource pools.


Note

If you re-register the vCenter cluster, your HX native snapshot schedules are lost. If this happens, reconfigure HX native snapshot schedules.

When scheduling an HX native snapshots consider your back up requirements. For critical data, retain more frequent HX native snapshots. If there is a disaster, it is possible to restore recent HX native snapshots or create a custom real-time HX native snapshot. For less critical data, considering creating less frequent HX native snapshots.

HX native snapshot scheduling helps control backup costs. For each VM in a storage cluster, you can schedule hourly, daily, or weekly snapshots. The maximum frequency for any specific VM is once per hour. Hourly settings are available in 15 minute increments.

For example, HX native snapshots are taken each day, given the following settings.
  • VM 1 hourly snapshots to run at hour:15 minutes, between 10 PM and 1 AM.
  • VM 2 hourly snapshots to run at hour:30 minutes, between 8 PM and 12 AM.
  • VM 3 and 4 hourly snapshots to run at hour:45, between 6 AM and 8 AM.
  • VM 5 daily snapshot to run at 6:00 AM
Each day these HX native snapshots are taken. Notice that the last HX native snapshot is before the ending hour:00.
  • 6:00 AM — VM 5
  • 6:45 AM — VM 3, VM 4
  • 7:45 AM — VM 3, VM 4
  • 8:30 PM — VM2
  • 9:30 PM — VM2
  • 10:15 PM — VM1
  • 10:30 PM — VM2
  • 11:15 PM — VM1
  • 11:30 PM — VM2
  • 12:15 AM — VM1

To schedule an HX native snapshot every hour over 24 hours:

Procedure

Step 1

Set the start time
Step 2

Set the end time one hour before the start time.

Example:

hour:15, start 4 PM, end 3 PM.
This takes an HX native snapshot at 4:15 PM, 5:15 PM, ... 12:15 AM, 1:15 AM ... 2:15 PM, 3:15 PM. Then restarts the 24 hour cycle.
Note 
The maximum number of HX native snapshots per VM is 31. One HX SENTINEL snapshot is also required. So, it is possible to take an hourly HX native snapshot and retain the most recent 30 HX native snapshots.

The HX native schedule snapshot displays the set time for the snapshot based on the current time zone setting for the storage controller VM. So, if an HX native snapshot was set at 7 pm PST and the storage controller VM time zone is changed to EST. The next time you open the HX native scheduler window, it automatically updates to 10 pm EST.

Scheduling HX Native Snapshots

Procedure

Step 1

From the vSphere client (HTML5) Navigator Home page, select the VM or resource pool list.

For example, vCenter Inventory Lists > Virtual Machines to display the list of VMs in vCenter.

Step 2

Select a storage cluster VM or resource pool and open the Actions menu.

Either right-click the object or click the Actions menu.

Step 3

From the Actions menu, select Cisco HX Data Platform > Schedule Snapshot to display the Schedule Snapshot dialog box.

Step 4

Select the snapshot frequency.

Click the boxes for hourly, daily, and/or weekly frequency and set the starting days, times, and duration.

Step 5

Set the number of snapshots to retain.

When the maximum number is reached, older snapshots are removed as newer snapshots are created.

Step 6

Unselect existing scheduled items, as needed.

If a previous schedule existed, unselecting items deletes those items from the future schedule.

Step 7

Click OK to accept the schedule and close the dialog.

Setting the Frequency of HX Native Scheduled Snapshots

Create a snapshot every hour at specific times, daily at a specific time, or weekly on selected days and times.

Before you begin

Open the Schedule Snapshot dialog box for a VM or resource pool.

Procedure

Step 1

From the Schedule Snapshot dialog box, select the Enable Hourly Snapshot, Enable Daily Snapshot, or Enable Weekly Snapshot check box.

Step 2

Click the Start at drop-down list to select a start time. Select hour, minutes in 15 minute increments, and AM or PM.

Step 3

For an hourly snapshot schedule, click the Until drop-down list to select an end time. Select hour, minutes in 15 minute increments, and AM or PM. Set the minute to the same value as the Start at time.

The HX Data Platform plug-in creates a snapshot of the VM every hour between the start and end times.

Step 4

Select the corresponding check box to specify Days of the week on which you want to take the snapshots.

Step 5

Under Retention, either type a number or use the arrow button to specify the maximum number of copies to retain for each schedule.

Deleting HX Native Snapshot Schedules

Procedure

Step 1

From the HX vSphere client (HTML5), select the VM or resource pool list.

For example, vCenter Inventory Lists > Virtual Machines to display the list of VMs in vCenter.

Step 2

Select a storage cluster VM or resource pool and open the Actions menu.

Either right-click the object or click the Actions menu.

Step 3

From the Actions menu, select Cisco HX Data Platform > Schedule Snapshot to display the Schedule HX Native Snapshot dialog box.

Step 4

Uncheck the scheduled options that are no longer required.

Step 5

Click OK to accept the changes, this includes deleting previously configured schedules, and exit the dialog.

Step 6

Confirm the schedule is deleted.

Select a storage cluster VM or resource pool. Click the HX vCenter tabs, Manage > Scheduled Tasks. The previous HX native snapshot schedule should not be listed.

Reverting to an HX Native Snapshot

Reverting a snapshot is returning a VM to a state stored in a snapshot. Reverting to a snapshot is performed on one VM at a time. Reverting snapshots is performed through the vCenter Snapshot Manager and not through the HX Data Platform plug-in.

Before you begin

Snapshots of the VM must exist.

Procedure

Step 1

From the vSphere client (HTML5), select the VM level, VM folder level, or resource pool level. For example, vCenter Inventory Lists > Virtual Machines to display the list of VMs in vCenter.

Step 2

Select a storage cluster VM and open the Actions menu. Either right-click the VM or click the Actions menu in the VM information portlet.

Step 3

Select Snapshots > Manage Snapshots to open the vSphere Snapshot Manager.

Step 4

Select a snapshot to revert to from the hierarchy of snapshots for the selected VM.

Step 5

Click Revert to > Yes > Close.

The reverted VM is included in the list of VMs and powered off. In selected cases, a VM reverted from a VM snapshot is already powered on. See the following table for more details.

Table 3. VM Power State After Restoring a HX VM Snapshot

VM State When HX VM Snapshot is Taken

VM State After Restoration

Powered on (includes memory)

Reverts to the HX VM snapshot, and the VM is powered on and running.

Powered on (does not include memory)

Reverts to the HX VM snapshot, and the VM is powered off.

Powered off (does not include memory)

Reverts to the HX VM snapshot, and the VM is powered off.
Step 6

If the reverted VM is powered off, then select the VM and power it on.

Deleting HX Native Snapshots

Deleting HX native snapshots is managed through the vSphere interface and not through the HX vSphere plug-in.

Procedure

Step 1

From the vSphere client (HTML5), select VMs and Templates > vcenter_server > Snapshots > datacenter > vm.

Step 2

Right-click the vm and select Snapshots > Manage Snapshots.

Step 3

Select an HX native snapshot and click Delete.

Note 

Delete the HX SENTINEL snapshot by using Delete All option only. Do not delete the HX SENTINEL snapshot individually.