Back Up and Restore Data on CMX 10.5 and Above
Available Languages
Contents
Introduction
This document describes how to back up and restore configuration and client data on a Cisco CMX 10.5 and later
Prerequisites
Requirements
General knowledge of CMX is required.
Components Used
All tests were performed on a CMX 10.6.0-177 running on MSE 3375 appliance, MacOS 10.4 and Windows 10 October 2018 update.
This includes CMX installed on a physical 3365/3375 appliance, as well as on a virtual machine. These components of CMX can be backed up:
- Database - Stores configuration data, such as, maps, controllers, location, and aggregated analytics data
- Cache - Stores analytics repeat visits
- Cassandra - Stores location history data and analytics raw visits
- Influxdb - Stores metrics data for systems (not included by default)
- Consul - Stores Consul configurations
- Floormaps - Stores floor images for UI display
- Licenses - Stores Cisco CMX license information
- Setup - Stores CMX setup data
- Connectimages - Stores Images on Connect Captive Portal
- Conf - Stores node configuration
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Backup process
Creating the backup bundle
CMX, no matter where installed, can be backed up using a cmxos backup command. By default, backup includes database, cache, cassandra, floormaps, licenses, setup, connectimages and configuration. Add the --all parameter to include the Influxdb data as well. By default, backup process stops CMX services while being performed. Add the parameter --online to perform the backup without stopping the CMX services. You are prompted to enter the directory where you want to save the backup tar.gz archive. Directory needs to have read, write and execute permissions. It is recommended to use the default /tmp directory.
On a freshly installed CMX, backup process takes ~30 seconds. On a fully loaded and utilized CMX, creation of backup bundle can take up to an hour.
Make sure to enable keepalive messages in your SSH client so that the session does not time out while backup is being created. In PuTTY, this can be done under "Connection" Tab:
[cmxadmin@mse33752 ~]$ cmxos backup --online --all Please enter the path for backup file [/tmp]: backup name: cmx_backup_mse33752_2019_04_28_22_39 backup dir: /tmp/cmx_backup_mse33752_2019_04_28_22_39 tar file: /tmp/cmx_backup_mse33752_2019_04_28_22_39.tar.gz running: sudo -u cmx /opt/cmx/bin/cmxctl version ---------------------------------------------------------------------- Build Version : 10.6.0-331 Build Time : 2019-01-24 13:27:35.937025 ---------------------------------------------------------------------- Image Version : 10.6.0-177 ---------------------------------------------------------------------- Preparing backup of following services: ['database', 'cache', 'cassandra', 'influxdb', 'floormaps', 'licenses', 'setup', 'connectimages', 'conf'] [22:39:56] Preparing for backup... Preparing for backup... Database size 51226723 Cache size 7794 Cassandra size 67462961 Floormaps size 1014394 Licenses size 6 Setup size 1912 Connectimages size 6 running: sudo -u cmx /opt/cmx/bin/cmxctl dump running locally Dumping configuration information... [localhost] Executing task 'dump_config_only' Done. . .
.
.
.
.
.
copy snapshot took 0.804718971252 seconds Backup Cassandra DB took: 8.50579595566 seconds [22:40:07] Backup InfluxDb... Backup InfluxDb... Backup Influx DB took: 0.0411479473114 seconds [22:40:07] Backup Floormaps... Backup Floormaps... Backup floor maps took: 0.055881023407 seconds [22:40:07] Backup licenses... Backup licenses... Backup licenses took: 0.000136137008667 seconds [22:40:07] Backup setup... Backup setup... Backup setup took: 0.00061297416687 seconds [22:40:07] Backup connect images... Backup connect images... Backup connect images took: 0.000127077102661 seconds [22:40:07] Backup node configuration... Backup node configuration... running: sudo -u cmx /opt/cmx/bin/cmxctl dump running locally Dumping configuration information... [localhost] Executing task 'dump_config_only' Done. Backup configuration took: 0.383893013 seconds [22:40:07] Creating tar file.. Creating tar file.. running: tar -chf /tmp/cmx_backup_mse33752_2019_04_28_22_39.tar.gz --use-compress-program=pigz -C /tmp cmx_backup_mse33752_2019_04_28_22_39 running: chmod a+rw /tmp/cmx_backup_mse33752_2019_04_28_22_39.tar.gz running: chown cmxadmin:cmxadmin /tmp/cmx_backup_mse33752_2019_04_28_22_39.tar.gz Post backup took: 0.17880988121 seconds Done Backup. Created backup file /tmp/cmx_backup_mse33752_2019_04_28_22_39.tar.gz [22:40:07] Done Backup. Created backup file /tmp/cmx_backup_mse33752_2019_04_28_22_39.tar.gz running: /opt/apache-cassandra-3.9/bin/nodetool --ssl -h cassandra.service.consul -p 7199 clearsnapshot Requested clearing snapshot(s) for [all keyspaces]
At the end of the output, name of the backup archive is specified:
[22:40:07] Done Backup. Created backup file /tmp/cmx_backup_mse33752_2019_04_28_22_39.tar.gz
Backing up a High Availability setup
If the High Availability is currently up and running, since all databases are synced between primary and secondary, taking the backup from primary CMX is enough to save all client data. Simply run cmxos backup --all --online command and transfer the files from the primary server.
If High Availability is not currently established between primary and secondary server, first determine which CMX has complete and most recent data and create a backup from it.
Moving the bundle from CMX to a different machine
If something happens to the hard drive of the CMX or files get corrupted during the upgrade process, backup files saved on the CMX could be lost. It is recommended to move the data from CMX to another machine using Secure Copy Protocol (SCP). Bellow you can find examples on how to do that on Windows, MacOS and Linux PC:
Windows:
Easiest way to do this on Windows is via WinSCP program. After installation, enter the IP address and credentials of the cmxadmin user and establish the SCP connection. Navigate to the folder where the backup is saved, find the backup file and drag it to the desired location on your local machine (left window).
MacOS and Linux:
MacOS and most Linux distributions come with native scp client. Files can be move using a simple terminal command:
scp cmxadmin@<cmx_ip_address>:/<file_path_and_name_on_cmx> <file_path_and_name_on_local_machine>
Example:
VAPEROVI-M-H1YM:~ vaperovi$ scp cmxadmin@10.48.71.41:/tmp/cmx_backup_mse33752_2019_04_28_19_38.tar.gz /Users/vaperovi/cmx_backup_mse33752_2019_04_28_19_38.tar.gz cmxadmin@10.48.71.41's password: cmx_backup_mse33752_2019_04_28_19_38.tar.gz 100% 186KB 1.4MB/s 00:00
CMX shows a prompt to enter the credentials of the cmxadmin user, after which the data is transferred to the specified location in your local machine.
Deleting the backup archive from CMX
On CMX version 10.5.x, files can be deleted by logging in as root user via su command, navigating to the /tmp directory where the backup files has been saved and deleting it via rm -f command:
[cmxadmin@mse33752 ~]$ su Password: [root@mse33752 cmxadmin]# [root@mse33752 cmxadmin]# cd /tmp [root@mse33752 tmp]# rm -f cmx_backup_mse33752_2019_04_28_19_38.tar.gz
Starting version 10.6.0, root access has been restricted. Without special patch that can only be given out by Cisco TAC, deleting the files like on 10.5 is not possible. Some space can be freed up using the cmxos clean normal --delete command:
[cmxadmin@mse33752 ~]$ cmxos clean normal --delete Are you sure you wish to remove files? [y/N]: y Removing files in: /opt/cmx/var/log Remove: /opt/cmx/var/log/entropy.err Remove: /opt/cmx/var/log/backup.log.2 Remove: /opt/cmx/var/log/techsupport/cmx_tech_support_2019-04-28.log Removing files in: /opt/influxdb/shared Removing files in: /tmp
Restoring the backup
If you want to restore the backup, transfer the backup file from the remote machine to CMX. In windows, you can simply drag and drop the files using WinSCP. On MacOS and Linux, use this command:
$ scp <file_path_and_name_on_local_machine> cmxadmin@<cmx_ip_address>:/tmp
Example:
VAPEROVI-M-H1YM:~ vaperovi$ scp /Users/vaperovi/cmx_backup_mse33752_2019_04_28_19_38.tar.gz cmxadmin@10.48.71.41:/tmp cmxadmin@10.48.71.41's password: cmx_backup_mse33752_2019_04_28_19_38_copy.tar.gz 100% 186KB 1.3MB/s 00:00
To restore data, CMX needs to have free disk space 4 times the size of the backup bundle. If there is not enough space, you can try increasing the space of the VM or by running cmxos clean normal --delete command. Restoration process can be initiated using the cmxos restore command. Adding the -i parameter, you can backup only certain elements (database, cache, cassandra, floormaps, licenses, setup, conf). Its recommended to perform full backups.
Restoration process requires all services to be stopped. Make sure to prepare big enough maintenance window for this process as it can take over an hour.
[cmxadmin@mse33752 ~]$ cmxos restore Please enter the backup file path: /tmp/cmx_backup_mse33752_2019_04_28_22_39.tar.gz Please enter the path for untar backup file [/tmp]: Stopping monit (via systemctl): [ OK ] [23:49:19] Preparing for restore... Restore size 30383753 Available disk space in /tmp is 1812541169664 Available disk space is 1817753817088 [23:49:19] Untarring backup file... Backing up existing licenses on the system... Successfully saved existing licenses Stopping all services... Pre restore took: 41.672647953 seconds [23:50:00] Restoring Database... Created temporary database temp_mse Running command /usr/bin/sudo -u postgres pg_restore -j 8 -d temp_mse -Fc /tmp/cmx_backup_mse33752_2019_04_28_22_39/postgres/mse.dump Restored temporary database temp_mse Dropping database mse Renaming database temp_mse to mse Restarting database... Starting database... Restore database took: 10.2765719891 seconds [23:50:11] Restoring Cache... Stopping cache_6378... Restarting cache_6378... Stopping cache_6379... Restarting cache_6379... Stopping cache_6385... Restarting cache_6385... Stopping cache_6380... Restarting cache_6380... Stopping cache_6381... Restarting cache_6381... Stopping cache_6382... Restarting cache_6382... Stopping cache_6383... Restarting cache_6383... Stopping cache_6384... Restarting cache_6384... Restore Cache took: 61.1865711212 seconds [23:51:12] Restoring Cassandra... Stopping Cassandra... Starting Cassandra after wipe... starting cassandra Creating empty cassandra schemas Stopping Cassandra... Starting Cassandra after restore ... starting cassandra Restore Cassandra took: 117.123826981 seconds [23:53:09] Restoring floormaps... Restore floor maps took: 0.0736980438232 seconds [23:53:09] Restoring licenses... Restore licenses took: 0.000176906585693 seconds [23:53:09] Restoring setup... Restore setup took: 0.00758194923401 seconds [23:53:09] Restoring connect images... Restore connect images took: 0.000188827514648 seconds [23:53:09] Running Post Restore Tasks... [23:53:09] Migrating Schemas... [23:53:10] Migrating Cassandra Schemas... stopping cassandra Local licenses wont be retained. Running full vacuum command on postgres Performing cleanup of redis cache 6378 and 6383 to evict bloom filter stale entries. Performing cleanup of redis cache 6378 to evict stale records by qlesspyworker. Update CMX default certificate Post restore took: 61.7358779907 seconds [23:54:11] Starting all services... [23:56:04] Done Starting monit (via systemctl): [ OK ]
Additional information
Restrictions of backup and restoration process
- Backups from CMX 10.3 or older cannot be imported into CMX 10.5.x and later. Backups from 10.5.x can be imported into CMX 10.6.x
- In order to stay compliant with GDPR, backup of database, floor maps, licenses and setup components needs to be performed
- Make sure that the port 22 is not blocked between CMX and the machine which is used to access it
- If restoring backups from a different type of CMX deployment, refer to the table to check if they are compatible:
|
Restore from...
|
Restore to...
|
Recommendations
|
|---|---|---|
|
Same machine specs |
Same machine specs |
OK |
|
Cisco MSE 3365 appliance |
Cisco 3375 appliance |
OK |
|
Cisco MSE 3365 appliance |
High-end MSE virtual (vMSE) |
OK |
|
High-end vMSE |
Cisco 3375 and Cisco MSE 3365 appliances |
OK unless the high-end machine has more RAM allocated than the recommended specs |
|
Standard vMSE |
Cisco 3375 and Cisco MSE 3365 appliances |
OK |
|
Standard vMSE |
High-end vMSE |
OK |
|
Low-end vMSE |
Cisco 3375 and Cisco MSE 3365 appliances |
OK |
|
Low-end vMSE |
High-end vMSE |
OK |
|
Low-end vMSE |
Standard vMSE |
OK |
|
Cisco 3375 appliance |
Cisco MSE 3365 appliance |
Not Recommended |
|
Cisco MSE 3365 appliance |
Standard vMSE |
Not Recommended |
|
Cisco MSE 3365 appliance |
Low-end vMSE |
Not Recommended |
|
High-end vMSE |
Standard vMSE |
Not Recommended |
|
High-end vMSE |
Low-end vMSE |
Not Recommended |
|
Standard vMSE |
Low-end vMSE |
Not Recommended |
Difference between backup and virtual machine snapshot
Virtual Machine snapshots cannot be considered a backup tool as they do not do anything to preserve the integrity of the VMDK file that the Virtual Machine uses for its data storage.
Snapshots operate by "freezing" the original VMDK storage file and creating additional snapshot files which capture the changes made to the original VMDK file (so called disk chain). This way the state of the disk file can be preserved in time and rolled back to if needed after some changes are made.
Therefore, If the original (parent) VMDK file is lost or is corrupted in any way, snapshot data cannot be used to restore it back to its previous state and the stored data is effectively lost.
VMware best practices for using snapshots in the vSphere environment mentions the following:
- Do not use snapshots as backups (reasons outlined)
- VMware recommends not to use a single snapshot for more than 72 hours (The snapshot file continues to grow in size when it is retained for a longer period. This can cause the snapshot storage location to run out of space and impact the system performance).
- Maximum of 32 snapshots are supported in a chain. However, for a better performance, limit the use only 2 to 3 snapshots.
For more information, refer to the VMware snapshot best practices article.
Revision History
| Revision | Publish Date | Comments |
|---|---|---|
2.0 |
20-Oct-2022 |
Added 3375 |
1.0 |
16-May-2019 |
Initial Release |
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)