Troubleshooting Cisco Elastic Services Controller Upgrades

Troubleshooting Cisco Elastic Services Controller Upgrades

Problem Statement:

How can I choose ESC upgrade approach?

Description:

There are three upgrade approaches for ESC Upgrade and following are the criteria for your reference to choose the best approach for upgrading your ESC setup.

Solution:

  • If your upgrade path is between ESC patch builds, for example, from ESC 3.1.0.116 to ESC 3.1.0.145, consider RPM upgrade as your first choice.

  • If your upgrade path is between ESC official FCS release, for example, from ESC 3.0.0 to ESC 3.1.0, RPM upgrade will not work in your situation and you can only do the image upgrade.

    • For a healthy ESC HA, consider in-service upgrade.

    • For Standalone ESC or ESC HA with only Master VM, take the database backup/restore approach.

Rolling Back Cisco Elastic Services Controller Upgrade

Problem Statement:

Upgrade does not go well, and you want to roll back.

Description:

There are three upgrade approaches for ESC Upgrade and following are the criteria for your reference to select the best approach for upgrading your ESC setup.

Solution:

The following are the two important points for ESC installation and upgrade:

  • Once you have successfully installed ESC or ESC HA, keep the complete bootvm.py command line in your record. You can use the bootvm.py command line next time when you take an image upgrade approach to perform the ESC upgrade.

  • Ensure you take the backup of ESC database while doing ESC upgrade. Before backing up the database, stop ESC services and wait until the services are in stop state.

    Example:

    #ESC backup DB ESC 2.X:
$ sudo /opt/cisco/esc/esc-scripts/esc_dbtool.py backup --file scp://<dest_user>:<dest_password>@<dest_host>:/tmp/db.tar.bz2
 
#ESC backup DB ESC 3.X and up:
sudo escadm backup --file /tmp/db.tar.bz
scp /tmp/db.tar.bz <dest_user>@<dest_host>:/tmp/db.tar.bz

  • Once you successfully take the backup of the database, it is easy to roll back the upgrade of ESC, if any problem arises during or after the ESC upgrade procedure.

If you have the bootvm.py commands in record with the database backup file, you can take the following approaches to roll back your ESC upgrade:

  • If you are performing the in-service upgrade and have not deleted the old Primary ESC VM (or just renamed the old Primary ESC VM), you can delete the upgraded VM instances first and then turn on the old (renamed) Primary ESC. In this case, the database is retained with no changes in the old Primary VM. Re-install the Backup ESC VM using the complete bootvm.py command line from your records with the old released image. The redeployed Backup VM automatically syncs its database from the Primary VM. No database restore is required.

  • If you have deleted both the older version of Primary and Backup ESC VMs, or you are performing a DB backup/restore upgrade for a standalone ESC, redeploy the ESC VM using the complete bootvm.py command line from your records with the old release image. Then follow the database restore guide to restore the database for the rollback. Before restoring the DB, stop ESC services and wait till the services are in a stop state.

    Example:
    #ESC Restore DB ESC 2.X:
$ sudo /opt/cisco/esc/esc-scripts/esc_dbtool.py restore --file scp://<dest_user>:<dest_password>@<dest_host>:/tmp/db.tar.bz2
 
#ESC Restore DB ESC 3.X and up:
scp <src_user>@<src_host>:/tmp/db.tar.bz /tmp/db.tar.bz
sudo escadm restore --file /tmp/db.tar.bz