ESC in Maintenance Mode

This chapter contains the following chapters:

Setting ESC in a Maintenance Mode

ESC must be put to maintenance mode to backup and restore ESC database. To do so, use the escadm tool as specified in the below section.

Before you begin


Note

From ESC release 4.4, ESC continues to be in maintenance mode after HA Active/Standby switch over or DB restoration, if ESC was in maintenance mode before HA Active/Standby switch over or DB restoration.

During maintenance mode,

  • Northbound requests are blocked by ESC and ESC responds with maintenance mode notification.

  • Only REST requests receive response that ESC is unavailable temporarily. ConfD requests get the maintenance mode rejection message, or an OK message for all idempotent request such as create tenant request when the tenant already exists.

  • Monitoring actions are paused.

  • All ongoing requests and transactions continue to progress.

Using the escadm Tool

ESC can be put to maintenance mode using the escadm tool.

Procedure

Step 1

Put ESC to maintenance mode from the VM shell: 

sudo escadm op_mode set --mode=maintenance
Set mode to MAINTENANCE
Operation Mode = MAINTENANCE
For releases before ESC 5.6 release if you have D-MONA deployed, do the following to each D-MONA:
curl -i -k --user {REST username}:{REST password} -X POST https:/{DMONA IP}:8443/mona/v1/rules/pause_all
Note 

Starting from ESC 5.6 release, when ESC is set to maintenance mode the corresponding D-MONA instances will also be set to maintenence mode.

If D-MONA is not reachable while setting ESC to maintenance mode, you will receive the following warning:

Set mode to MAINTENANCE
Operation Mode = MAINTENANCE
Warning: There is at least one suspend monitoring operation failure.
Please check the logs for more details.
Step 2

To query operation mode at any time, 

sudo escadm op_mode show

Example:

Operation Mode = OPERATION
Step 3

Set maintenance mode when there is no in-flight transaction. Using the ipt_check flag with the escadm tool, you can choose to set ESC in the maintenance mode only if there are no ongoing transactions in ESC. Set the flag to true, if you do not want ESC to set in the maintenance mode, if there are ongoing transactions in ESC. 

sudo escadm op_mode set --mode=maintenace --ipt_check=true

With the ipt_check option set to true, escadm tool checks if there is any on going operation, if so, the escadm tool will not set ESC to maintenance mode.

Note 

ESC with D-MONA deployment, requires to do extra calls when going to maintenance mode.

Setting ESC in an Operation Mode

Put ESC in operation mode using the escadm tool: 
sudo escadm op_mode set --mode=operation
Response is as follows: 
Set mode to OPERATION
Operation Mode = OPERATION
For releases before ESC 5.6 release if you have D-MONA deployed, do the following to each D-MONA:
curl -i -k --user {REST username}:{REST password} -X POST https:/{DMONA IP}:8443/mona/v1/rules/resume_all

Note

Starting from ESC 5.6 release, when ESC is set to operational mode the corresponding D-MONA instances will also be set to Operational mode.

If D-MONA is not reachable while setting ESC to operation mode, you will receive the following warning:

Set mode to OPERATION
Operation Mode = OPERATION
Warning: There is at least one resume monitoring operation failure.
Please check the logs for more details.
Verify ESC's operation mode at any time using the following command: 
sudo escadm op_mode show

Backup the Database from the ESC Standalone Instances

  • The following assumptions should be taken into consideration:

    • A third machine is required to store the database and log backups.

    • ESC does not support database schema downgrade. Restoring database to the older ESC version could cause unexpected problems.

  • Before you start the backup process, ensure you have an external storage space (could be in the OpenStack controller or any system accessible by ESC). The backup/restore could be expressed in a generic format which will be used by the escadm tool: scp://<username>:<password>@<backup_ip>:<filename> . In this format, the credentials, IP address and file storage path of the third machine are required. You may also use localhost IP as the backup IP to backup database in a location of ESC VM and then copy the files to the external storage

To backup the ESC database from a standalone ESC or a HA Active/Standby (active node):

Procedure

Step 1

Log in to ESC VM and set it to maintenance mode, run: 

$ sudo escadm op_mode set --mode=maintenance
Step 2

To make sure ESC is in maintenance mode, run: 

$ sudo escadm op_mode show
Step 3

Backup database. Execute the commands below as a root user: 


# sudo escadm backup --file /tmp/db_file_name.tar.bz2
scp://<username>:<password>@<backup_vm_ip>:<filename>
Step 4

To put ESC back to operation mode, run: 


$ sudo escadm op_mode set --mode=operation
$ sudo escadm op_mode show
Step 5

Collect all the logs from the old ESC VM and back it up. Execute the below command as a root user. 


# sudo escadm log collect

A timestamped log file will be generated in: /var/tmp/esc_log <timestamp>.tar.bz2

Note 

If a dynamic mapping file is used by ESC service, the dynamic mapping file should be backed up at the same time with ESC logs. The default path of the dynamic mapping file is /opt/cisco/esc/esc-dynamic-mapping/dynamic_mappings.xml.

Step 6

After a successful database back-up, shut down the old ESC VM using Horizon/Kilo or Nova commands. For ESC VM instances based in VMware vSphere, shutdown the active instance through VMware client dashboard. An example of shutting down a VM in OpenStack is shown below : 

 $ nova stop OLD_ESC_ID
Step 7

Detach the old port from the old VM and rename the old ESC node. Examples of detaching and renaming the VM in OpenStack is shown below: 


nova interface-detach ESC_NAME port-id-of-ESC_NAME
nova rename ESC_NAME ESC_NAME.old

In VMWare, assign a different IP address to the old VM and then rename the old VM.

Backup the Database from the ESC HA Active/Standby Instances

  • The following assumptions should be taken into consideration:

    • A third machine is required to store the database and log backups.

    • ESC does not support database schema downgrade. Restoring database to the older ESC version could cause unexpected problems.

  • Before you start the backup process, ensure you have an external storage space available (could be in the OpenStack controller or any system accessable by ESC). The backup/restore could be expressed in a generic format which will be used by the escadm tool: scp://<username>:<password>@<backup_ip>:<filename> . In this format, the credentials, IP address and file storage path of the third machine are required. You may also use localhost IP as the backup IP to backup database in a location of ESC VM and then copy the files to the external storage.

To backup the ESC database from a standalone ESC or a HA Active/Standby (active node):

Procedure

Step 1

Perform the following steps on the Standby ESC node.

  1. Connect to the standby ESC instance using SSH: 

    $ ssh <username>@<standby_vm_ip>

  2. Verify that the ESC instance is standby and note the name of the standby ESC HA Active/Standby instance: 

    $ sudo escadm status --v

    If the output value shows "STANDBY", the node is the standby ESC node.

  3. Change access to an admin user. 

    sudo bash

  4. Collect all the logs from the standby ESC VM and back it up. 

    
$ sudo escadm log collect

    A timestamped log file will be generated in: /var/tmp/esc_log <timestamp>.tar.bz2

  5. Shutdown the standby ESC instance through OpenStack Kilo/Horizon using Nova command or VMware client. An example of shutting down the VM on OpenStack is shown below: 

    
$ nova stop OLD_ESC_STANDBY_ID
Step 2

Perform the following steps on the Active ESC node.

  1. Connect to the active ESC instance using SSH: 

    $ ssh <username>@<active_vm_ip>

  2. Change access to an admin user. 

    $ sudo bash

  3. Verify that the ESC instance is Active and note the name of the Active ESC HA Active/Standby instance 

    
$ sudo escadm status --v

    If the output value shows "ACTIVE", the node is the active ESC node.

  4. Back up the database files from the active node of ESC HA Active/Standby: 

    
$ sudo escadm backup --file /tmp/db_file_name.tar.bz2
scp://<username>:<password>@<standby_vm_ip>:<filename>

  5. Collect the logs from the active ESC VM and back it up. 

    
$ sudo escadm log collect

    A timestamped log file will be generated in: /var/tmp/esc_log <timestamp>.tar.bz2

    Note 

    If a dynamic mapping file is used by ESC service, the dynamic mapping file should be backed up at the same time with ESC logs. The default path of the dynamic mapping file is /opt/cisco/esc/esc-dynamic-mapping/dynamic_mappings.xml .

Step 3

Shutdown the active ESC instance through OpenStack Kilo/Horizon using Nova command. For ESC VM instances based in VMware vSphere, shutdown the active instance through VMware client dashboard. An example of shutting down the VM on OpenStack is shown below: 

$ nova stop OLD_ESC_ACTIVE

Use the nova list command or the nova show OLD_ESC_STANDBY command, to verify if the ESC HA Active/Standby instances have been successfully shut down.

Step 4

(Only for OpenStack) Detach the port from the old ESC VM and rename the old VM.

If upgraded ESC VM needs to operate with same IP addresses and same instance names as old instances, detach the ports from each instance, shutdown the old ESC VMs and then rename the old ESC instances.

If you intend to use old VMware active instance, assign a different IP address and rename the VM name. If not, you can delete the old VM and use the same IP address for the new upgraded VMware active instance. After deleting the old VM, you can continue with the old instance name and the IP address.

OpenStack commands for detaching the ports and renaming the old VMs are shown below: 


nova interface-list ESC_NAME
nova interface-detach ESC_NAME port-id-of-ESC_NAME
nova rename ESC_NAME ESC_NAME.old

Restoring ESC Database

Before you begin

To restore the database,

  • In standalone ESC instance, stop ESC services. Run # sudo escadm stop.

  • In HA Active/Standby type instances, stop escadm on the standby first, and later on the Active ESC HA Active/Standby instance. Run # sudo escadm stop.

  • All the services must be stopped. To check the status, run # sudo escadm status --v.

Procedure

Step 1

Restore the database. Execute the following command from the ESC VM: 


$ scp <username>@<server_ip>:/path/db.tar.bz2     /tmp/
$ sudo escadm restore –file /tmp/db.tar.bz2
Step 2

Enter the ESC password in the URL, or manually enter it after executing the above command.

Step 3

Restart the ESC service to complete the database restore by running the following command: 

$ sudo escadm restart
Note 
ESC maintenance mode blocks the northbound request and VNF monitoring. However, if there are some ongoing transactions because of northbound requests before ESC entering maintenance mode, those transactions may have following restriction with backup and restore:

  • ESC reports an error for the deployment, network creation, and subnet creation requests, if these transactions are interrupted by backup and restore. Northbound handles these error messages but it may cause network or subnet leakage in some cases (For example, ESC is interrupted before getting the UUID from OpenStack).

  • ESC reports an error for service chain upgrade, and requires service chain undeployment and deployment (rather than downgrade and upgrade) to re-create the service.