Auto Recovery

In Auto recovery, the recovery type parameter is set to Auto. ESC automatically recovers the VM with the specified <action-on-recovery> value in the recovery policy. The recovery type is auto by default if the user does not choose a recovery type.

<recovery_policy>
 <recovery_type>AUTO</recovery_type>
 <action_on_recovery>REBOOT_THEN_REDEPLOY</action_on_recovery>
 <max_retries>3</max_retries>
</recovery_policy>

Manual Recovery

Manual Recovery of a VM

In manual recovery, ESC sends the VM_MANUAL_RECOVERY_NEEDED notification to northbound (NB) and waits for the instruction from NB for recovery. ESC performs recovery when it receives recovery instruction from NB. Further, the recovery action is based on the action-on-recovery parameter in the recovery policy. For manual recovery of a complete deployment, see Manual Recovery of a Deployment

The manual recovery policy datamodel is as follows:

<vm_group>
......
 <recovery_policy>
 <recovery_type>MANUAL</recovery_type>
 <action_on_recovery>REBOOT_THEN_REDEPLOY</action_on_recovery>
 <max_retries>3</max_retries>
 </recovery_policy>
</vm_group>

For more information about recovery policy parameters in the datamodel, see Elastic Services Controller Deployment Attributes. For more information about configuring the recovery policy in the ESC Portal (VMware only), see the Deploying VNFs on VMware vCenter using ESC Portal.

The VM_MANUAL_RECOVERY_NEEDED notification is as follows:

===== SEND NOTIFICATION STARTS =====
 WARN  Type: VM_MANUAL_RECOVERY_NEEDED
 WARN  Status: SUCCESS
 WARN  Status Code: 200
 WARN  Status Msg: Recovery event for VM [manual-recover_error-g1_0_7d96ad0b-4f27-4a5a-bdf7-ec830e93d07e] triggered.
 WARN  Tenant: manual-recovery-tenant
 WARN  Service ID: NULL
 WARN  Deployment ID: 08491863-846a-4294-b305-c0002b9e8daf
 WARN  Deployment name: dep-error
 WARN  VM group name: error-g1
 WARN  VM Source:
 WARN      VM ID: ffea079d-0ea2-4d47-ba31-26a08e6dff22
 WARN      Host ID: 3a5351dc4bb7df0ee25e238a8ebbd6c6fcdf225aebcb9dff6ba10249
 WARN      Host Name: my-server-27
 WARN      [DEBUG-ONLY] VM IP: 192.168.0.3;
 WARN  =====  SEND NOTIFICATION ENDS  =====

APIs for Manual Recovery of a VM

You can perform manual recovery using the Confd and Rest APIs. The manual recovery request can be configured to override the predefined recovery action to any desired action.

• Netconf API recovery-vm-action DO generated vm name [xmlfile]

  To perform recovery using the API, login to esc_nc_cli and run the following command:

  $ esc_nc_cli --user <username> --password <password> recovery-vm-action DO [xmlfile]

  The recovery is performed and the recovery notification is sent to NB.

  Note

  Recovery (recovery-vm-action DO <VM-NAME>) can be performed after the VM is alive and the service is active. If the deployment is incomplete, it must be completed before performing recovery. If a failover happens during a configurable manual recovery, the manual recovery resumes with predefined recover action. The migration of any deployment must always use default recovery policy. You must not provide recovery action for VM/VNF manual recovery in an LCS based recovery.You must not use enable monitor and configurable manual recovery options together.

• REST API

  http://ip:8080/ESCAPI/#!/Recovery_VM_Operations/handleOperation

  POST /v0/{internal_tenant_id}/deployments/recovery-vm/{vm_name}

  Recovery VM operation payload:

  
  {
  "operation":"recovery_do",
  "properties":{
  "property":[
  {
  "name":"action",
  "value":"REDEPLOY_ONLY"
  }
  ]
  }
  }

Supported VM States and Service Combinations for Manual Recovery of a VM

The API, recovery-vm-action, applies to both auto and manual recovery types, but only under certain VM states and services. The following table shows the details. In general, during deployment, service update, undeployment and recovery, the manual recovery action is rejected by ESC.

Manual Recovery of a Deployment

ESC supports manual recovery of VMs at the service level, that is, recovery of a complete deployment. After the successful deployment of a service, the service may move into an error state because of failed VMs. ESC can manually recover these failed VMs, or the complete deployment through a deployment recovery request. For manual recovery of a single VM, see Manual Recovery.

APIs for Manual Recovery of a Deployment

You can perform manual recovery using the NETCONF and REST APIs.

The manual recovery request can be configured to override the predefined recovery action to any desired action.

Note

There is no service active notification after the deployment recovery. You must run a query, for example, esc_nc_cli --user <username> --password <password> get esc_datamodel to see if the service state of the deployment is active or not. If a failover happens during a configurable manual recovery, the manual recovery resumes with predefined recover action. The migration of any deployment must always use default recovery policy. You must not provide recovery action for VM/VNF manual recovery in an LCS based recovery.You must not use enable monitor and configurable manual recovery options together.

• NETCONF API svc-action RECOVER tenant-name deployment-name [xmlfile]

  To perform recovery using the API, login to esc_nc_cli.

• REST API

  
  POST /v0/{internal_tenant_id}/deployments/service/{internal_deployment_id}
  Content-Type: application/xml
  Accept: application/json
  Callback: http://172.16.0.1:9010/
  Callback-ESC-Events: http://172.16.0.1:9010/
  <service_operation xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
     <operation>recover</operation>
  </service_operation>
  

  where,

  internal_tenant_id—is the system admin tenant ID or the tenant name.

  internal_deployment_id—is the deployment name.

Supported VM States and Service Combinations for Manual Recovery of a Deployment

The API, svc-action RECOVER, applies to both auto and manual recovery types, but only under certain VM states and services. The following table shows the details. In general, during deployment, service update, undeployment and recovery, the manual recovery action is rejected by ESC.

Note	ESC accepts VM level recovery request when the service is in active or error state. Notifications are not sent to NB if all VMs are in the ALIVE state after a service recovery request.

During manual recovery, you can recover a VM depending on its monitoring parameters. If the VM is in error state, set the monitoring parameters to bring back the VM in error state to live state. If the VM is recovered, then ESC sends a RECOVERY_CANCELLED notification. If the VM does not come back live, then the recovery process is triggered. See Manual Recovery for more details.

svc-action SET_MONITOR_AND_RECOVER <tenant-name> <dep-name>

Recovery notification:

===== SEND NOTIFICATION STARTS =====
 WARN  Type: VM_RECOVERY_INIT
 WARN  Status: SUCCESS
 WARN  Status Code: 200
 WARN  Status Msg: Recovery with enabling monitor first event for VM Generated ID [dep-resource_g1_0_74132737-d0a4-4ef0-bd9e-86465c1017bf] triggered.

Note	Recovery enabled with monitoring parameters is for manual recovery at service level only.

The monitor_on_error parameter enables continuous monitoring of the VMs in error state.

			<recovery_policy>
                            <recovery_type>AUTO</recovery_type>
                            <action_on_recovery>REBOOT_ONLY</action_on_recovery>
                            <max_retries>1</max_retries>
                            <monitor_on_error>true</monitor_on_error>
                        </recovery_policy>

The default value is false.

if false, monitoring is unset on the vm in error state.

If true, monitoring is set on the vm in error state. If any VM Alive event occurs later ( after VM_RECOVERY_COMPLETE), the VM is moved back to alive state.

Supported Conditions and Predefined Action Combinations

The following table describes the supported LCS conditions and its actions for recovery and redeployment policies using the policy framework. For more details on the policy driven framework, see Recovery and Redeployment Policies.

Supported Lifecycle Stages (LCS)

Supported Predefined actions

Limiting the Number of Redeployments

Cisco Elastic Services Controller (ESC) limits the number of redeployments using the following parameters:

• max_redep: limits the maximum number of redeployments. By default, the max_redep value is -1, which indicates that there is no limit on the maximum number of redeployments. You can change this value using the bootvm.py arguments or REST API.

• redep_count: consists of the current number of redeployments. The redep_count automatically increases by 1 after a redeployment, irrespective of the success or failure of the redeployment.

  Note	The redeployment limit is for, redeployments triggered by REDEPLOY_ALL::DISABLE_HOST policy. deployments with single VIM configuration only.

Cisco Elastic Services Controller (ESC) performs redeployment,

• if the maximum number of redeployments is set to the default value of -1, that is max_redep = -1.

• if the current number of redeployments is less than the maximum number of redeployments (redep_count < max_redep), then ESC performs redeployment, and increases the redeployment count by 1 after the redeployment is complete.

ESC does not perform any redeployment if the redeployment count is more than or equal to the maximum number of redeployments, (redep_count >= max_redep).

You can use the bootvm.py parameters and REST APIs to configure the values.

Using the bootvm.py parameters

Specify the max_redep value in the esc_params.conf file that contains the following line: default.max_redep = 3

Run the command, bootvm.py ... --esc_params_file <path_to_file>/esc_params.conf ...

Using the REST APIs

You can retrieve, and reset the redep_count parameter using the following APIs:

• To retrieve the current value of redep_count:

  GET http://<ESC IP>:8080/ESCManager/v0/systemstate/redep_count

• To reset redep_count:

  POST http://<ESC IP>:8080/ESCManager/v0/systemstate/redep_count/reset

You can also use the REST API to retrieve and change the max_redep value.

• To retrieve the current value of max_redep:

  GET http://<ESC IP>:8080/ESCManager/v0/config/default/max_redep

• To change the max_redep value:

  PUT http://<ESC IP>:8080/ESCManager/v0/config/default/max_redep/<value>

  where <value> can be,

  -1, which is the default value with no limit

  0, which does not allow any redeployment

  more than zero (> 0), which specifies the maximum number of redeployments allowed.

You can also use the escadm tool to configure these values. For more information on the escadm tool, see the Elastics Services Controller Install and Upgrade Guide.

For more details on the redeployment policy, see Redeployment Policy.

The VMs that are not redeployed because of the redeployment limit are moved to error state. ESC manually recovers these VMs in error state by enabling the monitoring operation on each VM.

To enable monitoring operation on a single VM in error state:

POST http://<ESC IP>:8080/ESCManager/v0/<internal-tenant-id>/deployments/vm/<vm-name> { "operation" : "enable_monitoring" }

You can also enable monitoring using the esc_nc_cli command:

As part of the manual recovery process, the enable monitoring operation moves the VMs from error state to alive state. If manual recovery fails for these VMs, then auto recovery is triggered.

To enable the monitoring operation on VMs (in error state) in a deployment:

POST http://<ESC IP>:8080/ESCManager/v0/<internal-tenant-id>/deployments/service/<internal-deployment-id> { "operation" : "enable_monitoring" }

You can also enable monitoring using the esc_nc_cli command:

As part of the manual recovery process, the enable monitoring operation moves all the VMs in a deployment from error state to alive state. If manual recovery fails, then auto recovery is triggered on all the VMs in the deployment.

For more information, see Monitoring Operations and Manual Recovery of VMs.