Troubleshooting Cisco Elastic Services Controller Micro-Services

ESC is designed based on Micro-Service software architecture and there are many components or applications integrated together in a micro-service software architecture to provide vendor-agnostic, customizable and programmable platform.

Overview of Cisco Elastic Services Controller Micro-Services

ESC is based on Micro-Service software architecture which provide various services to its vendors. To ensure that all micro-services are running in healthy state. Following are the methods to check the health condition of ESC's micro-services and the overall service status:

To check ESC's overall status, run the following command in ESC VM or the Active VM of ESC HA.

The first line of the output shows the overall status of ESC service and the following lines indicate the status of each micro-service or component.

# escadm status --v
0 ESC status=0 ESC HA Master Healthy
vimmanager (pgid 6432) is running
monitor (pgid 6450) is running
mona (pgid 6529) is running
drbd (pgid 0) is master
confd (pgid 6656) is running
keepalived (pgid 1374) is running
pgsql (pgid 6760) is running
filesystem (pgid 0) is running
snmp (pgid 6598) is running
escmanager (pgid 6924) is running

Cisco Elastic Services Controller status is Not Healthy

Problem :

In some scenarios, when you check your ESC status, the output of ESC health status displays that ESC is in critical error status and probably one or some of the ESC's component or micro-service are in stopped/dead or not running state. For example:
# escadm status --v
2 ESC status=0 ESC HA Master Critical
vimmanager (pgid 6432) is running
monitor (pgid 6450) is running
mona is stopped
drbd (pgid 0) is master
confd (pgid 6656) is running
keepalived (pgid 1374) is running
pgsql (pgid 6760) is running
filesystem (pgid 0) is running
snmp (pgid 6598) is running
escmanager (pgid 6924) is running

Solution :

Do the following actions to get the ESC service back.

  • Restart ESC service

    Following commands help you to restart ESC services in a Standalone ESC:

    For ESC 2.X:
    $ sudo service esc_service stop
    $ sudo service esc_service status (make sure ESC service is stopped)
    $ sudo service esc_service start
    For ESC 3.X and later releases:
    $ sudo escadm stop
    $ sudo escadm status (make sure ESC service is stopped)
    $ sudo escadm start

    For ESC HA, run the following commands to restart ESC service.

    For ESC 2.X:
    $ sudo service keepalived stop
    $ sudo service keepalived status (make sure ESC service is stopped)
    $ sudo service keepalived start
    For ESC 3.X and later releases:
    $ sudo escadm stop
    $ sudo escadm status (make sure ESC service is stopped)
    $ sudo escadm start

    Note that ESC HA failover will be triggered when you restart the ESC service. The BACKUP VM will switch running as the HA MASTER node after execution of the above mentioned commands. If you do not want to trigger the switchover, the two extra steps mentioned below should be taken care.


    Note

    ESC HA failover triggers when you restart the ESC service. The BACKUP VM switches and starts running as the HA Active VM when you execute the previous commands.


    Use the following steps if you do not want to trigger the switchover:

    Before you execute the service restart commands in Active VM, login to the Backup VM first and run the following commands:

    For ESC 2.X:
    $ sudo service keepalived stop
    $ sudo service keepalived status (make sure ESC service is stopped)
    For ESC 3.X and later releases:
    
    $ sudo escadm stop
    $ sudo escadm status (make sure ESC service is stopped)

    Once you restart the ESC service in the Active VM, log into the Backup VM again and run the following commands:

    For ESC 2.X:
    $ sudo service keepalived start
    
    For ESC 3.X and later releases:
    $ sudo escadm start
    
  • Reboot ESC VM

    If restart ESC service still doesn't help, run the following command to reboot the ESC VM:
    $ sudo reboot
    

Note

ESC HA switchover triggers when you reboot the ESC Active VM. The Backup VM becomes the new Active and start all the services.


  • Check ESC's start up logs.

If ESC service still gets stuck at the startup, check the ESC logs to find out the details. You must verify the following logs files:
  • /var/log/esc/escadm.log -The log of ESC service start/stop to check which micro-service causes the problem.

  • /var/log/esc/escmanager.log -The log of ESCManager about ESCManager service start/stop.

  • /var/log/messages -The OS message logging file also contains fatal startup errors at the system level.

If you cannot find the problem, collect the ESC logs and send the log files from ESC VMs (two VMs in HA) to the technical support team. To collect the logs, use the following command:
$ sudo escadm log collect