Ultra M Troubleshooting

Ultra M Component Reference Documentation

The following sections provide links to troubleshooting information for the various components that comprise the Ultra M solution.

Catalyst 2960 Switch

UAS

Refer to the USP Deployment Automation Guide.

UGP

Refer to the Ultra Gateway Platform System Administration Guide.

Collecting Support Information

From Host/Server/Compute/Controler/Linux:

  • Identify if Passthrought/SR-IOV is enabled.

  • Run sosreport:


    Note
    This functionality is enabled by default on Red Hat, but not on Ubuntu. It is recommended that you enable sysstat and sosreport on Ubuntu (run apt-get install sysstat and apt-get install sosreport ). It is also recommended that you install sysstat on Red Hat (run yum install sysstat ).

  • Get and run the os_ssd_pac script from Cisco:

    • Compute (all):

      
./os_ssd_pac.sh -a 
./os_ssd_pac.sh -k -s

      Note
      For initial collection, it is always recommended to include the -s option (sosreport). Run ./os_ssd_pac.sh -h for more information.

    • Controller (all):

      
./os_ssd_pac.sh -f 
./os_ssd_pac.sh -c -s

      Note
      For initial collection it is always recommended to include the -s option (sosreport). Run ./os_ssd_pac.sh -h for more information.

  • For monitoring purposes, from crontab use option: -m ( for example run every 5 or 10 minutes)

From Switches

From all switches connected to the Host/Servers. (This also includes other switches which have same vlans terminated on the Host/Servers.) 


    show tech-support 
    syslogs 
    snmp traps

Note
It is recommended that mac-move notifications are enabled on all switches in network by running mac address-table notification mac-move.

From ESC (Active and Standby)


Note
It is recommended that you take a backup of the software and data before performing any of the following operations. Backups can be taken by executing opt/cisco/esc/esc-scripts/esc_dbtool.py backup. (Refer to https://www.cisco.com/c/en/us/td/docs/net_mgmt/elastic_services_controller/2-3/user/guide/Cisco-Elastic-Services-Controller-User-Guide-2-3/Cisco-Elastic-Services-Controller-User-Guide-2-2_chapter_010010.html#id_18936 for more information.) 

/opt/cisco/esc/esc-scripts/health.sh 
/usr/bin/collect_esc_log.sh 
./os_ssd_pac -a

From UAS

  • Monitor ConfD:

    
confd -status 
confd --debug-dump /tmp/confd_debug-dump 
confd --printlog /tmp/confd_debug-dump

    Note
    Once the file /tmp/confd_debug-dump> is collected, it can be removed (rm /tmp/confd_debug-dump).

  • Monitor UAS Components:

    
source /opt/cisco/usp/uas/confd-6.1/confdrc 
confd_cli -u admin -C 
show uas 
show uas ha-vip  
show uas state 
show confd-state 
show running-config 
show transactions date-and-time 
show log | display xml 
show errors displaylevel 64 
show notification stream uas_notify last 1000 
show autovnf-oper:vnfm 
show autovnf-oper:vnf-em 
show autovnf-oper:vdu-catalog 
show autovnf-oper:transactions 
show autovnf-oper:network-catalog 
show autovnf-oper:errors 
show usp 
show confd-state internal callpoints 
show confd-state webui listen 
show netconf-state

    Important

    Executing the confd_cli -u admin -C command prompts you to enter admin user password.

  • Monitor Zookeeper:

    
/opt/cisco/usp/packages/zookeeper/current/bin/zkCli.sh  -server x.x.x.x:2181  ls /config/control-function 
/opt/cisco/usp/packages/zookeeper/current/bin/zkCli.sh  -server x.x.x.x:2181  ls /config/element-manager 
/opt/cisco/usp/packages/zookeeper/current/bin/zkCli.sh  -server x.x.x.x:2181  ls /config/session-function 
/opt/cisco/usp/packages/zookeeper/current/bin/zkCli.sh  -server x.x.x.x:2181  ls / 
/opt/cisco/usp/packages/zookeeper/current/bin/zkCli.sh  -server x.x.x.x:2181  ls /stat 
/opt/cisco/usp/packages/zookeeper/current/bin/zkCli.sh  -server x.x.x.x:2181  ls /log

    Note

    Port number (2181) is not mandatory when executing the zkCli.sh command.

  • Collect Zookeeper data:

    
cd /tmp 
tar zcfv zookeeper_data.tgz /var/lib/zookeeper/data/version-2/ 
ls -las /tmp/zookeeper_data.tgz

  • Get support details

    
./os_ssd_pac -a

From UEM (Active and Standby)

  • Collect logs

    
/opt/cisco/em-scripts/collect-em-logs.sh

  • Monitor NCS:

    
ncs -status 
ncs --debug-dump /tmp/ncs_debug-dump 
ncs --printlog /tmp/ncs_debug-dump

    Note
    Once the file /tmp/ncs_debug-dump is collected, it can be removed (rm /tmp/ncs_debug-dump ).

  • Collect support details:

    
./os_ssd_pac -a

From UGP (Through StarOS)

  • Collect the multiple outputs of the show support details.


    Note
    It is recommended to collect at least two samples, 60 minutes apart if possible.

  • Collect raw bulkstats before and after events.

  • Collect syslogs and snmp traps before and after events.

  • Collect PCAP or sniffer traces of all relevant interfaces if possible.


    Note
    Familiarize yourself with how running SPAN/RSPAN on Nexus and Catalyst switches. This is important for resolving Passthrough/SR-IOV issues.

  • Collect console outputs from all nodes.

  • Export CDRs and EDRs.

  • Collect the outputs of monitor subscriber next-call or monitor protocol depending on the activity

  • Refer to https://supportforums.cisco.com/sites/default/files/cisco_asr5000_asr5500_troubleshooting_guide.pdf for more information.

About Ultra M Manager Log Files

All Ultra M Manager log files are created under “/var/log/cisco/ultram-manager”.


cd /var/log/cisco/ultram-manager 
ls -alrt

Example output:


total 116
drwxr-xr-x. 3 root root  4096 Sep 10 17:41 ..
-rw-r--r--. 1 root root     0 Sep 12 15:15 ultram_health_snmp.log
-rw-r--r--. 1 root root   448 Sep 12 15:16 ultram_health_uas.report
-rw-r--r--. 1 root root   188 Sep 12 15:16 ultram_health_uas.error
-rw-r--r--. 1 root root   580 Sep 12 15:16 ultram_health_uas.log
-rw-r--r--. 1 root root 24093 Sep 12 15:16 ultram_health_ucs.log
-rw-r--r--. 1 root root  8302 Sep 12 15:16 ultram_health_os.error
drwxr-xr-x. 2 root root  4096 Sep 12 15:16 .
-rw-r--r--. 1 root root 51077 Sep 12 15:16 ultram_health_os.report
-rw-r--r--. 1 root root  6677 Sep 12 15:16 ultram_health_os.log

NOTES:

  • The files are named according to the following conventions:

    • ultram_health_os: Contain information related to OpenStack

    • ultram_health_ucs: Contain information related to UCS

    • ultram_health_uas: Contain information related to UAS

  • Files with the “*.log” extension contain debug/error outputs from different components. These files get added to over time and contain useful data for debugging in case of issues.

  • Files with the “.report” extension contain the current report. These files get created on every tun.

  • Files with the “.error” extension contain actual data received from the nodes as part of health monitoring. These are the events that causes the Ultra M health monitor to send traps out. These files are updated every time a component generates an event.