Troubleshooting Cisco EPOM

This chapter contains Cisco EPOM specific troubleshooting procedures. For information on troubleshooting the Cisco BTS 10200 Softswitch, refer to the Cisco BTS 10200 Softswitch Operations Manual.

Problem

Recreating the Cisco EPOM database if needed (for example, if you forget the admin password and cannot access the database)

Problems in Cisco EPOM communicating to the Cisco BTS EMS server, such as the one in the message below

Could not retrieve object attributes for object name. The most possible reason is failure to log into the Cisco BTS EMS server or CORBA agent on Cisco EMS server is not working. Please make sure hostname/login/password/siteid is correct. Also check the log file

Troubleshooting Action

Reinitialize the Cisco EPOM databases: 

/opt/CSCOepom/mysql/install/bin/install/ 
MySQLDB -ifs

/opt/CSCOepom/mysql/install/bin/install/EPOMDB -ifs

•View log files: Log files are stored in /var/opt/CSCOepom/logs

Use log files for debugging Cisco EPOM problems or for supplying information to Cisco TAC.

•Check the information in the Cisco BTS EMS server definition to make sure it is correct. See the "To Determine a BTS EMS Server Site ID" section on page 1-11.

•Check connectivity between Cisco EPOM and the Cisco BTS EMS (if they are not co-resident): Log in to the Cisco EPOM server and ping the Cisco BTS EMS server.

•Verify that the correct CORBA adapter has been installed on the Cisco BTS EMS server.

•Check the log files.

•View log files: Log files are stored in:

/var/$EPOM_INST_DIR/logs,
where $EPOM_INST_DIR is the Cisco EPOM installation directory.

By default log directory is /var/opt/CSCOepom/logs

Most important log file is trace.log, current activity can be seen by logging onto Cisco EPOM server and running the command:

tail -f /var/$EPOM_INST_DIR/logs/trace.log

•View Cisco BTS 10200 CIS log file, /opt/ems/log/CIS.log. Current activity can be seen by logging onto Cisco BTS server and running command:

tail -f /opt/ems/log/CIS.log

•Check whether BTScis package is installed on Cisco BTS EMS server. Log onto Cisco BTS EMS server and running command:

pkginfo BTScis

•Check whether cis and ins services are running on Cisco BTS EMS server. Log onto Cisco BTS EMS server and invoke commands:

ps -ef | grep cis

ps-ef |grep ins

•Check whether ports 683 and 14001 are in LISTEN state on the Cisco BTS EMS server. Log onto Cisco BTS EMS server and invoke commands:

netstat -an |grep 683

netstat -an |grep 14001

•Check whether ports 683 and 14001 on Cisco EPOM server are in ESTABLISHED state with the Cisco BTS EMS server. Log onto Cisco EPOM server and invoke commands:

netstat -an |grep 683

netstat -an |grep 14001

•Check connectivity between Cisco EPOM server and Cisco BTS EMS server. Log onto Cisco EPOM server and ping Cisco BTS EMS server from Cisco EPOM server.

Check whether Cisco EPOM server can send packets to Cisco BTS EMS server at ports 683 and 14001. Log onto Cisco BTS EMS server and invoke commands:
telnet bts_server 683

telnet bts_server 14001

•Check whether Cisco EPOM server can resolve the hostname of the Cisco BTS EMS server. As a workaround insert hostname entries into the /etc/hosts file of Cisco EPOM server.

•Log onto both Cisco EPOM server and Cisco BTS EMS server and use snoop command to verify that sizeable amount of data is exchanged between the 2 servers on port 683 and 14001.

•Verify that the hostname of the Cisco BTS EMS server is same as the name used by the Cisco BTS EMS server for its ins service. Log onto Cisco BTS EMS server and invoke following command:

ps -ef | grep ins

Output would also contain the hostname to which ins service is bound.

•Use IP address of the Cisco BTS EMS server, while adding to Cisco EPOM server.

•Use IP address of the Cisco BTS EMS server to initialize ins and cis services on the Cisco BTS EMS server. This makes it for Cisco EPOM Corba client to locate the Cisco BTS EMS server Corba server naming context.

•If this happens too often for a Cisco BTS EMS Server that was accessible some time before, it means that the particular Cisco BTS EMS Server has been restarted or the CIS services have been restarted. In this case even Cisco EPOM needs to be restarted. In general a Cisco BTS EMS server reboot is not done very often, so chances of such issues are minimal.

Problem

Cisco EPOM BTS Command Navigation Tree, problems. Tree appears without commands or JavaScript is rendered on the page.

Troubleshooting Action

Cisco EPOM has introduced a new Clear Tree button, click on this button to re-build Cisco EPOM BTS Command Navigation Tree. This tree is otherwise cached and without clicking the Clear Tree buttons, invalid tree is displayed.


Problem

Cisco EPOM logs not being renewed.

Troubleshooting Action

•Check disk usage of /var directory on Cisco EPOM server. Log onto Cisco EPOM server and invoke command:

df -k |grep var

•The logs with older date stamps can be safely archived in some other directory.

•The logs can be moved to another directory that has more disk space than /var directory e.g /opt , but that needs restart of Cisco EPOM server. As an example if /opt/CSCOepom is the install directory, then the logs can be moved from their current default location /var/opt/CSCOepom/logs to /opt/CSCOepom/logs. Log onto Cisco EPOM Server and invoke following commands:

cd /var/opt/CSCOepom

mv logs /opt/CSCOepom/logs

ln -s /opt/CSCOepom/logs logs

/opt/CSCOepom/bin/epom stop

/opt/CSCOepom/bin/epom start

Problem

Cisco EPOM takes time to build its Corba Cache.

Troubleshooting Action

This is default behavior of Cisco EPOM, generally Cisco EPOM should be used at least 10 minutes after it has been started. The time taken by Cisco EPOM depends on the number of Cisco BTS EMS servers added to Cisco EPOM server. If one of the Cisco BTS EMS server faces Corba communication problem, then it would have a tail effect on the Cisco BTS EMS servers following it. Once the Cisco EPOM Corba Cache is built, it takes very less time to communicate via Corba with the Cisco BTS EMS servers.

Problem

Cisco EPOM JVM tunings

Troubleshooting Action

The best Sun JVM settings have been observed as:

-server -XX:+UseParallelGC -Xms512M -Xmx512M
These settings have major impact on performance of EPOM and Cisco doesn't suggest changing these parameters.

If really essential, the tunings can be changed by modifying EPOM_OPTS variable in the file /opt/CSCOepom/tomcat/bin/epomcatalina.sh

If it is felt that the current Sun JVM settings are not fine for your installation, then enable the:

-verbosegc

option in EPOM_OPTS, as defined above and contact Cisco TAC with your installation details and logfile:

/var/opt/CSCOepom/logs/Catalina.out

Problem

Cisco EPOM bts_export errors

Troubleshooting Action

Many errors arise while usage of bts_export, due 2 main reasons:

Corba Communication is not established properly; try using IP address rather than hostname.

root or other such Cisco BTS EMS logins are used, which are not associated with a BTS shell. Try using Cisco BTS EMS logins that are associated with BTS CLI shell, like optiuser.