Cisco CallManager Serviceability System Guide, Release 4.1(2)
Real-Time Monitoring Tool

Table Of Contents

Real-Time Monitoring Tool

Understanding the Real-Time Monitoring Tool

Monitoring Objects

Devices Category

Servers Category

Services Category

Call Activities Category

RTMT Service Parameters

RTMT Logon

RTMT Window Overview

RTMT Default Configuration

RTMT Window Configuration Checklist

RTMT Window Components

Menu Bar

System Menu

Monitor Menu

Search Menu

Device Search

CTI Search

Edit Menu

Device Menu

Perfmon Menu

Alert Menu

Window Menu

Application Menu

Help Menu

Monitor Window

View Tab

Summary

Server Status Monitoring

Call Process Activity Monitoring

Service Monitoring

Cisco TFTP

Directory Servers

RTMT Heartbeat Feature

Device Monitoring

CTI Monitoring

Perfmon Monitoring

Category Tabs

Sample Rate

Perfmon Counters Table View/Chart View

Remove Chart/TableEntry

Understanding Perfmon Monitoring

Add a Counter to Monitor

Configure Alert Notification for Counter

Zoom Counter

Counter Properties

Property Description

Data Sample

View All Data for a Counter

Perfmon Monitoring Configuration Checklist

Alert Tab

Alert Customization

Alert Action Configuration

Preconfigured Alerts

Alert Configuration Checklist

Logging in RIS

Logger

Alert Log

Server Log

Service Log

Call Log

Device Log

Perfmon Log

Where to Find More Information


Real-Time Monitoring Tool


This chapter provides information on the Real-Time Monitoring Tool (RTMT) for Cisco CallManager Serviceability and comprises the following topics:

Understanding the Real-Time Monitoring Tool

RTMT Service Parameters

RTMT Logon

RTMT Window Overview

RTMT Default Configuration

RTMT Window Configuration Checklist

RTMT Window Components

Menu Bar

Monitor Window

View Tab

Understanding Perfmon Monitoring

Alert Tab

Alert Configuration Checklist

Logging in RIS

Where to Find More Information

Understanding the Real-Time Monitoring Tool

Cisco CallManager Serviceability provides a client side standalone plugin, RTMT, that monitors real-time behavior of the components in a Cisco CallManager cluster. RTMT runs as an application and uses HTTP and TCP to monitor device status, system performance, device discovery, and CTI applications. It also connects directly to devices by using HTTP for troubleshooting system problems.

RTMT continuously monitors a set of management objects that are preconfigured and generates various alerts, in the form of e-mails, for these objects when values go over/below user-configured thresholds. RTMT then generates daily reports for these objects. See the "Monitoring Objects" section for more information about the preconfigured monitoring objects.


Note RTMT continues to monitor real-time behavior of Cisco CallManager components in the background even when RTMT is not running.


Monitoring Objects

The RTMT arranges the preconfigured monitoring objects into four major categories:

Devices

Call Activities

Servers

Services

The following sections provide more information about each category, including what objects RTMT monitors, the alert, threshold and defaults, and what kind of reports RTMT generates for each. Refer to Chapter 11, "Serviceability Reports Archive" in the Cisco CallManager Serviceability System Guide for more information on viewing the RTMT reports.

Devices Category

The Devices category monitors phones, gateways, and media devices on each Cisco CallManager and Cisco CallManager cluster. Table 9-1 provides information about the objects that RTMT monitors, the alert, thresholds, and defaults, and what kind of reports that RTMT generates for those devices.

Table 9-1 Devices Category 

Monitored Objects (displayed)
Alert/Threshold/Default
Report

Number of registered phones for each Cisco CallManager and cluster.

Total number of registered phones drops by X% in consecutive polls. Default specifies 10%.

Daily reports on number of registered devices. See the "Device Statistics Report" section on page 11-4 for detailed information.

Number of registered gateways on each Cisco CallManager and cluster.

(Warning) Clusterwide total number of registered gateways decreased in consecutive polls.

(Informational) Clusterwide total number of registered gateways increased in consecutive polls.

Number of registered media devices on each Cisco CallManager and cluster.

(Warning) Clusterwide total number of registered media devices decreased in consecutive polls.

(Informational) Clusterwide total number of registered media devices increased in consecutive polls.

Media List exhausted.


Servers Category

The Servers category monitors CPU usage, disk space usage, and critical services on each Cisco CallManager server. Table 9-2 provides information about the objects that RTMT monitors, the alert, thresholds, and defaults, and what kind of reports that RTMT generates for those servers.

Table 9-2 Servers Category 

Monitored Objects (displayed)
Alert/Threshold/Default
Report

CPU Usage (100% Idle) on each server.

Memory Usage (% memory in use out of total) on each server.

CPU and memory usage for all processes on each server (% Processor Time, VirtualBytes, PrivateBytes, Working Set, Open handles, Thread count, and PID).

Call Processing node - CPU usage pegged X% over X seconds. Default specifies 90%, 30 seconds.

Non-CallProcessing node - CPU Usage pegged X% over X seconds. Default specifies 99%, 120 seconds.

Available memory equals less than X%. Default specifies 10%.

When CPU peg or excessive memory usage alerts occur, the name of the top process displays in the alert message. In case of dllhost service, the real meaningful name displays instead of generic name dllhost.

Daily CPU, memory, and disk usage reports. See the "Server Statistics Report" section on page 11-8 for detailed information.

Disk space usage for all logical drives on each server.

Available disk space on the largest logical drive equals less than X%. Default specifies 10%.

State of activated critical services on each server.

Service state changed from Up->Down.


Services Category

The Services category monitors CTI Manager information for each CTI Manager, Cisco TFTP server information, directory server information, and heartbeat rate information. Table 9-3 provides information about the objects that RTMT monitors, the alert, thresholds, and defaults, and what kind of reports that RTMT generates for those services.

Table 9-3 Services Category 

Monitored Objects (displayed)
Alert/Threshold/Default
Report

Number of open devices, lines, CTI connections, and active Cisco CallManager links for each CTI Manager.

N/A

Daily report on CTI and Cisco TFTP usage statistics. See the "Service Statistics Report" section on page 11-11 for detailed information.

TotalTftpRequests and TotalTftpRequestsAborted, for each Cisco TFTP server.

N/A

Connection and replication status for each Directory server.

Connection failed

Replication failed

Heartbeat rate for each Cisco CallManager, Cisco TFTP, and TCD services.

Cisco CallManager heartbeat rate equals <0.x. Default is 0.5.

Cisco TFTP heartbeat rate equals <0.x. Default specifies 0.5.

TCD heartbeat rate equals <0.x. Default specifies 0.5.


Call Activities Category

The Call Activities category monitors all call activity information for each Cisco CallManager node and cluster. Table 9-3 provides information about the objects that RTMT monitors, the alert, thresholds, and defaults, and what kind of reports that RTMT generates for those call activities.

Table 9-4 Call Activities Category 

Monitored Objects (displayed)
Alert/Threshold/Default
Report

CallsAttempted, CallsCompleted, and CallsInProgress for each Cisco CallManager node and cluster.

N/A

Daily report on Cisco CallManager call activites. See the "Call Activities Report" section on page 11-15 for more detailed information.

CallsAttempted, CallsCompleted, and CallsInProgress of each type of MGCP FXS/FXO/PRI/T1CAS/H.323 gateway, as well as SIP and H.323 Trunks for each Cisco CallManager node and cluster.

N/A

Channel/Port Status of each MGCP FXS/FXO/PRI/T1CAS gateway.

N/A

SDL Queue activity on each Cisco CallManager node.

N/A

MGCP FXS Gateway - Number of In-Service and Active ports for each Cisco CallManager node and cluster.

Route-List exhausted

MGCP FXO Gateway - number of In-Service and Active ports for each Cisco CallManager node and cluster.

Route-List exhausted

MGCP PRI Gateway - Number of In-Service and Active channels for each Cisco CallManager node and cluster.

D-Channel out of service

Route List exhausted

 

MGCP T1CAS Gateway - number of In-Service and Active ports for each Cisco CallManager node and cluster.

Route List exhausted


RTMT Service Parameters

The following list comprises the service parameters that are associated with RTMT:

Data Collection Enabled

Data Collection Polling Rate

Data Collection Reenumeration Interval

Server Synchronization Period

Primary Collector

Failover Collector

For more information about service parameter configuration, refer to the Cisco CallManager Administration Guide.

Figure 9-1 and Figure 9-2 show examples of the RTMT service parameters in the Cisco CallManager Administration web pages.

Figure 9-1 Service Parameter Configuration Window

Figure 9-2 Service Parameter Descriptions Window

RTMT Logon

You will find that RTMT is available as a download from the Cisco CallManager Install Plugins web page. When you are ready to download RTMT, you can click on Cisco CallManager Serviceability Real-Time Monitoring Tool from the Tools menu in Cisco CallManager Serviceability Administration.

When you launch RTMT from the desktop icon, or from Start > Programs > Cisco CallManager Serviceability > RTMT, a window displays as shown in Figure 9-3.

Figure 9-3 RTMT Launch Window

To connect to a Cisco CallManager node (any node in the Cisco CallManager cluster) you must enter the user authentication information in the User Name and Password fields. The Host IP Address field can be either digital IP address or host name.

If the authentication fails or if the server is not reachable, you will be prompted to reenter the server and authentication details, or you can click the Cancel button to exit the application. If the authentication succeeds, RTMT launches the monitoring module either from local cache or from a remote node when the local cache does not contain a monitoring module that matches the backend version.

After you enter the authentication information, the RTMT window opens and displays with the default configuration. See the "RTMT Default Configuration" section.

RTMT Window Overview

See the following sections for some general information about the RTMT window:

RTMT Default Configuration

RTMT Window Components

Figure 9-4 shows the RTMT main window.

Figure 9-4 Real-Time Monitoring Tool Window

RTMT Default Configuration

When you initially load RTMT, the load includes a default configuration that is called CM-Default. Dynamically created, CM-Default monitors all registered phones in all the Cisco CallManager nodes. If your cluster has five configured Cisco CallManager nodes, CM-Default displays all the registered phones for each node in a Cisco CallManager cluster, as well as calls in progress and active gateway ports and channels. See Figure 9-5 and Figure 9-6 for CM-Default examples.

Figure 9-5 RTMT Window with CM-Default Dialog Box

Figure 9-6 RTMT Screen with CM-Default Configuration

You can save, restore, and delete Cisco CallManager configuration information by using Profile in the System menu. See the "RTMT Window Components" section.

RTMT Window Configuration Checklist

Table 9-5 provides an overview of the steps for configuring the RTMT window.

Table 9-5 RTMT Window Configuration Checklist 

Configuration Steps
Related Procedures and Topics

Step 1 

Load RTMT.

Loading Real-Time Monitoring, Cisco CallManager Serviceability Administration Guide

Step 2 

View existing configuration preferences.

RTMT Default Configuration, Cisco CallManager Serviceability Administration Guide

Step 3 

Change to another configuration setting.

Creating Configuration Profiles, Cisco CallManager Serviceability Administration Guide


RTMT Window Components

The RTMT window comprises two main components: the Menu Bar and the Monitor Window. You can access several different monitoring functions by using both and/or either of these components.

See the "Menu Bar" section and the "Monitor Window" section for information about the menu bar and monitor window functions.

Menu Bar

The RTMT menu bar comprises several menu items that you can use to access different monitoring components. See the following sections for information about RTMT menu bar items:

System Menu

Monitor Menu

Search Menu

Edit Menu

Device Menu

Perfmon Menu

Alert Menu

Window Menu

Application Menu

Help Menu

System Menu

The System menu provides access to applicationwide functionality, such as Profile and logout.

Figure 9-7 shows a System menu example.

Figure 9-7 System Menu

The System menu includes the following menu items:

Profile—This menu item allows you to save, restore, and delete existing Cisco CallManager configurations.

JVM Information—This menu contains Java Heap Memory Usage monitoring.

Serviceability Reports Archive—This menu takes you to the Serviceability Report Archive web page in Cisco CallManager Serviceability. See Chapter 11, "Serviceability Reports Archive,"in the Cisco CallManager Serviceability System Guide for more information.

Log Off—This menu item allows you to log off of RTMT.

Monitor Menu

The Monitor menu provides access capability to precanned (preconfigured) items in the menu system. It includes the following categories:

Cluster Summary—This menu item provides a summary of every Cisco CallManager in a cluster.

Server—This menu item provides server monitoring information such as CPU and memory status, disk usage, and the status of critical services.

Call Process—This menu item provides call-processing monitoring information such as call, gateway, and trunk activity monitoring, and SDL queue information.

Service—This menu item provides service status monitoring, and includes directory server and system heartbeat information. See the"Directory Servers" section and the "RTMT Heartbeat Feature" section for more information.

Device Summary—This menu item provides a summary of the devices within a Cisco CallManager cluster.

CTI Manager—This menu item provides information that pertains to CTI Manager such as the number of open lines, devices, and CTI connections per CTI Manager.


Tip To monitor the precanned set of monitoring objects in RTMT, you can use either the menu bar at the top of the monitor window or the left controlling panel of the monitor window. For more information about the left controlling panel, see the "Monitor Window" section.


Figure 9-8 shows a Monitor menu example.

Figure 9-8 Monitor Menu

Search Menu

This menu provides device-based and CTI-based search functionality. Using the Search menu, you can search for any and all devices within a Cisco CallManager, as well as the CTI applications, lines, and devices per each CTI Manager. See the "Device Search" section, and the "CTI Search" section for more information.

Figure 9-9 shows a Search menu example.

Figure 9-9 Search Menu

Device Search

The Device search menu comprises the following menu items on which you can search:

Phone

Gateway Devices

H323 Devices

CTI Devices

VoiceMail Devices

Media Resources

Hunt List

SIP Trunk

Within these menu items, you can search on the basis of any device in the Cisco CallManager cluster and choose the status of the devices, including registered, unregistered, rejected, any status, and devices that are only configured in the database. You can also search by any model, or a specific device model and set up criteria that include several different attributes.

You can monitor chosen devices on the basis of the criteria that you specify for that device. You can search by the status of the device, device name, and application pattern. See Figure 9-10 through Figure 9-13 for examples of these monitoring criteria windows, where the device being searched is a phone.

Figure 9-10 Search with Device Status Window

Figure 9-11 Search with Device Model Window

Figure 9-12 Search with Device Name Window

Figure 9-13 Search with Device Attributes Window

RTMT queries RIS to find the matching device. Results display in a table with a row for each matched device, a column for each of the specified attributes, and a time stamp of the device that has been opened/closed and the application that controls the device media. See Figure 9-14 for a search results example.

Figure 9-14 Device Search Results

CTI Search

The CTI search menu allows you to search on the following CTI components:

CTI Applications

CTI Devices

CTI Lines

Using the CTI Search menu, you can search chosen CTI components on the basis of the criteria that you specify. You can also choose any or all the CTI Managers on which you want to search.

You can specify criteria for the CTI applications, devices, and lines that include CTI status, device name, application pattern, and attributes. After you set up your search criteria, RTMT queries the RIS for matched CTI applications, devices, or lines. The results display in a table with a row for each matched application and a column for each attribute; a Time Stamp of provider opened/closed status also displays.


Note You can monitor each search result or bring up Cisco CallManager Administration in a separate browser by right-clicking on each item. Figure 9-15 shows a right-click pop-up menu example.


Figure 9-15 Right-click Popup Menu Example

Edit Menu

Use this menu to specify context-sensitive edit actions that are also available in right-click popup menus. You can also use the "Trace Setting" item in the menu to trace and debug the RTMT application.

The Edit menu comprises the following menu items:

New TAB—You can use this menu item to create a new category tab. Refer to Chapter 12, "Real-Time Monitoring Configuration," in the Cisco CallManager Serviceability Administration Guide for procedural information about the New TAB item.

Delete TAB—You can use this menu item to delete an existing category tab. Refer to "Real-Time Monitoring Tool," in the Cisco CallManager Serviceability Administration Guide for procedural information about the Delete TAB item.

Rename TAB—You can use this menu item to rename an existing category tab.

Polling Rate—With this menu item, you can set the polling rate of a perfmon counter.


Note The "Polling Rate" item does not apply for precanned monitoring objects.


Hide QuickLaunch Channel—Check this menu item if you want to hide the QuickLaunch channel on the left controlling center panel of the RTMT.

Trace Setting—You can use this menu item to trace and debug the RTMT application. Available trace settings in this menu include:

None

Fatal

Error

Warning

Information

Debug

Figure 9-16 shows an Edit menu example.

Figure 9-16 Edit Menu

Refer to "Real-Time Monitoring Tool," in the Cisco CallManager Serviceability Administration Guide for procedural information about the menu items in the Edit menu.

Device Menu

The Device menu provides the following device-related menu items:

Open Device Search

App Info

Port/Channel Status

Properties

Open

Figure 9-17 shows a Device menu example.

Figure 9-17 Device Menu

Perfmon Menu

The Perfmon menu provides access to Perfmon functionality and includes the following menu items:

Open Perfmon Monitoring

Counter Description

Counter Instances

Counter Monitoring

Monitoring Properties

Zoom Chart

Remove Chart/TableEntry

Figure 9-18 shows a Perfmon menu example.

Figure 9-18 Perfmon Menu

Alert Menu

The Alert menu provides all alert-related functionality including alert defining, setting, and viewing. This menu comprises the following menu options:

Alert Central—This option comprises the current status and history of every alert in the Cisco CallManager cluster.


Note You can also access Alert Central by using the Alert tab in the left controlling center in the RTMT monitoring pane.


Set Alert/Properties—This menu category allows you to set alerts and alert properties.

Remove Alert—This menu category allows you to remove an alert.

Enable Alert—With this menu category, you can enable alerts.

Disable Alert—You can disable an alert with this category.

Suspend cluster/node Alerts—This menu category allows you to temporarily suspend alerts on a particular Cisco CallManager node or on the entire cluster.

Clear Alerts—This menu category allows you to reset an alert (change the color of an alert item from red to black) to signal that an alert has been taken care of. Once an alert has been raised, its color will automatically change to red in RTMT and will stay that way until you manually clear the alert.

Clear All—This menu category allows you to clear all alerts.

Alert Events Detail—This menu category provides detailed information on alert events.

Config Email Server—In this category, you can configure your e-mail server to enable alerts.

Config Alert Action—This category allows you to set actions to take for specific alerts that you can configure to send to desired e-mail recipiants.

Two kinds of alerts occur: preconfigured and user-defined. You can further configure both, but the difference is that you cannot delete preconfigured alerts whereas you can add and delete user-defined alerts.

For more information on customizing alerts, refer to Chapter 12, "Real-Time Monitoring Configuration," in the Cisco CallManager Serviceability Administration Guide.

Figure 9-19 through Figure 9-25 show examples of different alert menu categories and their associated windows.

Figure 9-19 Alert Menu

Figure 9-20 Alert Properties Window

Figure 9-21 Alert Properties Threshold & Duration Window

Figure 9-22 Alert Action Window

Figure 9-23 Alert Frequency & Schedule Window

Figure 9-24 Alert Properties E-mail Notification Window

Figure 9-25 Disable Alert Example

Window Menu

You can use the Window menu to close opened monitoring windows by using two menu items: Close and Close All Windows. You can use the Close menu item to close the current active window in the right content viewing panel. Use the Close All Windows menu item to close all opened monitoring windows. Figure 9-26 shows a Window menu example.

Figure 9-26 Window Menu

Application Menu

The Application menu provides a way to access Cisco CallManager Adminstration and Cisco CallManager Serviceability web pages. Figure 9-27 shows an Applications menu example.

Figure 9-27 Application Menu

Help Menu

The Help menu provides information about product name, client and server software versions, and third-party libraries software version information, as well as RTMT help pages.

. Figure 9-28 shows a Help menu example.

Figure 9-28 Help Menu

Monitor Window

To support the precanned set of monitoring objects, the RTMT main user interface comprises two parts: left controlling center pane and right content viewing panel.

The controlling center comprises the View tab and the Alert tab. The View tab comprises several different monitoring categories, and the Alert tab comprises only the Alert category.

The View and Alert tabs differ in that anything related to the View tab gets saved as a profile that can be restored at any time that RTMT is invoked. The Alert tab relates only to the systemwide alert functionality. The states cannot be saved. See the "View Tab" section and the "Alert Tab" section for more information about each tab.

The polling rate in each precanned monitoring window remains fixed, and the default value specifies 30 seconds. If the collecting rate of RISDC (real-time information system directory) changes in Cisco CallManager Administration service parameters web pages, the polling rate in the precanned window also updates. In addition, the local time of the RTMT client application and not the backend server time, provides the basis for the time stamp in each chart.

View Tab

The View tab in the RTMT monitor window comprises several different monitoring categories. See the following sections for more information on each category:

Summary

Server Status Monitoring

Call Process Activity Monitoring

Service Monitoring

Device Monitoring

CTI Monitoring

Perfmon Monitoring

Summary

The Summary page provides monitoring of some important common information on a central page. Monitored information includes

Memory usage

CPU usage

Registered phones

Calls in progress

Active gateway ports and channels

Figure 9-29 shows a summary monitoring page example.

Figure 9-29 Summary Page

Server Status Monitoring

The Server category monitors precanned objects such as CPU and memory usage, disk usage, and critical Cisco CallManager services. See Figure 9-30 through Figure 9-32 for examples of each monitored object.

Figure 9-30 CPU and Memory Usage

Figure 9-31 Disk Usage

Figure 9-32 Critical Services

The Critical Services monitoring category provides the name of the critical service, the status (whether the service is up, down, or activated), and the elapsed time during which the services have been in a particular status for a particular Cisco CallManager node.

Call Process Activity Monitoring

The Call Process monitoring category monitors Cisco CallManager call processing activity. Call processing monitored items include

Call Activity—You can monitor the number of calls that were attempted, calls that were completed, and call in progress for a particular Cisco CallManager node or the entire cluster.

Gateway Activity—You can monitor gateway activity for each gateway type. Gateway activity monitoring includes the number of active ports, the number of ports in service, and the number of calls that were completed for each gateway type for a particular Cisco CallManager node or the entire cluster.

Trunk Activity—The system monitors trunk activity by trunk type for a particular node or cluster. Trunk activity monitoring includes the number of calls in progress and the number of calls that were completed for a particular trunk type.

SDL Queue—SDL Queue monitoring monitors the number of signals in the SDL queue and the number of signals that were processed for a particular signal distribution layer (SDL) queue type. The SDL queue types comprise high, normal, low, and lowest queue. You can monitor the SDL queue for a particular node or the entire cluster.

See Figure 9-33 through Figure 9-36 for examples of call-processing activity monitoring.

Figure 9-33 Call Activity Monitoring

Figure 9-34 Gateway Activity Monitoring

Figure 9-35 Trunk Activity Monitoring

Figure 9-36 SDL Queue Monitoring

Service Monitoring

The Service monitoring category monitors the activities of the Cisco TFTP, the directory server, and the heartbeat information. See the following sections for more information on the directory server and heartbeat:

Cisco TFTP

Directory Servers

RTMT Heartbeat Feature

Cisco TFTP

The Cisco TFTP service builds and serves files that are consistent with the trivial file transfer protocol, which is a simplified version of the File Transfer Protocol (FTP). Cisco TFTP builds configuration files and serves embedded component executables, ringer files, and device configuration files.

You can view the total Cisco TFTP requests, requests not found, and requests that were aborted.

Directory Servers

Directory servers comprise database repositories that store user and device information such as user name, password, and location. The Cisco CallManager publisher contains the directory where the directory settings are similar to SQL. Each node in the cluster has its own copy of the directory. The directory supports three types of directory: embedded, active, and Netscape. An embedded directory resides on the same node as its associated Cisco CallManager (other types reside on other nodes in the cluster). A directory that resides on the publisher node provides write permission.

You can view the directory servers connection status. Directory server connection status and replication status get checked when an alert is pending. If the server is not connected, an alert occurs. The directory server connection status gets polled every 10 minutes; replication status gets polled once every hour.

RTMT Heartbeat Feature

The Real-Time Monitoring tool monitors the heartbeat of Cisco CallManagers, Cisco TFTPs, and Cisco TCDs. The heartbeat acts as an indicator of the life of whatever it is monitoring. When the heartbeat is lost, a blinking icon appears in the lower, right corner of the RTMT window. To find when the heartbeat loss was detected, click the blinking icon. An e-mail can notify you of the heartbeat loss.

Device Monitoring

The Device monitoring category provides a summary of devices and device search capability. The device summary monitors registered phone, gateway, and media resource devices on a particular Cisco CallManager and an entire cluster. The device search category allows you to search for every device in a Cisco CallManager cluster on the basis of the search criteria that you set up.


Note You can also access the device summary information in Device Summary in the Monitor menu in the menu bar. See "Monitor Menu" section for more information. You can also search for devices by using the Search menu in the menu bar. See the "Search Menu" section for more information.


CTI Monitoring

The CTI category monitors CTI Manager activities and provides CTI search capability. With CTI Manager, you can monitor the number of open devices, lines, and CTI connections.


Note You can also monitor the CTI Manager from CTI Manager in the Monitor menu in the menu bar. See the "Monitor Menu" section for more information. You can also search on CTI applications, devices, and lines by using the Search menu in the menu bar. See the "Search Menu" section for more information.


Perfmon Monitoring

RTMT displays performance information of all Cisco CallManager components in a cluster. The tool integrates with Cisco CallManager administration and serviceability software. RTMT provides an alert notification mechanism to ease the system administrator troubleshooting tasks. It also monitors various aspects of the Cisco CallManager performance by periodically polling NT/2000 performance counter values. The system continuously monitors a set of common, preconfigured objects, and various alerts in the form of e-mail messages get generated for these objects when values are over or below user-configured thresholds. In addition, the system generates daily reports for these objects.

Perfmon monitoring provides the following services:

From the Cisco CallManager cluster, monitors performance counters including Cisco CallManager nodes, TFTP servers, and database servers.

Continuously monitors a set of preconfigured objects and notifies administrator in the form of an e-mail message.

Presents counters hierarchically for easy navigation.

Associates counter threshold settings to alert notification. An e-mail or popup message provides notification to the administrator.

Permits saving and restoring settings, such as counters being monitored, threshold settings, and alert notifications, for customized troubleshooting tasks.

Displays up to six perfmon counters in one chart for performance comparisons.

See the following sections for configuration options in the RTMT perfmon monitoring pane:

Category Tabs

Sample Rate

Perfmon Counters Table View/Chart View

Remove Chart/TableEntry

Refer to Chapter 12, "Real-Time Monitoring Configuration," in the Cisco CallManager Serviceability Administration Guide for more configuration options.

Category Tabs

A category comprises monitored performance counters. A tab in the RTMT monitoring pane contains the category name. All performance counters that are monitored in this tab belong to a category. The system polls the performance counters in the tab at the same rate, with each category configured to have its own polling rate.

You can create custom categories in the RTMT monitoring pane to view information that helps you troubleshoot specific performance or device problems. If your Cisco CallManager system is experiencing performance problems with specific objects, create custom categories to monitor the performance of the counters within the object. If the system is experiencing problems with specific devices, create custom categories to monitor the devices within the cluster. In addition, you can create alert notifications for counters and gateways in these custom categories.

To create custom categories, you add a new category tab. When the tab is created, you specify the specific performance counters, devices, and alerts within that tab and then save your custom category by using Profile.

Sample Rate

The Cisco CallManager software polls counters, devices, and gateway ports to gather status information. In the RTMT monitoring pane, you configure the polling intervals for the performance counters, devices, and gateway ports for each category tab that you create.


Note High-frequency polling rate affects Cisco CallManager performance. The minimum polling rate for monitoring a performance counter in chart view equals 5 seconds; the minimum rate for monitoring a performance counter in table view equals 1 second. The default for both specifies 10 seconds.


Perfmon Counters Table View/Chart View

The Cisco CallManager software displays performance counters in chart or table format. Chart format looks like a miniature screen of information. Up to six charts display in the RTMT performance monitoring pane for each category tab that you create. You can display a particular counter by double clicking the counter in the perfmon monitoring pane. Because chart view represents the default, you configure the performance counters to display in table format when you create a category.

Remove Chart/TableEntry

You can remove a counter chart (table entry) with the Remove Chart/TableEntry menu item in the Perfmon menu in the menu bar. See the "Perfmon Menu" section for an example of the Perfmon menu.

Understanding Perfmon Monitoring

You can monitor the performance of the Cisco CallManager software by choosing the counters for any object. The counters for each object display when the folder is expanded. Figure 9-37 shows an example of the perfmon monitoring objects and counters in the expanded view.

Figure 9-37 Perfmon Objects and Counters in Expanded View

Add a Counter to Monitor

To troubleshoot system performance problems, you add the counter that is associated with the perfmon object to the RTMT performance monitoring pane, which displays a chart for the counter. Before you add counters, see the "Category Tabs" section.

Category tabs contain up to six perfmon counter charts.

Configure Alert Notification for Counter

Using the alert notification feature, Cisco CallManager notifies you of system problems. Perform the following configuration setup to activate alert notifications for a system counter:

From the RTMT Perfmon Monitoring pane, choose the system perfmon counter.

Set up an e-mail or a message popup window for alert notification.

Determine the threshold for the alert (for example, an alert activates when calls in progress exceed the threshold of over 100 calls or under 50 calls).

Determine the frequency of the alert notification (for example, the alert occurs once or every hour).

Determine the schedule for when the alert activates (for example, on a daily basis or at certain times of the day).

Figure 9-38 provides an example of an alert e-mail that was sent to the system administrator when the RegisteredPhones was below the low threshold.

Figure 9-38 Alert Notification E-Mail

Refer to "Real-Time Monitoring Configuration" in the Cisco CallManager Serviceability Administration Guide for more detailed information.

Zoom Counter

To get a closer look at performance monitors, zoom the monitor counter in the RTMT perfmon monitoring pane by highlighting the counter chart and choosing Zoom Chart in the Perfmon menu.

Counter Properties

Counter properties provide two capabilities: display a description of the counter and configure data sampling parameters.

Property Description

Use one of two ways to obtain a description of the counter:

In the monitoring objects pane, right-click the counter to display only the description of the counter.

In the RTMT permon monitoring pane, right-click the counter chart to display the description and configure the data sampling parameters.

Data Sample

The Counter Property window contains the option to configure data samples for a counter. The performance counters that display in the RTMT performance monitoring pane contain green dots that represent samples of data over time. You can configure the number of data samples to collect and the number of data points to show in the chart. After the data sample is configured, view the information by using the View All Data/View Current Data menu selection. See View All Data for a Counter.

View All Data for a Counter

Use this view data for a monitor option to view all the data that was collected for a perfmon counter. See Counter Properties, for a description of data sample configuration.

Perfmon Monitoring Configuration Checklist

Table 9-6 provides an overview of the steps for monitoring perfmon counters.

Table 9-6 Performance Monitoring Configuration Checklist 

Configuration Steps
Related Procedures and Topics

Step 1 

Load RTMT.

Loading Real-Time Monitoring, Cisco CallManager Serviceability Administration Guide

Step 2 

Choose a Cisco CallManager object/counter to monitor and activate alert notification for the counter.

Adding a Counter to Monitor by Using the Perfmon Category, Cisco CallManager Serviceability Administration Guide


Alert Tab

In RTMT you can configure alert notification for perfmon counter value thresholds, schedule for alert checking, and status change of device (for example, port is out of service).

The Alert tab in the left controlling center in the RTMT monitor window, comprises the Alert Central category. Alert Central provides both the current status and the history of all the alerts in the Cisco CallManager cluster.

Use the Alert tab in conjuction with the Alert menu in the RTMT menu bar to monitor Cisco CallManager alerts. See the "Alert Menu" section for more information about the alert menu.

Figure 9-39 shows an alert tab example.

Figure 9-39 Alert Tab

Alert Customization

You can configure both preconfigured and user-defined alerts in RTMT; however, you cannot delete preconfigured alerts. You can add and delete user-defined alerts. You can also disable both preconfigured and user-defined alerts in RTMT.

Table 9-7 provides a list of fields that you will use to configure each alert; all fieldspreconfigured are user-configurable unless otherwise marked.

Table 9-7 Alert Customization 

Field
Description
Comment

Alert Name

High-level name of the monitoring item with which RTMT associates an alert

Descriptive name. For preconfigured alerts, you cannot change this field. See the "Preconfigured Alerts" section.

Description

Description of the alert

You cannot edit this field for preconfigured alerts. See the "Preconfigured Alerts" section.

Perfmon Counters

List of source perfmon counters

For preconfigured alerts, you cannot change this field.

Value Threshold

Condition to raise alert (value is...)

Specify up < - > down, less than #, %, rate greater than #, %, rate.

Evaluation Method

Method used to check the threshold condition

Specify value to be evaluated as absolute, delta (present - previous), or % delta.

Duration Threshold

Condition to raise alert (how long value threshold has to persist before raising alert)

Options include right away and specify at least X minutes.

Number of Events Threshold

Raise alert only when a configurable number of events exceed a configurable time interval (in minutes).

For ExcessiveVoiceQualityReports, the default thresholds are 10 within 60 minutes. For RouteListExhausted and MediaListExhausted, the defaults are 0 within 60 minutes.

Node IDs

Cluster or list of nodes to monitor

Cisco CallManager nodes, Cisco TFTP node, or publisher node.


Note When you deactivate both Cisco CallManager and Cisco TFTP services of a node, that node is virutally considered removed from the currently monitored node list. When you reactivate both Cisco CallManager and Cisco TFTP services, that node is added back and its settings get restored to default values.


Alert Action ID

ID of alert action to take (system always logs alerts no matter what the alert action).

Alert action gets defined first (see the "Alert Action Configuration" section). If this field is blank, that indicates that e-mail is disabled.

Disabled

Alert disabled

Options include enabled or disabled

Clear Alert

To reset alert (change the color of an alert item from red to black) to signal that the alert has been resolved.

After an alert has been raised, its color will automatically change to red and stay that way until you manually clear the alert. Use Clear All to clear all alerts.

In Safe Range

To indicate if an alert threshold condition has been met (not configurable).

This field is not applicable to MaliciousCall and Registered YYY types of alerts. For DChannel OOS alert, this field is YES only when you no longer have any outstanding OOS devices. The Code Yellow alert will be back in safe range only when you receive a CodeYellowExit event from Cisco CallManager.

Alert Details

To view the detail of an alert (not configurable).

For VoiceQualityReports, RouteListExhausted, and MediaListExhausted, up to 30 current event details get displayed in the current monitoring interval if an alert has been raised in the current interval. Otherwise, the previous 30 event details in the previous interval get displayed. For DChannel OOS alert, the list of outstanding OOS devices at the time the alert was raised gets displayed.

Alert Generation Rate

How often to generate alert when alert condition persists

Specify every X minutes. (Raise alert once every X minutes if condition persists.)

Specify every X minutes up to Y times. (Raise alert Y times every X minutes if condition persists.)

User Provide Text

Administrator to append text on top of predefined alert text

N/A

Severity

For viewing purposes (for example, show only Sev. 1 alerts)

Specify defaults that are provided for predefined (for example, Error, Warning, Information).

Collection Polling Rate

Same rate for both preconfigured and user-defined alerts

Although not configurable through RTMT Alert Central, you can customize this through Cisco CallManager Administration in Service Parameters.


Alert Action Configuration

In RTMT, you can configure alert actions for every alert that is generated and have the alert action sent to e-mail recipients that you specify in the alert action list.

RTMT provides a list of fields you will use to configure alert actions. All are user-configurable unless otherwise marked.

Table 9-8 Alert Action Configuration 

Field
Description
Comment

Alert Action ID

ID of alert action to take

Specify descriptive name.

Mail Recipients

List of e-mail addresses. You can selectively enable/disable an individual e-mail in the list.

N/A

Global Alert Action Flag

Flag to effectively disable all e-mails

If you have disabled this flag, you disabled all of the e-mails and no e-mails get sent out, even though alerts occur.


Preconfigured Alerts

RTMT comprises a set of preconfigured alerts. You cannot delete these alerts; however, you can enable or disable them in Alert Central.

The following list comprises the preconfigured alerts in RTMT:

NumberOfRegisteredPhonesDropped

NumberOfRegisteredGatewayDevicesDecreased

NumberOfRegistedGatewayDevicesIncreased

NumberOfRegisteredMediaDevicesDecreased

NumberOfRegisteredMediaDevicesIncreased

MediaListExhausted

MgcpDChannelOutOfService

RouteListExhausted

CriticalServiceDown

CallProcessingNodeCpuPegging

NonCallProcessingNodeCpuPegging

LowInAvailableMemory

LowInAvailableDiskSpace

LowInAvailableHeartbeatRate

LowTFTPServerHeartbeatRate

LowTcdServerHeartbeatRate

DirectoryConnectionFailed

DirectoryReplicationFailed

MaliciousCallTrace

ExcessiveVoiceQualityReports

Code Yellow

Alert Configuration Checklist

Table 9-9 provides an overview of the steps for configuring alerts.

Table 9-9 Alert Configuration Checklist 

Configuration Steps
Related Procedures and Topics

Step 1 

Load RTMT.

Loading Real-Time Monitoring, Cisco CallManager Serviceability Administration Guide

Step 2 

For non-preconfigured alerts, set up alert for counters within a category.

Adding a Counter to Monitor by Using the Perfmon Category, Cisco CallManager Serviceability Administration Guide

Step 3 

Set up alert notification for a perfmon counter.

Configuring Alert Notification for a Counter, Cisco CallManager Serviceability Administration Guide


Logging in RIS

See the following sections for information about logging in RIS in RTMT:

Logger

Alert Log

Server Log

Service Log

Call Log

Device Log

Perfmon Log

Logger

RTMT Collector uses the logger to log preconfigured monitoring objects information and Alert Manager uses logger to log alert histories into log files. Each preconfigured object belongs to one of the four categories, namely devices, services, servers, and call activities. Each category has a separate log file. Alert details get logged in a separate file. Also, another log file exists for perfmon that logs the important perfmon object values for Cisco CallManager-related services and processes.

The primary collector invokes the logger with the data from the whole of the Cisco CallManager cluster. The locally written log files appear in the primary collector server at C:\Program Files\Common Files\Cisco\Logs\RTMTLogger. As the primary collector can change due to failover/fallback scenarios, the log files can exist in more than one server in the Cisco CallManager cluster.

Log files exist in csv format. You can read log files, except alert log file, by native NT perfmon viewer. New log files get created every day at 00:00 hours on the local system. New logs for devices, services, servers, and calls get created when the time zone changes or when a new node is added to the cluster, or during failover/fallback scenarios. The first column of all these logs comprises the time zone information and the number of minutes from the Greenwich Meridian Time (GMT). Reporter uses these log files as data source to generate daily summary reports.

Alert Log

The Alert log stores the alert, which is also stored in memory. The memory gets cleared at a constant interval, leaving the last 30 minutes of data in the memory. When the service starts/restarts, the last 30 minutes of alert data loads into the memory by the system reading from the alert logs that exist in all servers in the cluster. The alert data in the memory gets sent to the RTMT clients on request.

Upon RTMT startup, RTMT shows all logs that occurred in the last 30 minutes in the Alert Central log history. Alert log periodically gets updated and new logs get inserted into the log history window. After the number of logs reaches 100, RTMT removes the oldest 40 logs.

The file name format of the alert log will be AlertLog_MM_DD_YYYY_hh_mm.csv.

The alert log includes the following attributes:

Time Stamp—Time when RTMT logs the data

Alert Name—Descriptive name of the alert

Node ID—Node name for where RTMT raised the alert

Alert Message—Detailed description about the alert

Monitored Object Name—Name of the object monitored

Severity—Severity of the alert

PollValue—Value of the monitored object where the alert condition occurred

Action—Alert action taken

The first line of each log file comprises the header. Details of each alert gets written in a single line, separated by a comma.

Server Log

The server data accumulates in the memory whenever the LogServer API is called. Every 5 minutes, the data gets logged into the file as a single record, and RTMT cleans the memory.

The system logs the data every 5 minutes for the following counters, based on the following calculation:

cpuUsage—Average of all the values that were collected in the last 5 minutes

MemoryInUse—Average of all the values that were collected in the last 5 minutes

DiskSpaceInUse—Average of all the values that were collected in the last 5 minutes

RISDC logs the server data in windows Performance tool-compatible csv format. The header of the log comprises the time zone information and a set of columns with the previous counters for a Cisco CallManager node. These sets of columns get repeated for every node.

The following file name format of the server log applies: ServerLog_MM_DD_YYYY_hh_mm.csv.

The first line of each log file comprises the header.

Service Log

The service data accumulates in the memory whenever RTMT calls the LogService API. Every 5 minutes, RTMT logs the data into the file as a single record and cleans the memory.

The data gets logged every 5 minutes for the following counters, based on the following calculation:

ctiOpenDevices—Average of all the values that were collected in last 5 minutes

ctiLines—Average of all the values that were collected in last 5 minutes

ctiConnections—Average of all the values that were collected in last 5 minutes

ctiActiveCMLinks—Average of all the values that were collected in last 5 minutes

tftpRequests—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

tftpAbortedRequests—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

RISDC logs the service data in windows Performance tool-compatible scv format. The header of the log comprises the time zone information and a set of columns with the counters that were previously listed for a Cisco CallManager node. These sets of columns repeat for every node.

The following file name format of the Service Log applies: ServiceLog_MM_DD_YYYY_hh_mm.csv.

The first line of each log comprises the header.

Call Log

The system accumulates call data in the memory whenever RTMT calls the LogCall API. Every 5 minutes, RTMT logs the data into the file as a single record and cleans the memory.

The system logs data every 5 minutes for the following counters on the basis of the following calculation:

cmCallsAttempted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

cmCallsCompleted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

cmCallsInProgress—Average of all the values that were collected in last 5 minutes

gwMGCP_FXS_CallsCompleted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

gwMGCP_FXO_CallsCompleted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

gwMGCP_PRI_CallsCompleted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

gwMGCP_T1_CAS_CallsCompleted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

gwH323_CallsAttempted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

gwH323_CallsInProgress—Average of all the values that were collected in last 5 minutes

gwH323_CallsCompleted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

trunkH323_CallsAttempted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

trunkH323_CallsInProgress—Average of all the values collected in last 5 minutes

trunkH323_CallsCompleted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

trunkSIP_CallsAttempted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

trunkSIP_CallsInProgress—Average of all the values that were collected in last 5 minutes

trunkSIP_CallsCompleted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

gwMGCP_FXS_PortsInService—Average of all the values that were collected in last 5 minutes

gwMGCP_FXO_PortsInService—Average of all the values that were collected in lasts 5 minutes

gwMGCP_PRI_SpansInService—Average of all the values that were collected in last 5 minutes

gwMGCP_T1_CAS_SpansInService—Average of all the values that were collected in last 5 minutes

gwMGCP_FXS_ActivePorts—Average of all the values that were collected in last 5 minutes

gwMGCP_FXO_ActivePorts—Average of all the values that were collected in last 5 minutes

gwMGCP_PRI_ActiveChannels—Average of all the values that were collected in last 5 minutes

gwMGCP_T1_CAS_ActiveChannels—Average of all the values that were collected in last 5 minutes

RISDC logs the call data in windows Performance tool-compatible csv format. The header of the log comprises the time zone information and a set of columns with the previously listed counters for a node. These sets of columns repeat for every node.

The following file name format of the Call Log is CallLog_MM_DD_YYYY_hh_mm.csv.

The first line of each log file is the header.

Device Log

The device data accumulates in the memory whenever RTMT calls the LogDevice API. Every 5 minutes, RTMT logs the data into the file as a single record and cleans the memory.

The data gets logged every 5 minutes for the following counters based on the following calculation:

gatewayDevicesFXS—Average of all the values that were collected in last 5 minutes

gatewayDevicesFXO—Average of all the values that were collected in last 5 minutes

gatewayDevicesPRI—Average of all the values that were collected in last 5 minutes

gatewayDevicesT1—Average of all the values that were collected in last 5 minutes

gatewayDevicesH323—Average of all the values that were collected in last 5 minutes

RISDC logs the device data in windows Performance tool-compatible csv format. The header of the log comprises the time zone information and a set of columns with the previously listed counters for a node. These sets of columns repeat for every node.

The following file name format of the Device Log applies: DeviceLog_MM_DD_YYYY_hh_mm.csv.

The first line of each log file comprises the header.

Perfmon Log

The Perfmon data is logged whenever RTMT calls the LogPerfMon API. The file log is compatible with the Windows Performance tool csv format and can be opened using the Performance tool for analysis.

When you add new counters, RTMT changes the header to accommodate the new counters and logs the values correspondingly. When data is unavailable for an already existing counter (already added to header), then RTMT inserts blank values in the file. If the character length of the new counters added is greater than 2000, then the new file gets generated with all the counters.

The file name format of the PerfMon Log is PerfMonLog_<NodeName>_MM_DD_YYYY_hh_mm.csv.

The following lists comprises the perfmon counters that RTMT logs:

At System level:

Cisco CallManager System Performance\QueueSignalsPresent 1-High

Cisco CallManager System Performance\QueueSignalsPresent 2-Normal

Cisco CallManager System Performance\QueueSignalsPresent 3-Low

Cisco CallManager System Performance\QueueSignalsPresent 4-Lowest

Cisco CallManager System Performance\QueueSignalsProcessed 1-High

Cisco CallManager System Performance\QueueSignalsProcessed 2-Normal

Cisco CallManager System Performance\QueueSignalsProcessed 3-Low

Cisco CallManager System Performance\QueueSignalsProcessed 4-Lowest

For each process that is running on the system:

Process\% Processor Time

Process\ID Process

Process\Private Bytes

Process\Virtual Bytes

Where to Find More Information

Related Topics

Chapter 12, "Microsoft Performance"

Chapter 18, "Simple Network Management Protocol"

Appendix A, "Cisco CallManager Performance Counters, RTMT, and CISCO-CCM-MIB"

Chapter 12, "Real-Time Monitoring Configuration," Cisco CallManager Serviceability Administration Guide

Chapter 23, "Microsoft Performance," Cisco CallManager Serviceability Administration Guide

Chapter 31, "Simple Network Management Protocol Configuration," Cisco CallManager Serviceability Administration Guide