Implementing System Logging

This module describes the tasks you need to implement logging services on the router.

The Cisco IOS XR Software provides basic logging services. Logging services provide a means to gather logging information for monitoring and troubleshooting, to select the type of logging information captured, and to specify the destinations of captured system logging (syslog) messages.

Feature History for Implementing System Logging

Release

Modification

Release 7.0.11

This feature was introduced.

Implementing System Logging

System Logging (Syslog) is the standard application used for sending system log messages. Log messages indicates the health of the device and point to any encountered problems or simplify notification messages according to the severity level. The IOS XR router sends its syslog messages to a syslog process. By default, syslog messages will be sent to the console terminal. But, syslog messages can be send to destinations other than the console such as the logging buffer, syslog servers, and terminal lines.

Syslog Message Format

By default, the general format of syslog messages generated by the syslog process on the Cisco IOS XR software is as follows: 

node-id : timestamp : process-name [pid] : % message category -group -severity -message -code : message-text

The following table describes the general format of syslog messages on Cisco IOS XR software.

Table 1. Format of Syslog Messages

Field

Description

node-id

Node from which the syslog message originated.

timestamp

Time stamp in the month day HH:MM:SS format, indicating when the message was generated.

Note

 

The time-stamp format can be modified using the service timestamps command.

process-name

Process that generated the syslog message.

[ pid ]

Process ID (pid) of the process that generated the syslog message.

%message -group -severity -message-code

Message category, group name, severity, and message code associated with the syslog message.

message-text

Text string describing the syslog message.

Syslog Message Severity Levels

In the case of logging destinations such as console terminal, syslog servers and terminal lines, you can limit the number of messages sent to a logging destination by specifying the severity level of syslog messages. However, for the logging buffer destination, syslog messages of all severity will be sent to it irrespective of the specified severity level. In this case, the severity level only limits the syslog messages displayed in the output of the command show logging , at or below specified value. The following table lists the severity level keywords that can be supplied for the severity argument and the corresponding UNIX syslog definitions in order from the most severe level to the least severe level.

Table 2. Syslog Message Severity Levels

Severity Keyword

Level

Description

emergencies

0

System unusable

alert

1

Immediate action needed

critical

2

Critical conditions

errors

3

Error conditions

warnings

4

Warning conditions

notifications

5

Normal but significant condition

informational

6

Informational messages only

debugging

7

Debugging messages

Prerequisites for Configuring System Logging

These prerequisites are required to configure the logging of system messages in your network operating center (NOC):

  • You must be in a user group associated with a task group that includes the proper task IDs. The command reference guides include the task IDs required for each command. If you suspect user group assignment is preventing you from using a command, contact your AAA administrator for assistance.
  • You must have connectivity with syslog servers to configure syslog server hosts as the recipients for syslog messages.

Syslog Messages Sent to Syslog Servers

The Cisco IOS XR Software provides these features to help manage syslog messages sent to syslog servers:

UNIX System Logging Facilities

You can configure the syslog facility in which syslog messages are sent by using the logging facility command. Consult the operator manual for your UNIX operating system for more information about these UNIX system facilities. The syslog format is compatible with Berkeley Standard Distribution (BSD) UNIX version 4.3.

This table describes the facility type keywords that can be supplied for the type argument.

Table 3. Logging Facility Type Keywords

Facility Type Keyword

Description

auth

Indicates the authorization system.

cron

Indicates the cron facility.

daemon

Indicates the system daemon.

kern

Indicates the Kernel.

local0–7

Reserved for locally defined messages.

lpr

Indicates line printer system.

mail

Indicates mail system.

news

Indicates USENET news.

sys9

Indicates system use.

sys10

Indicates system use.

sys11

Indicates system use.

sys12

Indicates system use.

sys13

Indicates system use.

sys14

Indicates system use.

syslog

Indicates the system log.

user

Indicates user process.

uucp

Indicates UNIX-to-UNIX copy system.

Hostname Prefix Logging

To help manage system logging messages sent to syslog servers, Cisco IOS XR Software supports hostname prefix logging. When enabled, hostname prefix logging appends a hostname prefix to syslog messages being sent from the router to syslog servers. You can use hostname prefixes to sort the messages being sent to a given syslog server from different networking devices.

To append a hostname prefix to syslog messages sent to syslog servers, use the logging hostname command in mode.

Configuration Example
This example shows how to add the hostname prefix host1 to messages sent to the syslog servers from the router.
RP/0/RP0/CPU0:Router# configure
RP/0/RP0/CPU0:Router(config)# logging hostnameprefix host1
RP/0/RP0/CPU0:Router(config)# commit

Syslog Source Address Logging

By default, a syslog message contains the IP address (IPv4 and IPv6 are supported) of the interface it uses to leave the router when sent to syslog servers. To set all syslog messages to contain the same IP address, regardless of which interface the syslog message uses to exit the router, use the logging source-interface command in mode.

Configuration Example
This example shows how to specify that the IP address for HundredGigE interface 0/1/0/0 be set as the source IP address for all messages.
RP/0/RP0/CPU0:Router# configure
RP/0/RP0/CPU0:Router(config)# logging source-interface HundredGigE interface 0/1/0/0
RP/0/RP0/CPU0:Router(config)# commit

Configuring System Logging

Perform the tasks in this section for configuring system logging as required.

Configuring Logging to the Logging Buffer

Syslog messages can be sent to multiple destinations including an internal circular buffer known as logging buffer. You can send syslog messages to the logging buffer using the logging buffered command.

Configuration Example
This example shows the configuration for sending syslog messages to the logging buffer. The size of the logging buffer is configured as 3000000 bytes. The default value for the size of the logging buffer is 2097152 bytes. 
RP/0/RP0/CPU0:Router# configure
RP/0/RP0/CPU0:Router(config)# logging buffered 3000000
RP/0/RP0/CPU0:Router(config)# commit

Configuring Logging to a Remote Server

Syslog messages can be sent to destinations other than the console, such as logging buffer, syslog servers, snmp server and terminal lines. You can send syslog messages to an external syslog server by specifying the ip address or hostname of the syslog server using the logging command. Also you can configure the syslog facility in which syslog messages are send by using the logging facility command.

The following table list the features supported by Cisco IOS XR Software to help managing syslog messages sent to syslog servers.

Table 4. Features for Managing Syslog Messages

Features

Description

UNIX system log facility

Facility is the identifier used by UNIX to describe the application or process that submitted the log message. You can configure the syslog facility in which syslog messages are sent by using the logging facility command.

Hostname prefix logging

Cisco IOS XR Software supports hostname prefix logging. When enabled, hostname prefix logging appends a hostname prefix to syslog messages being sent from the router to syslog servers. You can use hostname prefixes to sort the messages being sent to a given syslog server from different networking devices. Use the logging hostname command to append a hostname prefix to syslog messages sent to syslog servers

Syslog source address logging

By default, a syslog message sent to a syslog server contains the IP address of the interface it uses to leave the router. Use the logging source-interface command to set all syslog messages to contain the same IP address, regardless of which interface the syslog message uses to exit the router.

Configuration Example for Logging to Syslog Server

This example shows the configuration for sending syslog messages to an external syslog server. The ip address 209.165.201.1 is configured as the syslog server. 

Router# configure
Router(config)# logging 209.165.201.1 vrf default
Router(config)# logging facility kern (optional)
Router(config)# logging hostnameprefix 203.0.113.1 (optional)
Router(config)# logging source-interface HundredGigE 0/0/0/0 (optional) 
Router(config)# commit
Configuration Example for Logging to SNMP Server

This example shows the configuration for sending syslog messages to an SNMP server. The logging trap command is used to limit the logging of messages sent to the snmp servers based on severity. 

Router# configure
Router(config)# snmp-server traps syslog
Router(config)# logging trap warnings
Router(config)# commit

For more information on SNMP server configurations, see the Configuring Simple Network Management Protocol chapter in the System Management Configuration Guide for Cisco 8000 Series Routers

Related Topics

Configuring Logging to Terminal Lines

By default syslog messages will be sent to the console terminal. But, syslog messages can also be send to terminal lines other than the console. You can send syslog messages to the logging buffer using the logging monitor command.

Configuration Example
This example shows the configuration for sending syslog messages to terminal lines other than console. In this example, severity level is configured as critical. The terminal monitor command is configured to display syslog messages during a terminal session. The default severity level is debugging. 
RP/0/RP0/CPU0:Router# configure
RP/0/RP0/CPU0:Router(config)# logging monitor critical
RP/0/RP0/CPU0:Router(config)# commit
RP/0/RP0/CPU0:Router# terminal monitor

Enable Message Logs for Third-Party Software Containers

Table 5. Feature History Table

Feature Name

Release Information

Feature Description

Message Logs for Third-Party Software Containers

Release 7.3.15

 This feature introduces the logging container all command to monitor messages from a third-party container logs, such as Docker.

Cisco IOS XR operating system can host third-party software containers, such as Docker. To monitor logs from such software containers, use the logging container all command.

Configuration Example

This example shows how to enable third-party software container logging and how to view the logs for the third-party software container named Docker: 


Router# configure
Router(config)# logging container all
Router(config)# commit


Router# show running-config logging

logging container all

Router# show logging | inc DOCKER
Syslog logging: enabled (0 messages dropped, 0 flushes, 0 overruns)
    Console logging: level warnings, 5 messages logged
    Monitor logging: level debugging, 0 messages logged
    Trap logging: level informational, 0 messages logged
    Buffer logging: level debugging, 148 messages logged

Log Buffer (2097152 bytes):

RP/0/RP0/CPU0:Mar  5 06:56:11.913 UTC: exec[66927]: %SECURITY-LOGIN-6-AUTHEN_SUCCESS : Successfully authenticated user 'lab' from 'console' on 'con0_RP0_CPU0'
RP/0/RP0/CPU0:Mar  5 06:58:13.053 UTC: config[66985]: %MGBL-SYS-5-CONFIG_I : Configured from console by lab 
RP/0/RP0/CPU0:Mar  5 06:59:04.775 UTC: ubuntu-1[67232]: %OS-SYSLOG-6-DOCKER_APP : ^[]0;root@c382b2e7bed6: /^Groot@c382b2e7bed6:/# testlog 
RP/0/RP0/CPU0:Mar  5 06:59:04.830 UTC: config[67139]: %MGBL-CONFIG-6-DB_COMMIT : Configuration committed by user 'lab'. Use 'show configuration commit changes 1000000012' to view the changes. 
RP/0/RP0/CPU0:Mar  5 06:59:45.028 UTC: config[67139]: %MGBL-SYS-5-CONFIG_I : Configured from console by lab 
RP/0/RP0/CPU0:Mar  5 06:59:48.552 UTC: run_cmd[67780]: %INFRA-INFRA_MSG-5-RUN_LOGIN : User lab logged into shell from con0/RP0/CPU0 
RP/0/RP0/CPU0:Mar  5 06:59:56.073 UTC: ubuntu-1[67976]: %OS-SYSLOG-6-DOCKER_APP : testlog-123 
RP/0/RP0/CPU0:Mar  5 07:00:12.471 UTC: ubuntu-1[68099]: %OS-SYSLOG-6-DOCKER_APP : testlog-new1 
RP/0/RP0/CPU0:Mar  5 07:01:55.747 UTC: ubuntu-1[68245]: %OS-SYSLOG-6-DOCKER_APP : testlog-new1 
RP/0/RP0/CPU0:Mar  5 07:02:02.869 UTC: run_cmd[67780]: %INFRA-INFRA_MSG-5-RUN_LOGOUT : User lab logged out of shell from con0/RP0/CPU0

Modifying Logging to Console Terminal

By default syslog messages will be sent to the console terminal. You can modify the logging of syslog messages to the console terminal

Configuration Example

This example shows how to modify the logging of syslog messages to the console terminal. 

RP/0/RP0/CPU0:Router# configure
RP/0/RP0/CPU0:Router(config)# logging console alerts
RP/0/RP0/CPU0:Router(config)# commit

Modifying Time Stamp Format

By default, time stamps are enabled for syslog messages. Time stamp is generated in the month day HH:MM:SS format indicating when the message was generated.

Configuration Example

This example shows how to modify the time-stamp for syslog and debugging messages. 

RP/0/RP0/CPU0:Router# configure
RP/0/RP0/CPU0:Router(config)# service timestamps log datetime localtime msec or service timestamps log uptime
RP/0/RP0/CPU0:Router(config)# service timestamps debug datetime msec show-timezone or service timestamps debug uptime
RP/0/RP0/CPU0:Router(config)# commit

Suppressing Duplicate Syslog Messages

Suppressing duplicate messages, especially in a large network, can reduce message clutter and simplify the task of interpreting the log. The duplicate message suppression feature substantially reduces the number of duplicate event messages in both the logging history and the syslog file.

Configuration Example

This example shows how to suppress the consecutive logging of duplicate syslog messages. 

RP/0/RP0/CPU0:Router# configure
RP/0/RP0/CPU0:Router(config)# logging suppress duplicates
RP/0/RP0/CPU0:Router(config)# commit

Displaying System Logging Messages

You can display the syslog messages stored in the logging buffer by using the show logging command.

Configuration Example

This example shows how to display the syslog messages stored in the logging buffer.

RP/0/RP0/CPU0:Router# show logging
RP/0/RP0/CPU0:Router# show logging location 0/1/CPU0
RP/0/RP0/CPU0:Router# show logging process init
RP/0/RP0/CPU0:Router# show logging string install
RP/0/RP0/CPU0:Router# show logging start december 1 10:30:00
RP/0/RP0/CPU0:Router# show logging end december 2 22:16:00

Note

The commands can be entered in any order.

 
RP/0/RP0/CPU0:Router# show logging
Syslog logging: enabled (0 messages dropped, 0 flushes, 0 overruns)
    Console logging: level warnings, 0 messages logged
    Monitor logging: level debugging, 0 messages logged
    Trap logging: level informational, 0 messages logged
    Buffer logging: level debugging, 82 messages logged

Log Buffer (307200 bytes):

RP/0/0/CPU0:Feb  7 15:36:16.655 IST: init[68452]: %OS-INIT-7-MBI_STARTED : total time 0.215 seconds
RP/0/0/CPU0:Feb  7 15:36:16.759 IST: sysmgr[55]: %OS-SYSMGR-5-NOTICE : Card is COLD started
RP/0/0/CPU0:Feb  7 15:36:16.893 IST: init[68452]: %OS-INIT-7-INSTALL_READY : total time 0.453 seconds
RP/0/0/CPU0:Feb  7 15:36:17.125 IST: sysmgr[278]: %OS-SYSMGR-6-INFO : Backup system manager is ready
RP/0/0/CPU0:Feb  7 15:36:17.149 IST: syslogd[405]: %SECURITY-XR_SSL-6-INFO : XR SSL info: Setting fips register
RP/0/0/CPU0:Feb  7 15:36:17.177 IST: spp[52]: %L2-VTIO-6-NO_PORTS : Plug-in found no ports to manage
RP/0/0/CPU0:Feb  7 15:36:18.651 IST: dsc[210]: %PLATFORM-DSC-6-INFO_I_AM_DSC : Setting myself as DSC
RP/0/0/CPU0:Feb  7 15:36:18.653 IST: sysmgr[55]: %OS-SYSMGR-7-DEBUG : node set to DSC

RP/0/RP0/CPU0:Router# show logging process smartlicserver
Syslog logging: enabled (0 messages dropped, 0 flushes, 0 overruns)
    Console logging: level warnings, 0 messages logged
    Monitor logging: level debugging, 0 messages logged
    Trap logging: level informational, 0 messages logged
    Buffer logging: level debugging, 82 messages logged

Log Buffer (307200 bytes):

RP/0/0/CPU0:Feb  7 15:37:12.434 IST: smartlicserver[119]: %SMART_LIC-6-AGENT_ENABLED:Smart Agent for Licensing is enabled
RP/0/0/CPU0:Feb  7 15:37:23.029 IST: smartlicserver[119]: %SMART_LIC-6-REPORTING_REQUIRED:A Usage report acknowledgement will be required in 353 days.
RP/0/0/CPU0:Feb  7 15:37:24.030 IST: smartlicserver[119]: %SMART_LIC-5-IN_COMPLIANCE:All entitlements and licenses in use on this device are authorized
RP/0/0/CPU0:Feb  7 15:37:29.474 IST: smartlicserver[119]: %SMART_LIC-5-SLR_IN_COMPLIANCE:The entitlement regid.2019-03.com.cisco.ENXR-TRK,1.0_2b015ca9-b01d-40eb-80b6-e6647f8fcf76 in use on this device is authorized
RP/0/0/CPU0:Feb  7 15:37:38.045 IST: smartlicserver[119]: %SMART_LIC-3-COMM_FAILED:Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name
RP/0/0/CPU0:Feb  7 15:37:39.047 IST: smartlicserver[119]: %SMART_LIC-5-IN_COMPLIANCE:All entitlements and licenses in use on this device are authorized
RP/0/0/CPU0:ios#

RP/0/RP0/CPU0:Router# show logging start february 7 15:37:24
Syslog logging: enabled (0 messages dropped, 0 flushes, 0 overruns)
    Console logging: level warnings, 0 messages logged
    Monitor logging: level debugging, 0 messages logged
    Trap logging: level informational, 0 messages logged
    Buffer logging: level debugging, 82 messages logged

Log Buffer (307200 bytes):

RP/0/0/CPU0:Feb  7 15:37:24.030 IST: smartlicserver[119]: %SMART_LIC-5-IN_COMPLIANCE:All entitlements and licenses in use on this device are authorized
RP/0/0/CPU0:Feb  7 15:37:24.112 IST: iedged[462]: %SUBSCRIBER-SUB_UTIL-5-SESSION_THROTTLE : Subscriber Infra is ready. Reason: [V6 Subscriber infra process(es) is available].
RP/0/0/CPU0:Feb  7 15:37:24.112 IST: subdb_svr[207]: %SUBSCRIBER-SUB_UTIL-5-SESSION_THROTTLE : Subscriber Infra is ready. Reason: [V6 Subscriber infra process(es) is available].
RP/0/0/CPU0:Feb  7 15:37:24.112 IST: pppoe_ma[216]: %SUBSCRIBER-SUB_UTIL-5-SESSION_THROTTLE : Subscriber Infra is ready. Reason: [V6 Subscriber infra process(es) is available].
RP/0/0/CPU0:Feb  7 15:37:29.474 IST: smartlicserver[119]: %SMART_LIC-5-SLR_IN_COMPLIANCE:The entitlement regid.2019-03.com.cisco.ENXR-TRK,1.0_2b015ca9-b01d-40eb-80b6-e6647f8fcf76 in use on this device is authorized
RP/0/0/CPU0:Feb  7 15:37:38.045 IST: smartlicserver[119]: %SMART_LIC-3-COMM_FAILED:Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name
RP/0/0/CPU0:Feb  7 15:37:39.047 IST: smartlicserver[119]: %SMART_LIC-5-IN_COMPLIANCE:All entitlements and licenses in use on this device are authorized
RP/0/0/CPU0:Feb  7 15:46:19.976 IST: /pkg/sbin/sysmgr_log[68415]: %OS-SYSMGR-7-CHECK_LOG : /pkg/bin/sysmgr_debug_script invoked for : (ltrace_data_export) sysmgr_level_ready_timeout: EOI required, but never received from ltrace_data_export,jid=293   Output is in /disk0://sysmgr_debug/debug.node0_0_CPU0.174906
RP/0/0/CPU0:ios#

Archiving System Logging Messages to a Local Storage Device

Syslog messages can also be saved to an archive on a local storage device, such as the hard disk or a flash disk. Messages can be saved based on severity level, and you can specify attributes such as the size of the archive, how often messages are added (daily or weekly), and how many total weeks of messages the archive will hold.

You can create a logging archive and specify how the logging messages will be collected and stored by using the logging archive command.

The following table lists the commands used to specify the archive attributes once you are in the logging archive submode.

Table 6. Commands Used to Set Syslog Archive Attributes

Features

Description

archive-length weeks

Specifies the maximum number of weeks that the archive logs are maintained in the archive. Any logs older than this number are automatically removed from the archive.

archive-size size

Specifies the maximum total size of the syslog archives on a storage device. If the size is exceeded then the oldest file in the archive is deleted to make space for new logs.

device {disk0 | disk1 | harddisk}

Specifies the local storage device where syslogs are archived. By default, the logs are created under the directory <device>/var/log. If the device is not configured, then all other logging archive configurations are rejected. We recommend that syslogs be archived to the harddisk because it has more capacity than flash disks.

file-size size

Specifies the maximum file size (in megabytes) that a single log file in the archive can grow to. Once this limit is reached, a new file is automatically created with an increasing serial number.

frequency {daily | weekly}

Specifies if logs are collected on a daily or weekly basis.

severity severity

Specifies the minimum severity of log messages to archive. All syslog messages greater than or equal to this configured level are archived while those lesser than this are filtered out.

threshold

Specifics the threshold percentage for archive logs.
Configuration Example

This example shows how to save syslog messages to an archive on a local storage device. 

RP/0/RP0/CPU0:Router# configure
RP/0/RP0/CPU0:Router(config)# logging archive
RP/0/RP0/CPU0:Router(config-logging-arch)# device disk1
RP/0/RP0/CPU0:Router(config-logging-arch)# frequency weekly
RP/0/RP0/CPU0:Router(config-logging-arch)# severity warnings
RP/0/RP0/CPU0:Router(config-logging-arch)# archive-length 6
RP/0/RP0/CPU0:Router(config-logging-arch)# archive-size 50
RP/0/RP0/CPU0:Router(config-logging-arch)# file-size 10
RP/0/RP0/CPU0:Router(config)# commit

Platform Automated Monitoring

Platform Automated Monitoring (PAM) is a system monitoring tool integrated into Cisco IOS XR software image to monitor the following issues:

  • process crashes

  • memory leaks

  • CPU hogs

  • tracebacks

  • disk usage

PAM is enabled by default. When the PAM tool detects any of these system issues, it collects the required data to troubleshoot the issue, and generates a syslog message stating the issue. The auto-collected troubleshooting information is then stored as a separate file in harddisk:/cisco_support/ or in /misc/disk1/cisco_support/ directory.

Table 7. Feature History Table

Feature Name

Release

Description

Platform Automated Monitoring for Blocked Processes

Release 7.5.2

You can enable the Platform Automated Monitoring tool integrated into the Cisco IOS XR software image and receive alerts if any process is blocked. Several system failures can cause a blocked process, such as memory leak, network connection loss, and so on.

The tool collects the required data to troubleshoot the issue and generates a system log message with the name of the process that is currently blocked.

This feature introduces the following commands:

PAM Events

When PAM detects a process crash, traceback, potential memory leak, CPU hog, a full file system, or blocked process on any node, it automatically collects logs and saves these logs (along with the core file in applicable cases) as a .tgz file in harddisk:/cisco_support/ or in /misc/disk1/cisco_support/ directory. PAM also generates a syslog message with severity level as warning, mentioning the respective issue.

The format of the .tgz file is: PAM-<platform>-<PAM event>-<node-name>-<PAM process>-<YYYYMMDD>-<checksum>.tgz .For example, PAM-cisco8000-crash-xr_0_RP0_CPU0-ipv4_rib-2016Aug16-210405.tgz is the file collected when PAM detects a process crash.

Because PAM assumes that core files are saved to the default archive folder (harddisk:/ or /misc/disk1/), you must not modify the location of core archive (by configuring exception filepath) or remove the core files generated after PAM detects an event. Else, PAM does not detect the process crash. Also, once reported, the PAM does not report the same issue for the same process in the same node again.

For the list of commands used while collecting logs, refer Files Collected by PAM Tool.

The Platform Automated Monitoring for blocked processes detects and alerts if any of the processes are blocked, except for the processes which are expected to be blocked by their design. These processes are listed in the table below:

Blocked process

Blocked on

lpts_fm

lpts_pa

isis

lspv_server

Ospf

lspv_server

l2vpn_mgr

lspv_server

mpls_ldp

lspv_server

bgp

lspv_server

te_control

lspv_server

xtc_agent

lspv_server

The sections below describe the main PAM events:

Crash Monitoring

The PAM monitors process crash for all nodes, in real time. This is a sample syslog generated when the PAM detects a process crash: 


RP/0/RP0/CPU0:Aug 16 21:04:06.442 : logger[69324]: %OS-SYSLOG-4-LOG_WARNING : PAM detected crash for ipv4_rib on 0_RP0_CPU0. 
All necessary files for debug have been collected and saved at 
0/RP0/CPU0 : harddisk:/cisco_support/PAM-cisco8000-crash-xr_0_RP0_CPU0-ipv4_rib-2016Aug16-210405.tgz 
Please copy tgz file out of the router and send to Cisco support. This tgz file will be removed after 14 days.)
Traceback Monitoring

The PAM monitors tracebacks for all nodes, in real time. This is a sample syslog generated when the PAM detects a traceback: 


RP/0/RP0/CPU0:Aug 16 21:42:42.320 : logger[66139]: %OS-SYSLOG-4-LOG_WARNING : PAM detected traceback for ipv4_rib on 0_RP0_CPU0. 
All necessary files for debug have been collected and saved at 
0/RP0/CPU0 : harddisk:/cisco_support/PAM-cisco8000-traceback-xr_0_RP0_CPU0-ipv4_rib-2016Aug16-214242.tgz 
Please copy tgz file out of the router and send to Cisco support. This tgz file will be removed after 14 days.)
Memory Usage Monitoring

The PAM monitors the process memory usage for all nodes. The PAM detects potential memory leaks by monitoring the memory usage trend and by applying a proprietary algorithm to the collected data. By default, it collects top output on all nodes periodically at an interval of 30 minutes.

This is a sample syslog generated when the PAM detects a potential memory leak: 


RP/0/RP0/CPU0:Aug 17 05:13:32.684 : logger[67772]: %OS-SYSLOG-4-LOG_WARNING : PAM detected significant memory increase 
(from 13.00MB at 2016/Aug/16/20:42:41 to 28.00MB at 2016/Aug/17/04:12:55) for pam_memory_leaker on 0_RP0_CPU0. 
All necessary files for debug have been collected and saved at 
0/RP0/CPU0 : harddisk:/cisco_support/PAM-cisco8000-memory_leak-xr_0_RP0_CPU0-pam_memory_leaker-2016Aug17-051332.tgz 
(Please copy tgz file out of the router and send to Cisco support. This tgz file will be removed after 14 days.)
CPU Monitoring

The PAM monitors CPU usage on all nodes periodically at an interval of 30 minutes. The PAM reports a CPU hog in either of these scenarios:

  • When a process constantly consumes high CPU (that is, more than the threshold of 90 percentage)

  • When high CPU usage lasts for more than 60 minutes

This is a sample syslog generated when the PAM detects a CPU hog: 


RP/0/RP0/CPU0:Aug 16 00:56:00.819 : logger[68245]: %OS-SYSLOG-4-LOG_WARNING : PAM detected CPU hog for cpu_hogger on 0_RP0_CPU0. 
All necessary files for debug have been collected and saved at 0/RP0/CPU0 : 
harddisk:/cisco_support/PAM-cisco8000-cpu_hog-xr_0_RP0_CPU0-cpu_hogger-2016Aug16-005600.tgz 
(Please copy tgz file out of the router and send to Cisco support. This tgz file will be removed after 14 days.)
RP/0/RP0/CPU0:Jun 21 15:33:54.517 : logger[69042]: %OS-SYSLOG-1-LOG_ALERT : PAM detected ifmgr is hogging CPU on 0_RP0_CPU0!
File System Monitoring

The PAM monitors disk usage on all nodes periodically at an interval of 30 minutes. This is a sample syslog generated when the PAM detects that a file system is full: 


RP/0/RP0/CPU0:Jun 20 13:59:04.986 : logger[66125]: %OS-SYSLOG-4-LOG_WARNING : PAM detected /misc/config is full on 0_1_CPU0 
(please clean up to avoid any fault caused by this). All necessary files for debug have been collected and saved at 
0/RP0/CPU0 : harddisk:/cisco_support/PAM-cisco8000-disk_usage-xr_0_1_CPU0-2016Jun20-135904.tgz 
(Please copy tgz file out of the router and send to Cisco support. This tgz file will be removed after 14 days.)

Disable and Re-enable PAM

The PAM tool consists of the following monitoring processes:

  • monitor_cpu.pl

  • monitor_crash.pl

  • monitor_show_logging.pl

  • monitor_process.pl


    Note

    Monitor process.pl in PAM monitors all nodes and generates a system log message with the process name that is blocked if it detects any process is blocked for more than 30 minutes. It prevents multiple alarms for the same blocked process.

Before disabling or re-enabling the PAM, use these options to check if the PAM is installed in the router:

  • From Cisco IOS XR Command Line Interface: 

    
Router# show pam status
Tue Jun 14 17:58:42.791 UTC
PAM is enabled

  • From router shell prompt: 

    
Router# run ps auxw|egrep perl

root     12559  0.0  0.0  57836 17992 ?        S    Apr24   0:00 /usr/bin/perl /pkg/opt/cisco/pam//pam_plugin.pl
Disable PAM

To disable PAM agent systemwide, execute the following command from the XR EXEC mode: 

Router# disable-pam
Re-enable PAM

To re-enable PAM agent systemwide, execute the following command from XR EXEC mode: 

Router# enable-pam

Data Archiving in PAM

At any given point of time, PAM does not occupy more than 200 MB of harddisk: space. If more than 200 MB is needed, then PAM archives old files and rotates the logs automatically.

The PAM collects CPU or memory usage (using top -b -n1 command) periodically at an interval of 30 minutes. The files are saved under harddisk:/cisco_support/ directory with the filename as <node name>.log (for example, harddisk:/cisco_support/xr-0_RP0_CPU0.log). When the file size exceeds the limit of 15MB, the file is archived (compressed) into .tgz file, and then rotated for a maximum of two counts (that is, it retains only two .tgz files). The maximum rotation count of .tgz files is three. Also, the old file (ASCII data) is archived and rotated if a node is reloaded. For example, xr-0_RP0_CPU0.log is archived if RP0 is reloaded.

You must not manually delete the core file generated by the PAM. The core file is named as <process name>_pid.by_user.<yyyymmdd>-<hhmmss>.<node>.<checksum>.core.gz .

Files Collected by PAM Tool

The table below lists the various PAM events and the respective commands and files collected by the PAM for each event.

You can attach the respective.tgz file when you raise a service request (SR) with Cisco Technical Support.

Event Name

Commands and Files Collected by PAM

Process crash

  • show install active

  • show platform

  • show version

  • core (gz) file

  • core.txt file

Process traceback

  • show dll

  • show install active

  • show logging

  • show platform

  • show version

Memory leak

  • show install active

  • show platform

  • show version

  • core (gz) file

  • dumpcore running

  • continuous memory usage snapshots

Show logging event

  • show install active

  • show logging

  • show platform

  • show version

  • core (gz) file

  • core.txt file

CPU hog

  • follow process

  • pstack

  • show dll

  • show install active

  • show platform

  • show version

  • top -H

  • core (gz) file

  • CPU usage snapshots

Disk usage

  • show install active

  • show platform

  • show version

  • console log

  • core (gz) file

  • Disk usage snapshots

Process Blockage

  • show version

  • show install active

  • show platform

  • show logging

  • show running-config

  • show process blocked location all

  • core (gz) file