Implementing System Logging

This module describes the tasks you need to implement logging services on the router.

The Cisco IOS XR Software provides basic logging services. Logging services provide a means to gather logging information for monitoring and troubleshooting, to select the type of logging information captured, and to specify the destinations of captured system logging (syslog) messages.

Feature History for Implementing System Logging

Release

Modification

Release 6.1.2

Platform Automated Monitoring (PAM) tool was introduced for all Cisco IOS XR 64-bit platforms.

Implementing System Logging

System Logging (Syslog) is the standard application used for sending system log messages. Log messages indicates the health of the device and point to any encountered problems or simplify notification messages according to the severity level. The IOS XR router sends its syslog messages to a syslog process. By default, syslog messages will be sent to the console terminal. But, syslog messages can be send to destinations other than the console such as the logging buffer, syslog servers, and terminal lines.

Syslog Message Format

By default, the general format of syslog messages generated by the syslog process on the Cisco IOS XR software is as follows: 

node-id : timestamp : process-name [pid] : % message category -group -severity -message -code : message-text

The following table describes the general format of syslog messages on Cisco IOS XR software.

Table 1. Format of Syslog Messages

Field

Description

node-id

Node from which the syslog message originated.

timestamp

Time stamp in the month day HH:MM:SS format, indicating when the message was generated.

Note 

The time-stamp format can be modified using the service timestamps command.

process-name

Process that generated the syslog message.

size

Process ID (pid) of the process that generated the syslog message.

[ pid ]

Message category, group name, severity, and message code associated with the syslog message.

message-text

Text string describing the syslog message.

Syslog Message Severity Levels

In the case of logging destinations such as console terminal, syslog servers and terminal lines, you can limit the number of messages sent to a logging destination by specifying the severity level of syslog messages. However, for the logging buffer destination, syslog messages of all severity will be sent to it irrespective of the specified severity level. In this case, the severity level only limits the syslog messages displayed in the output of the command show logging , at or below specified value. The following table lists the severity level keywords that can be supplied for the severity argument and the corresponding UNIX syslog definitions in order from the most severe level to the least severe level.

Table 2. Syslog Message Severity Levels

Severity Keyword

Level

Description

emergencies

0

System unusable

alert

1

Immediate action needed

critical

2

Critical conditions

errors

3

Error conditions

warnings

4

Warning conditions

notifications

5

Normal but significant condition

informational

6

Informational messages only

debugging

7

Debugging messages

Prerequisites for Configuring System Logging

These prerequisites are required to configure the logging of system messages in your network operating center (NOC):

  • You must be in a user group associated with a task group that includes the proper task IDs. The command reference guides include the task IDs required for each command. If you suspect user group assignment is preventing you from using a command, contact your AAA administrator for assistance.
  • You must have connectivity with syslog servers to configure syslog server hosts as the recipients for syslog messages.

Configuring System Logging

Perform the tasks in this section for configuring system logging as required.

Configuring Logging to the Logging Buffer

Syslog messages can be sent to multiple destinations including an internal circular buffer known as logging buffer. You can send syslog messages to the logging buffer using the logging buffered command.

Configuration Example
This example shows the configuration for sending syslog messages to the logging buffer. The size of the logging buffer is configured as 3000000 bytes. The default value for the size of the logging buffer is 2097152 bytes. 
RP/0/RP0/CPU0:Router# configure
RP/0/RP0/CPU0:Router(config)# logging buffered 3000000
RP/0/RP0/CPU0:Router(config)# commit

Configuring Logging to a Remote Server

Syslog messages can be sent to destinations other than the console, such as logging buffer, syslog servers, snmp servers and terminal lines. You can send syslog messages to an external syslog server by specifying the ip address or hostname of the syslog server using the logging command. Also you can configure the syslog facility in which syslog messages are send by using the logging facility command.

The following table list the features supported by Cisco IOS XR Software to help managing syslog messages sent to syslog servers.

Table 3. Features for Managing Syslog Messages

Features

Description

UNIX system log facility

Facility is the identifier used by UNIX to describe the application or process that submitted the log message. You can configure the syslog facility in which syslog messages are sent by using the logging facility command.

Hostname prefix logging

Cisco IOS XR Software supports hostname prefix logging. When enabled, hostname prefix logging appends a hostname prefix to syslog messages being sent from the router to syslog servers. You can use hostname prefixes to sort the messages being sent to a given syslog server from different networking devices. Use the logging hostname command to append a hostname prefix to syslog messages sent to syslog servers

Syslog source address logging

By default, a syslog message sent to a syslog server contains the IP address of the interface it uses to leave the router. Use the logging source-interface command to set all syslog messages to contain the same IP address, regardless of which interface the syslog message uses to exit the router.

Configuration Example for Logging to Syslog Server

This example shows the configuration for sending syslog messages to an external syslog server. The ip address 209.165.201.1 is configured as the syslog server. 

Router# configure
Router(config)# logging 209.165.201.1 vrf default
Router(config)# logging facility kern (optional)
Router(config)# logging hostnameprefix 203.0.113.1 (optional)
Router(config)# logging source-interface TenGigE 0/0/0/10 (optional) 
Router(config)# commit
Configuration Example for Logging to SNMP Server

This example shows the configuration for sending syslog messages to an SNMP server. The logging trap command is used to limit the logging of messages sent to the snmp servers based on severity. 

Router# configure
Router(config)# snmp-server traps syslog
Router(config)# logging trap warnings
Router(config)# commit

For more information on SNMP server configurations, see the Configuring Simple Network Management Protocol chapter in the System Management Configuration Guide for Cisco NCS 5000 Series Routers

Configuring Logging to Terminal Lines

By default syslog messages will be sent to the console terminal. But, syslog messages can also be send to terminal lines other than the console. You can send syslog messages to the logging buffer using the logging monitor command.

Configuration Example
This example shows the configuration for sending syslog messages to terminal lines other than console. In this example, severity level is configured as critical. The terminal monitor command is configured to display syslog messages during a terminal session. The default severity level is debugging. 
RP/0/RP0/CPU0:Router# configure
RP/0/RP0/CPU0:Router(config)# logging monitor critical
RP/0/RP0/CPU0:Router(config)# commit
RP/0/RP0/CPU0:Router# terminal monitor

Modifying Logging to Console Terminal

By default syslog messages will be sent to the console terminal. You can modify the logging of syslog messages to the console terminal

Configuration Example

This example shows how to modify the logging of syslog messages to the console terminal. 

RP/0/RP0/CPU0:Router# configure
RP/0/RP0/CPU0:Router(config)# logging console alerts
RP/0/RP0/CPU0:Router(config)# commit

Modifying Time Stamp Format

By default, time stamps are enabled for syslog messages. Time stamp is generated in the month day HH:MM:SS format indicating when the message was generated.

Configuration Example

This example shows how to modify the time-stamp for syslog and debugging messages. 

RP/0/RP0/CPU0:Router# configure
RP/0/RP0/CPU0:Router(config)# service timestamps log datetime localtime msec or service timestamps log uptime
RP/0/RP0/CPU0:Router(config)# service timestamps debug datetime msec show-timezone or service timestamps debug uptime
RP/0/RP0/CPU0:Router(config)# commit

Suppressing Duplicate Syslog Messages

Suppressing duplicate messages, especially in a large network, can reduce message clutter and simplify the task of interpreting the log. The duplicate message suppression feature substantially reduces the number of duplicate event messages in both the logging history and the syslog file.

Configuration Example

This example shows how to suppress the consecutive logging of duplicate syslog messages. 

RP/0/RP0/CPU0:Router# configure
RP/0/RP0/CPU0:Router(config)# logging suppress duplicates
RP/0/RP0/CPU0:Router(config)# commit

Archiving System Logging Messages to a Local Storage Device

Syslog messages can also be saved to an archive on a local storage device, such as the hard disk or a flash disk. Messages can be saved based on severity level, and you can specify attributes such as the size of the archive, how often messages are added (daily or weekly), and how many total weeks of messages the archive will hold. You can create a logging archive and specify how the logging messages will be collected and stored by using the logging archive command.

The following table lists the commands used to specify the archive attributes once you are in the logging archive submode.

Table 4. Commands Used to Set Syslog Archive Attributes

Features

Description

archive-length weeks

Specifies the maximum number of weeks that the archive logs are maintained in the archive. Any logs older than this number are automatically removed from the archive.

archive-size size

Specifies the maximum total size of the syslog archives on a storage device. If the size is exceeded then the oldest file in the archive is deleted to make space for new logs.

device {disk0 | disk1 | harddisk}

Specifies the local storage device where syslogs are archived. By default, the logs are created under the directory device/var/log. If the device is not configured, then all other logging archive configurations are rejected. We recommend that syslogs be archived to the harddisk because it has more capacity than flash disks.

file-size size

Specifies the maximum file size (in megabytes) that a single log file in the archive can grow to. Once this limit is reached, a new file is automatically created with an increasing serial number.

frequency {daily | weekly}

Specifies if logs are collected on a daily or weekly basis.

severity severity

Specifies the minimum severity of log messages to archive. All syslog messages greater than or equal to this configured level are archived while those lesser than this are filtered out.

Configuration Example

This example shows how to save syslog messages to an archive on a local storage device. 

RP/0/RP0/CPU0:Router# configure
RP/0/RP0/CPU0:Router(config)# logging archive
RP/0/RP0/CPU0:Router(config-logging-arch)# device disk1
RP/0/RP0/CPU0:Router(config-logging-arch)# frequency weekly
RP/0/RP0/CPU0:Router(config-logging-arch)# severity warnings
RP/0/RP0/CPU0:Router(config-logging-arch)# archive-length 6
RP/0/RP0/CPU0:Router(config-logging-arch)# archive-size 50
RP/0/RP0/CPU0:Router(config-logging-arch)# file-size 10
RP/0/RP0/CPU0:Router(config)# commit

Platform Automated Monitoring

Platform Automated Monitoring (PAM) is a system monitoring tool integrated into Cisco IOS XR software image to monitor issues such as process crash, memory leak, CPU hog, tracebacks, syslog and disk usage. PAM is enabled by default on all Cisco IOS XR 64 bit platforms. When the PAM tool detects any of these system issues, it collects the required data to troubleshoot the issue, and generates a syslog message stating the issue. The auto-collected troubleshooting information is then stored as a separate file in harddisk:/cisco_support/ or in /misc/disk1/cisco_support/ directory.

PAM Events

When PAM detects a process crash, traceback, potential memory leak, CPU hog or a full file system, it automatically collects logs and saves these logs (along with the core file in applicable cases) as a .tgz file in harddisk:/cisco_support/ or in /misc/disk1/cisco_support/ directory. PAM also generates a syslog message with severity level as warning, mentioning the respective issue.

The format of the .tgz file is: PAM-<platform>-<PAM event>-<node-name>-<PAM process>-<YYYYMMDD>-<checksum>.tgz .For example, PAM-ncs5k-crash-xr_0_RP0_CPU0-ipv4_rib-2016Aug16-210405.tgz is the file collected when PAM detects a process crash.

Because PAM assumes that core files are saved to the default archive folder (harddisk:/ or /misc/disk1/), you must not modify the location of core archive (by configuring exception filepath) or remove the core files generated after PAM detects an event. Else, PAM does not detect the process crash. Also, once reported, the PAM does not report the same issue for the same process in the same node again.

For the list of commands used while collecting logs, refer Files Collected by PAM Tool.

The sections below describe the main PAM events:

Crash Monitoring

The PAM monitors process crash for all nodes, in real time. This is a sample syslog generated when the PAM detects a process crash: 


RP/0/RP0/CPU0:Aug 16 21:04:06.442 : logger[69324]: %OS-SYSLOG-4-LOG_WARNING : PAM detected crash for ipv4_rib on 0_RP0_CPU0. 
All necessary files for debug have been collected and saved at 
0/RP0/CPU0 : harddisk:/cisco_support/PAM-ncs5k-crash-xr_0_RP0_CPU0-ipv4_rib-2016Aug16-210405.tgz 
Please copy tgz file out of the router and send to Cisco support. This tgz file will be removed after 14 days.)
Traceback Monitoring

The PAM monitors tracebacks for all nodes, in real time. This is a sample syslog generated when the PAM detects a traceback: 


RP/0/RP0/CPU0:Aug 16 21:42:42.320 : logger[66139]: %OS-SYSLOG-4-LOG_WARNING : PAM detected traceback for ipv4_rib on 0_RP0_CPU0. 
All necessary files for debug have been collected and saved at 
0/RP0/CPU0 : harddisk:/cisco_support/PAM-ncs5k-traceback-xr_0_RP0_CPU0-ipv4_rib-2016Aug16-214242.tgz 
Please copy tgz file out of the router and send to Cisco support. This tgz file will be removed after 14 days.)
Memory Usage Monitoring

The PAM monitors the process memory usage for all nodes. The PAM detects potential memory leaks by monitoring the memory usage trend and by applying a proprietary algorithm to the collected data. By default, it collects top output on all nodes periodically at an interval of 30 minutes.

This is a sample syslog generated when the PAM detects a potential memory leak: 


RP/0/RP0/CPU0:Aug 17 05:13:32.684 : logger[67772]: %OS-SYSLOG-4-LOG_WARNING : PAM detected significant memory increase 
(from 13.00MB at 2016/Aug/16/20:42:41 to 28.00MB at 2016/Aug/17/04:12:55) for pam_memory_leaker on 0_RP0_CPU0. 
All necessary files for debug have been collected and saved at 
0/RP0/CPU0 : harddisk:/cisco_support/PAM-ncs5k-memory_leak-xr_0_RP0_CPU0-pam_memory_leaker-2016Aug17-051332.tgz 
(Please copy tgz file out of the router and send to Cisco support. This tgz file will be removed after 14 days.)
CPU Monitoring

The PAM monitors CPU usage on all nodes periodically at an interval of 30 minutes. The PAM reports a CPU hog in either of these scenarios:

  • When a process constantly consumes high CPU (that is, more than the threshold of 90 percentage)

  • When high CPU usage lasts for more than 60 minutes

This is a sample syslog generated when the PAM detects a CPU hog: 


RP/0/RP0/CPU0:Aug 16 00:56:00.819 : logger[68245]: %OS-SYSLOG-4-LOG_WARNING : PAM detected CPU hog for cpu_hogger on 0_RP0_CPU0. 
All necessary files for debug have been collected and saved at 0/RP0/CPU0 : 
harddisk:/cisco_support/PAM-ncs5k-cpu_hog-xr_0_RP0_CPU0-cpu_hogger-2016Aug16-005600.tgz 
(Please copy tgz file out of the router and send to Cisco support. This tgz file will be removed after 14 days.)
RP/0/RP0/CPU0:Jun 21 15:33:54.517 : logger[69042]: %OS-SYSLOG-1-LOG_ALERT : PAM detected ifmgr is hogging CPU on 0_RP0_CPU0!
File System Monitoring

The PAM monitors disk usage on all nodes periodically at an interval of 30 minutes. This is a sample syslog generated when the PAM detects that a file system is full: 


RP/0/RP0/CPU0:Jun 20 13:59:04.986 : logger[66125]: %OS-SYSLOG-4-LOG_WARNING : PAM detected /misc/config is full on 0_1_CPU0 
(please clean up to avoid any fault caused by this). All necessary files for debug have been collected and saved at 
0/RP0/CPU0 : harddisk:/cisco_support/PAM-ncs5k-disk_usage-xr_0_1_CPU0-2016Jun20-135904.tgz 
(Please copy tgz file out of the router and send to Cisco support. This tgz file will be removed after 14 days.)

Disable and Re-enable PAM

The PAM tool consists of three monitoring processes—monitor_cpu.pl, monitor_crash.pl, and monitor_show_show_logging.pl.

Before disabling or re-enabling the PAM, use these options to check if the PAM is installed in the router:

  • From Cisco IOS XR Command Line Interface: 

    
Router# show processes pam_manager location all
Tue Jun 14 17:58:42.791 UTC
node:      node0_RP0_CPU0
                  Job Id: 317
                     PID: 14070
         Executable path: /opt/cisco/XR/packages/iosxr-infra.rp-6.1.1.17I/bin/pam_manager
              Instance #: 1
              Version ID: 00.00.0000
                 Respawn: ON
           Respawn count: 4
            Last started: Mon Jun 13 23:08:43 2016
           Process state: Run
           Package state: Normal
                    core: MAINMEM 
               Max. core: 0
                   Level: 999
               Placement: None
            startup_path: /opt/cisco/XR/packages/iosxr-infra.rp-6.1.1.17I/startup/pam_manager.startup
                   Ready: 0.166s
        Process cpu time: 0.200 user, 0.310 kernel, 0.510 total
JID    TID  Stack  pri  state        NAME             rt_pri
317    14070    0K 20 Sleeping     pam_manager         0 
317    14071    0K 20 Sleeping     lwm_debug_threa     0 
317    14076    0K 20 Sleeping     pam_manager         0 
317    14077    0K 20 Sleeping     lwm_service_thr     0 
317    14078    0K 20 Sleeping     qsm_service_thr     0 
317    14080    0K 20 Sleeping     pam_manager         0

  • From router shell prompt: 

    
Router# run ps auxw|egrep perl
Tue Jun 14 18:00:25.514 UTC
root     14324  0.0  0.2  84676 34556 ?  S Jun13   0:40 /usr/bin/perl /pkg/opt/cisco/pam//monitor_cpu.pl
root     14414  0.0  0.1  65404 14620 ?  S Jun13   0:00 /usr/bin/perl /pkg/opt/cisco/pam//monitor_crash.pl
Disable PAM

To shutdown PAM agents, execute these commands from the XR EXEC mode:

For local RP:

Router# process shutdown pam_manager

For all RPs: 

Router# process shutdown pam_manager location all
Re-enable PAM

Because pam_manager is not a mandatory process, it does not restart automatically if it was manually disabled (unless in the case of a system reload).To restart PAM agents, execute the following commands from XR EXEC mode:

For local RP:

Router# process start pam_manager

For all RPs: 

Router# process start pam_manager location all

Note

To start PAM on all locations, the pam_manager process should be restarted on all nodes by using the location all option in the process start pam_manager command.

Data Archiving in PAM

At any given point of time, PAM does not occupy more than 200 MB of harddisk: space. If more than 200 MB is needed, then PAM archives old files and rotates the logs automatically.

The PAM collects CPU or memory usage (using top -b -n1 command) periodically at an interval of 30 minutes. The files are saved under harddisk:/cisco_support/ directory with the filename as <node name>.log (for example, harddisk:/cisco_support/xr-0_RP0_CPU0.log). When the file size exceeds the limit of 15MB, the file is archived (compressed) into .tgz file, and then rotated for a maximum of two counts (that is, it retains only two .tgz files). The maximum rotation count of .tgz files is three. Also, the old file (ASCII data) is archived and rotated if a node is reloaded. For example, xr-0_RP0_CPU0.log is archived if RP0 is reloaded.

You must not manually delete the core file generated by the PAM. The core file is named as <process name>_pid.by_user.<yyyymmdd>-<hhmmss>.<node>.<checksum>.core.gz .

Files Collected by PAM Tool

The table below lists the various PAM events and the respective commands and files collected by the PAM for each event.

You can attach the respective.tgz file when you raise a service request (SR) with Cisco Technical Support.

Event Name

Commands and Files Collected by PAM

Process crash

  • show install active

  • show platform

  • show version

  • core (gz) file

  • core.txt file

Process traceback

  • show dll

  • show install active

  • show logging

  • show platform

  • show version

Memory leak

  • show install active

  • show platform

  • show version

  • core (gz) file

  • dumpcore running

  • continuous memory usage snapshots

Show logging event

  • show install active

  • show logging

  • show platform

  • show version

  • core (gz) file

  • core.txt file

CPU hog

  • follow process

  • pstack

  • show dll

  • show install active

  • show platform

  • show version

  • top -H

  • core (gz) file

  • CPU usage snapshots

Disk usage

  • show install active

  • show platform

  • show version

  • console log

  • core (gz) file

  • Disk usage snapshots