Nexus 7000 Logging Problem Troubleshoot

Introduction

This document describes troubleshoot procedures used in order to fix logging problems on the Cisco Nexus 7000 (N7K) platform. Specifically, this document describes two targeted scenarios: when the switch does not send new logs to the local logfile and when the new logs are dropped or delayed in transit to the local logfile and syslog server.

N7K Switch Does Not Show Log Messages in the Local Logfile Buffer

This section describes how to troubleshoot a N7K switch if log messages do not display in the local logfile buffer. In order to troubleshoot these issues, complete these steps:

1. Verify that the logs you expect to see in the local logfile buffer have a logging level less than or equal to the logfile logging level. For example, if the missing logs are for interface up/down notifications, verify that the logging level of ethpm (in this example, 5) is less than or equal to the severity of the logging logfile (notifications in this example, which is 5):

   Nexus# show logging level ethpm
   
   Facility  Default Severity  Current Session
                                   Severity
   --------  ----------------  ---------------
   ethpm             5                 5
   
   0(emergencies)  1(alerts)     2(critical)
   3(errors)       4(warnings)   5(notifications)
   6(information)  7(debugging)

   Nexus# show logging
   
   Logging logfile:enabled
    Name - test2: Severity - 
    notifications Size -
    10000000

2. If all of the new logs are not logged in the local logfile, not just logs that belong to a specific category, verify:
   
   
   1. If the /var/log directory is used up to 100%:

      Nexus# show system internal flash
      Mount-on                  1K-blocks      Used   Available   Use%  Filesystem
      /                            409600     61104      348496     15   /dev/root
      /proc                             0         0           0      0   proc
      /sys                              0         0           0      0   none
      /isan                        716800    315088      401712     44   none
      /var                          51200       612       50588      2   none
      /etc                           5120      1616        3504     32   none
      /nxos/tmp                     40960         4       40956      1   none
      /var/log                      51200     51200           0    100   none

      In this output, /var/log is 100% used.
      
      
   2. If the subdirectory /external has a very large libdt_helper.log file size:

      Nexus# show system internal dir /var/log/external/
      ./                     420
      ../                    380
      glbp.debug             231
      libfipf.24944            0
      vdc_4/                  80
      libfipf.24115            0
      vdc_3/                  80
      libfipf.23207            0
      vdc_2/                  80
      libdt_helper.log    51523584
      libfipf.5582             0
      libfipf.4797             0
      libfipf.4717             0
      messages              651264
      syslogd_ha_debug       19184
      startupdebug             0
      eobc_port_test_result    3
      mgmt_port_test_result    3
      bootup_test.log        18634
      bootup_test.3432        2526
      dmesg@                  31

      If it is very large, then delete the libdt_helper.log file from the directory with this command:

      Nexus# delete log:libdt_helper.log
      Nexus#

      Back up the old messages in the local logging buffer to a file on bootflash with this command:

      Nexus# show logging log > bootflash:oldlogs.txt
      Nexus#

      This command redirects the output to a file on bootflash called oldlogs.txt. This keeps a copy of the old logs because the local logging buffer must be cleared next with this command:

      Nexus# clear logging logfile
      Nexus#

3. At this point, the switch should show new logs in the local logfile. In order to confirm this, enter configuration mode, and then exit in order to trigger the VSHD-5-VSHD_SYSLOG_CONFIG log message:

   Nexus# conf t
   Enter configuration commands, one per line.  End with CNTL/Z.
   Nexus(config)# end
   Nexus# show logging log
   2013 Jul 30 11:53:55 Nexus %SYSLOG-1-SYSTEM_MSG :
    Logging logfile (test2) cleared by user
   2013 Jul 30 11:55:35 Nexus %VSHD-5-VSHD_SYSLOG_CONFIG_I:
    Configured from vty by admin on dhcp-peg3-vl30-144-254-7-77.ci
   Nexus#

Note: If you encounter the previously-described scenario, then the switch is affected by a software defect described in Cisco bug ID CSCue98451: New messages not being logged in logfile. The troubleshoot procedure described in this document is a workaround. The ultimate fix is to upgrade to a software release that has the fix for the aforementioned bug.

N7K Syslog and/or Logfile Logs Delayed or Have Incorrect Timestamp

New log messages sent to the syslog server and/or to the local logfile buffer might exhibit a delay (possibly between five to eight minutes). Some of the log messages might be dropped. The log timestamp either shows the timestamp of the actual event or the timestamp of message transmission to the syslog server or logfile buffer.

If any or all of these symptoms are encountered, then verify that:

1. Any debugs are enabled on the switch:

   Nexus# show debug
   
   Debug level is set to Minor(1)
   L2FM Daemon:
    Trace Detail  is on
    default for new sessions logging level: 3
   debug ip arp event
   debug ip arp packet
   
   Nexus#

   In this example, L2FM and ARP debugs are enabled.
   
   
2. The logging level is changed to a higher value than the default values for any processes:

   Nexus# show logging level
   
   Facility  Default Severity  Current Session
                                  Severity
   --------  ----------------  ---------------
   aaa               3                3
   acllog            2                2
   aclmgr            3                3
   auth              0                0
   authpriv          3                3

   The log messages are stored in an internal, circular buffer before they are delivered to the syslog/logfile. The number of messages held in this circular buffer is viewed with this command:

   Nexus# show logging internal info |
    include circular
   Pending msgs in circular buffer
    (head: 8632, tail: 7333)

   In this example, the buffer has (head - tail) 8632 - 7333 = 1299 messages.
   
   
3. If the previous step applies, then disable the active debugs and reduce the logging levels.

Note: If the second scenario is encountered, then the switch is affected by a sofware defect described is Cisco bug ID CSCud40436: Syslog messages delayed and droped when debugs are enabled. This bug does not affect the Nexus Operating System (NX-OS) Version 6.0; it is fixed in NX-OS Versions 6.1(3) and later.