Troubleshooting CUCM TFTP Service cored on January 2021

Introduction

This document describes the known issue of TFTP service core in Cisco Unified Communications Manager (CUCM) versions 11.5.1 SU7 and 11.5.1 SU8 that require a workaround to fix the issue. 

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on Cisco Unified Communications Manager (CUCM).

Problem

CUCM TFTP Service cored once the server clocks the time 00:00 2021

Symptoms

You will receive an alert in the RTMT for TFTP service going down and below are some steps to identify the issue.

Jan 1 00:03:19 ucm-pub local7 6 : 898: ucm-pub.ciscolab.com: Jan 01 2021 07:03:19.321 UTC : %UC_GENERIC-6-ServiceStopped: %[ServiceName=Cisco Tftp][AppID=Cisco Service Manager][ClusterID=][NodeID=cap-86-pub.bldr-caplab.com]: Service stopped.

Jan 1 00:03:21 ucm-pub local7 6 : 899: ucm-pub.ciscolab.com: Jan 01 2021 07:03:21.346 UTC : %UC_GENERIC-6-ServiceStarted: %[ServiceName=Cisco Tftp][ProcessID=6803][AppID=Cisco Se

From Server CLI:

1. Login to CUCM(TFTP) server and run the command utils core active list. the output will show multiple CTFTP cores. 
2. Run the command "utils core active analyze <core name>"
3. The output(backtrace) of the core should match with below.

#0 0x07adc3e8 in fputs () from /lib/libc.so.6
#1 0x0876cd38 in SdlSystemLog::writeToFile (this=0xf4a02930,_traceLinePnt=0xf4a05ba0) at SdlSystemLog.cpp:1971
#2 0x0876ce54 in SdlSystemLog::writer (this=0xf4a02930) at SdlSystemLog.cpp:1919
#3 0x0078dbc9 in start_thread () from /lib/libpthread.so.0
#4 0x07b5e07e in clone () from /lib/libc.so.6

Cisco CUCM Syslogs

Jan 1 00:03:19 ucm-pub local7 6 Cisco: Tftp: DBL SDI Initialization successful
Jan 1 00:03:19 ucm-pub local7 6 Cisco: Tftp: SysLevel [127] UserMask [1] Enable [TRUE] File [/var/log/active/cm/trace/dbl/sdi/dbl.log]
Jan 1 00:03:20 ucm-pub local7 4 Cisco: Tftp: SDIDBConfigData::Read failed to retrieve SDI folder location, will use location in DB. ServiceID=1, NodeName=ucm-pub.bldr-caplab.com
Jan 1 00:03:24 ucm-pub local7 6 Cisco: Tftp: DBNotify SDI Initialization successful

CUCM Cisco TFTP SDL Logs 

SDL001_600_000103.txt:00913050.000 |00:03:17.708 |AppInfo | TID[f660e740] CServiceModule::waitForSignals() TID[f660e740] Signal[15] Received to QUIT, quitting.

Note: After DST update, TFTP service will restarted and no core is observed which is by design.

Workaround

The core is happening when it tries to write something in the trace file. This may also result in the ability of the TFTP service to start.  

Below are the steps.

Step 1.  Set os to permissive mode by running "utils os secure permissive" with either Trace settings on or off.  

Step 2. Restart/Start TFTP service from Serviceability.

Step 3. Move back the cluster to Enforcing using "utils os secure enforce".

This issue is documented by Cisco bug ID CSCvw94324

Note: Workaround is temporary, Since this issue is being investigated by Cisco Engineering team.