裝置升級後，HCM-F服務處於停止狀態

簡介

本檔案介紹升級託管合作調解履行(HCM-F)後出現的問題以及疑難排解程式。

問題的根本原因與過期的Tomcat證書有關。許多計畫升級其HCM-F裝置（以便從新引入的功能中受益）的客戶在其現有的HCM-F Tomcat證書已過期且未重新生成時將面臨此問題。

必要條件

需求

思科建議您瞭解以下主題：

• 思科託管合作解決方案(HCS)
• 思科託管合作調解履行(HCM-F)

採用元件

本文中的資訊係根據以下軟體和硬體版本：

• HCM-F 11.5.4

本文中的資訊是根據特定實驗室環境內的裝置所建立。文中使用到的所有裝置皆從已清除（預設）的組態來啟動。如果您的網路正在作用，請確保您已瞭解任何指令可能造成的影響。

問題

升級HCM-F（11.5.1到11.5.4）和交換器版本後，Graphical User Interface(GUI)頁面無法再訪問，且服務處於停止狀態。

已執行故障排除

• 檢查服務狀態時發現Web使用者介面(UI)和Tomcat相關服務處於STOPPED狀態：

admin:utils service list 
Requesting service status, please wait...
System SSH [STARTED]
Name Service Cache [STOPPED] Service Not Activated
Entropy Monitoring Daemon [STARTED]
Cisco SCSI Watchdog [STARTED]Service Manager [STARTED]
Service Manager is running
Getting list of all services
>> Return code = 0
Cisco AMC Service[STARTED]
Cisco Audit Event Service[STARTED]
Cisco CDM Database[STARTED]
Cisco CDP[STARTED]
Cisco CDP Agent[STARTED]
Cisco Certificate Expiry Monitor[STARTED]
Cisco Configuration Manager[STARTED]
Cisco DRF Local[STARTED]
Cisco DRF Master[STARTED]
Cisco HCS Admin UI[STOPPED] Component is not running
Cisco HCS CSF UI Service[STOPPED]  Component is not running
Cisco HCS CUCDMSync Service[STARTED]
Cisco HCS ConfigMgr[STARTED]
Cisco HCS DMA-SA Service[STARTED]
Cisco HCS Data Access Manager Service[STARTED]
Cisco HCS Fulfillment Service[STARTED]
Cisco HCS IPA Service[STARTED]
Cisco HCS Intelligent Loader Service[STOPPED]  Component is not running
Cisco HCS License Manager Service[STARTED]
Cisco HCS NBI REST FF Web Service[STOPPED] Component is not running
Cisco HCS NBI REST SDR Web Service[STOPPED]  Component is not running
Cisco HCS NBI REST UCMon Web Service[STOPPED]  Component is not running
Cisco HCS North Bound Interface Web Service[STOPPED]  Component is not running
Cisco HCS Provisioning Adapter Service[STARTED]
Cisco HCS SDR Change Notification Service[STARTED]
Cisco HCS SDR UI[STOPPED]  Component is not running
Cisco HCS SI UI[STOPPED]  Component is not running
Cisco HCS SSO SP Service[STOPPED]  Component is not running
Cisco HCS Service Inventory[STOPPED]  Component is not running
Cisco HCS UCPA Service[STARTED]
Cisco HCS UCSMSync Service[STARTED]
Cisco HCS VCenterSync Service[STARTED]
Cisco JMS Broker[STARTED]
Cisco Log Partition Monitoring Tool[STARTED]
Cisco Platform Manager Service[STOPPED]  Component is not running
Cisco RIS Data Collector[STARTED]
Cisco RTMT Web Service[STOPPED]  Component is not running
Cisco Syslog Agent[STARTED]
Cisco Tomcat[STARTED]
Cisco Tomcat Stats Servlet[STOPPED]  Component is not running
Cisco Trace Collection Service[STARTED]
Cisco Trace Collection Servlet[STOPPED]  Component is not running
Host Resources Agent[STARTED]
MIB2 Agent[STARTED]
Platform Administrative Web Service[STOPPED]  Component is not running
SNMP Master Agent[STARTED]System Application Agent[STARTED]
Cisco HCS UC Monitor Service[STOPPED]  Service Not ActivatedPrimary Node =true

• 在CiscoSyslog中，我們看到不同Web相關服務的超文本傳輸協定(HTTP)404錯誤：

admin:file view activelog /syslog/CiscoSyslog 

Jun 17 16:14:19 hcs862-hcm-f local7 2 : 6: hcs862-hcm-f.XXXXXXXX.net:
Jun 17 2019 14:14:19.781 UTC :  %UC_SERVICEMANAGER-2-ServiceStartFailed: %[ServiceName=Cisco RTMT Web Service][Reason=HTTP 404 Error][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Failed to start service.
Jun 17 16:14:19 hcs862-hcm-f local7 2 : 7: hcs862-hcm-f.XXXXXXXXtest.net:
Jun 17 2019 14:14:19.781 UTC :  %UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Cisco RTMT Web Service][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Service terminated.
Jun 17 16:14:20 hcs862-hcm-f local7 2 : 8: hcs862-hcm-f.XXXXXXXXtest.net:
Jun 17 2019 14:14:20.791 UTC :  %UC_SERVICEMANAGER-2-ServiceStartFailed: %[ServiceName=Cisco Trace Collection Servlet][Reason=HTTP 404 Error][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Failed to start service.
Jun 17 16:14:20 hcs862-hcm-f local7 2 : 9: hcs862-hcm-f.XXXXXXXXtest.net: Jun 17 2019 14:14:20.791 UTC :  %UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Cisco Trace Collection Servlet][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Service terminated.
Jun 17 16:14:21 hcs862-hcm-f local7 2 : 10: hcs862-hcm-f.XXXXXXXXtest.net: Jun 17 2019 14:14:21.796 UTC :  %UC_SERVICEMANAGER-2-ServiceStartFailed: %[ServiceName=Cisco Tomcat Stats Servlet][Reason=HTTP 404 Error][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Failed to start service.
Jun 17 16:14:21 hcs862-hcm-f local7 2 : 11: hcs862-hcm-f.XXXXXXXXtest.net: Jun 17 2019 14:14:21.796 UTC :  %UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Cisco Tomcat Stats Servlet][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Service terminated.
Jun 17 16:14:22 hcs862-hcm-f local7 6 : 12: hcs862-hcm-f.XXXXXXXXtest.net: Jun 17 2019 14:14:22.800 UTC :  %UC_GENERIC-6-ServiceStarted: %[ServiceName=Cisco CDP][ProcessID=8426][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Service started.
Jun 17 16:14:23 hcs862-hcm-f local7 2 : 13: hcs862-hcm-f.XXXXXXXXtest.net:
Jun 17 2019 14:14:23.804 UTC :  %UC_SERVICEMANAGER-2-ServiceStartFailed: %[ServiceName=Cisco HCS Admin UI][Reason=HTTP 404 Error][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Failed to start service.
Jun 17 16:14:23 hcs862-hcm-f local7 2 : 14: hcs862-hcm-f.XXXXXXXXtest.net:
Jun 17 2019 14:14:23.804 UTC :  %UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Cisco HCS Admin UI][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Service terminated.
Jun 17 16:14:24 hcs862-hcm-f local7 2 : 15: hcs862-hcm-f.XXXXXXXXtest.net:
Jun 17 2019 14:14:24.808 UTC :  %UC_SERVICEMANAGER-2-ServiceStartFailed: %[ServiceName=Cisco HCS SDR UI][Reason=HTTP 404 Error][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Failed to start service.
Jun 17 16:14:24 hcs862-hcm-f local7 2 : 16: hcs862-hcm-f.XXXXXXXXtest.net: Jun 17 2019 14:14:24.808 UTC :  %UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Cisco HCS SDR UI][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Service terminated.
Jun 17 16:14:25 hcs862-hcm-f local7 6 : 17: hcs862-hcm-f.XXXXXXXXtest.net: Jun 17 2019 14:14:25.812 UTC :  %UC_GENERIC-6-ServiceStarted: %[ServiceName=Cisco Certificate Expiry Monitor][ProcessID=8774][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Service started.
Jun 17 16:14:25 hcs862-hcm-f local7 2 : 18: hcs862-hcm-f.XXXXXXXXtest.net:
Jun 17 2019 14:14:25.813 UTC :  %UC_SERVICEMANAGER-2-ServiceStartFailed: %[ServiceName=Cisco HCS NBI REST SDR Web Service][Reason=HTTP 404 Error][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Failed to start service.
Jun 17 16:14:25 hcs862-hcm-f local7 2 : 19: hcs862-hcm-f.XXXXXXXXtest.net:
Jun 17 2019 14:14:25.813 UTC :  %UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Cisco HCS NBI REST SDR Web Service][Reason=Service stopped abruptly][AppID=Cisco Service Manager][ClusterID=][NodeID=hcs862-hcm-f]: Service terminated.

• Catalina中的Tomcat.out報告CCMTrustCerts上下文問題：

SEVERE: Context [/CCMTrustCerts] startup failed due to previous errors
Jun 18, 2019 2:59:49 PM org.apache.catalina.startup.HostConfig deployDirectoryINFO: Deployment of web application directory /common/log/taos-log-a/tomcat/webapps/CCMTrustCerts has finished in 8,042 ms end of the file reachedoptions: q=quit, n=next, p=prev, b=begin, e=end (lines 1221 - 1223 of 1223) :INFO: Deploying web application directory /common/log/taos-log-a/tomcat/webapps/manager
Jun 18, 2019 2:59:33 PM org.apache.catalina.startup.SetContextPropertiesRule beginWARNING: [SetContextPropertiesRule]{Context} Setting property 'debug' to '0' did not find a matching property.
Jun 18, 2019 2:59:33 PM org.apache.tomcat.util.digester.SetPropertiesRule beginWARNING: [SetPropertiesRule]{Context/Realm} Setting property 'resourceName' to 'TomcatUsersDatabase' did not find a matching property.
Jun 18, 2019 2:59:41 PM org.apache.catalina.startup.TldConfig executeINFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Jun 18, 2019 2:59:41 PM org.apache.catalina.core.StandardContext startInternalSEVERE: One or more Filters failed to start. Full details will be found in the appropriate container log file
Jun 18, 2019 2:59:41 PM org.apache.catalina.core.StandardContext startInternalSEVERE: Context [/manager] startup failed due to previous errors
Jun 18, 2019 2:59:41 PM org.apache.catalina.startup.HostConfig deployDirectoryINFO: Deployment of web application directory /common/log/taos-log-a/tomcat/webapps/manager has finished in 7,483 ms
Jun 18, 2019 2:59:41 PM org.apache.catalina.startup.HostConfig deployDirectoryINFO: Deploying web application directory /common/log/taos-log-a/tomcat/webapps/CCMTrustCerts
Jun 18, 2019 2:59:49 PM org.apache.catalina.startup.TldConfig executeINFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Jun 18, 2019 2:59:49 PM org.apache.catalina.core.StandardContext startInternalSEVERE: One or more Filters failed to start. Full details will be found in the appropriate container log file
Jun 18, 2019 2:59:49 PM org.apache.catalina.core.StandardContext startInternal options: SEVERE: Context [/CCMTrustCerts] startup failed due to previous errors
Jun 18, 2019 2:59:49 PM org.apache.catalina.startup.HostConfig deployDirectoryINFO: Deployment of web application directory /common/log/taos-log-a/tomcat/webapps/CCMTrustCerts has finished in 8,042 ms  

• 從命令列介面(CLI)運行「utils diagnose test」時，在tomcat_connectors模組中顯示一個錯誤，表明與Tomcat的內部連線未正常工作： 

test - tomcat_connectors   : Failed - The HTTP and HTTPS ports are not responding to local requests. 
Please collect all of the Tomcat logs for root cause analysis: file get activelog tomcat/logs/*  

• Tomcat證書於3月26日到期。儘管如此，在HCM-F升級期間（或之前）未發現問題：

Jun 19 12:00:00 hcs862-hcm-f local99 0 : 42: hcs862-hcm-f.XXXXXXXXtest.net:
Jun 19 2019 10:00:00.19 UTC :  %UC_CERT-0-CertExpired: %[Message=Certificate expiration Notification. Certificate name:tomcat.der Unit:tomcat Type:own-cert Expiration:Sun Mar 26 12:30:59:000 CES][AppID=Cisco Certificate Monitor][ClusterID=][NodeID=hcs862-hcm-f]: Certificate has Expired and needs to be changed at the earliest
Jun 19 12:00:00 hcs862-hcm-f local99 0 : 43: hcs862-hcm-f.XXXXXXXXtest.net:
Jun 19 2019 10:00:00.21 UTC :  %UC_CERT-0-CertExpired: %[Message=Certificate expiration Notification. Certificate name:ipsec.der Unit:ipsec Type:own-cert Expiration:Sun Mar 26 12:30:57:000 CEST ][AppID=Cisco Certificate Monitor][ClusterID=][NodeID=hcs862-hcm-f]: Certificate has Expired and needs to be changed at the earliest       

解決方案

  1.重新生成Tomcat證書:

admin:set cert regen tomcat 
WARNING: This operation will overwrite any CA signed certificate previously imported for  tomcat
Proceed with regeneration (yes|no)? yes
Successfully Regenerated Certificate for tomcat.
You must restart services related to tomcat for the regenerated certificates to become active.  

 2.  重新啟動HCM-F伺服器：

admin:utils system restart
 Do you really want to restart ?Enter (yes/no)? yes 

驗證

• 驗證服務狀態：

admin:utils service list 
Requesting service status, please wait...
System SSH [STARTED]
Name Service Cache [STOPPED] Service Not Activated
Entropy Monitoring Daemon [STARTED]
Cisco SCSI Watchdog [STARTED]
Service Manager [STARTED]Service Manager is running
Getting list of all services
>> Return code = 0
Cisco AMC Service[STARTED]
Cisco Audit Event Service[STARTED]
Cisco CDM Database[STARTED]
Cisco CDP[STARTED]
Cisco CDP Agent[STARTED]
Cisco Certificate Expiry Monitor[STARTED]
Cisco Configuration Manager[STARTED]
Cisco DRF Local[STARTED]
Cisco DRF Master[STARTED]
Cisco HCS Admin UI[STARTED]
Cisco HCS CSF UI Service[STARTED]
Cisco HCS CUCDMSync Service[STARTED]
Cisco HCS ConfigMgr[STARTED]
Cisco HCS DMA-SA Service[STARTED]
Cisco HCS Data Access Manager Service[STARTED]
Cisco HCS Fulfillment Service[STARTED]
Cisco HCS IPA Service[STARTED]
Cisco HCS Intelligent Loader Service[STARTED]
Cisco HCS License Manager Service[STARTED]
Cisco HCS NBI REST FF Web Service[STARTED]
Cisco HCS NBI REST SDR Web Service[STARTED]
Cisco HCS NBI REST UCMon Web Service[STARTED]
Cisco HCS North Bound Interface Web Service[STARTED]
Cisco HCS Provisioning Adapter Service[STARTED]
Cisco HCS SDR Change Notification Service[STARTED]
Cisco HCS SDR UI[STARTED]
Cisco HCS SI UI[STARTED]
Cisco HCS SSO SP Service[STARTED]
Cisco HCS Service Inventory[STARTED]
Cisco HCS UCPA Service[STARTED]
Cisco HCS UCSMSync Service[STARTED]
Cisco HCS VCenterSync Service[STARTED]
Cisco JMS Broker[STARTED]
Cisco Log Partition Monitoring Tool[STARTED]
Cisco Platform Manager Service[STARTED]
Cisco RIS Data Collector[STARTED]
Cisco RTMT Web Service[STARTED]
Cisco Syslog Agent[STARTED]
Cisco Tomcat[STARTED]
Cisco Tomcat Stats Servlet[STARTED]
Cisco Trace Collection Service[STARTED]
Cisco Trace Collection Servlet[STARTED]
Host Resources Agent[STARTED]MIB2 Agent[STARTED]
Platform Administrative Web Service[STARTED]
SNMP Master Agent[STARTED]
System Application Agent[STARTED]
Cisco HCS UC Monitor Service[STOPPED]  
Service Not ActivatedPrimary Node =true