The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This document describes how to troubleshoot Hosted License Manager (HLM) and Prime License Manager (PLM) when a new PLM added. Perform this checklist before you open a TAC case.
Cisco recommends that you have knowledge of these topics:
The information in this document is based on these software and hardware versions:
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
This is the error message that is seen.
In order to troubleshoot this issue, the HLM detailed logs are required as several conditions can lead to the same GUI error.
1. Log into the HCM-F. CLI enables the logs level and collects the logs.
2. Run set trace tracelevel command.
3. Enter the tracelevel (use CLI show tracelevels in orderto find allowed tracelevels).
4. Enter the servicename: Cisco HCS License Manager Service.
5. Reproduce the issue and collect the logs with this command file get activelog /hcs/HLM/*.
HLM logs shows that HLM is not able to create a Representational State Transfer (REST) connection to the PLM due a connection timeout.
| 2016-09-02 04:52:55.025 | [63406] | SdlSig | CreateELMRequest | null0 | HLM(HLMCORE) | SDRUI(NBI-HLM-v90- LIB0) | null 2016-09-02 16:52:55,025 INFO [63406] HLMCoreAgent:null0_CreateELMRequest is fired... 2016-09-02 16:52:55,025 INFO [63406] HLMCoreAgent:null0_CreateELMRequest recieved 2016-09-02 16:52:55,025 INFO [63406] HLMCoreAgent:elmName PLM2 2016-09-02 16:52:55,025 INFO [63406] HLMCoreAgent:elmHost PLM2 2016-09-02 16:52:55,025 INFO [63406] HLMCoreAgent:elmUser administrator 2016-09-02 16:52:55,025 INFO [63406] HLMCoreAgent:elmPass ****** 2016-09-02 16:52:55,025 INFO [63406] HLMCoreAgent:deploymentMode HCS 2016-09-02 16:52:55,025 INFO [63406] HLMCoreAgent:null0_CreateELMRequest opening REST connection to supplied LM 2016-09-02 16:52:55,025 INFO [63406] ELMApi: init, connecting to PLM2 2016-09-02 16:52:55,025 DEBUG [63406] Got the sslCtx 2016-09-02 16:52:55,025 INFO [63406] Initialized the context 2016-09-02 16:52:55,096 INFO [63406] ELMApi: initialized with connection timeout, connected to PLM2 2016-09-02 16:52:55,096 INFO [63406] HLMCoreAgent:null0_CreateELMRequest validating connection to LM 2016-09-02 16:52:55,096 DEBUG [63406] ELMApi: validateConnection Called 2016-09-02 16:52:55,097 ERROR [63406] ELMApi: ClientHanglerException caught validating connection, throwing C onnectException 2016-09-02 16:52:55,097 ERROR [63406] HLMCoreAgent:CreateELMRequest Exception: java.net.UnknownHostException: PLM2 --------- 2016-09-02 16:52:55,100 DEBUG [63406] Agent: Sent message to SDRUI(NBI-HLM-v90-LIB0) 2016-09-02 16:52:55,100 DEBUG [63406] HLMCoreAgent:CreateELMResponse message was sent. 2016-09-02 16:52:55,100 ERROR [63406] Sdl... 5 | 2016-09-02 04:52:55.100 | [63406] | SdlExcept | CreateELMRequest | null0 | HLM(HLMCORE) | SDRUI(NBI-HLM-v 90-LIB0) | Error in fire: java.net.UnknownHostException: PLM2 2016-09-02 16:52:55,101 ERROR [63406] Agent caught exception while processing message type CreateELMRequest com.sun.jersey.api.client.ClientHandlerException: java.net.UnknownHostException: PLM2 at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.j ava:151)
Add a direct and reverse lookup in the HCS management of the Domain Name Server (DNS) and verify any connectivity issue, as shown in these images.
Verify
Run the commands.
admin:show network eth0 Ethernet 0 DHCP : disabled Status : up IP Address : 10.48.50.70 IP Mask : 255.255.255.128 Link Detected: yes Mode : Auto disabled, Full, 1000 Mbits/s Duplicate IP : no DNS Primary : 10.48.55.76 Secondary : Not Configured Options : timeout:5 attempts:2 Domain : hcslab.com Gateway : 10.48.50.1 on Ethernet 0 admin:utils network ping plm2 PING plm2.hcs10.com (10.48.50.72) 56(84) bytes of data. 64 bytes from plm2.hcs10.com (10.48.50.72): icmp_seq=1 ttl=64 time=0.484 ms 64 bytes from plm2.hcs10.com (10.48.50.72): icmp_seq=2 ttl=64 time=0.493 ms 64 bytes from plm2.hcs10.com (10.48.50.72): icmp_seq=3 ttl=64 time=0.441 ms 64 bytes from plm2.hcs10.com (10.48.50.72): icmp_seq=4 ttl=64 time=0.590 ms --- plm2.hcs10.com ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3021ms rtt min/avg/max/mdev = 0.441/0.502/0.590/0.054 ms
HLM logs shows that HLM is able to create a REST connection with the PLM instance but PLM replies back with error message of "404 Not Found".
2016-09-02 17:01:08,561 INFO [63414] ELMApi: initialized with connection timeout, connected to 10.48.55.29 2016-09-02 17:01:08,561 INFO [63414] HLMCoreAgent:null0_CreateELMRequest validating connection to LM 2016-09-02 17:01:08,561 DEBUG [63414] ELMApi: validateConnection Called 2016-09-02 17:01:08,614 ERROR [63414] HLMCoreAgent:CreateELMRequest Exception: GET https://10.48.55.29/elm-resources returned a response status of 404 Not Found 2016-09-02 17:01:08,616 DEBUG [63414] Agent: sending to [SDRUI] ------- <com.cisco.hcs.HLM.jms.CreateELMResponse> <messageType>CreateELMResponse</messageType> <source> <serviceName>HLM</serviceName> <instance>HLMCORE</instance> </source> <destination> <serviceName>SDRUI</serviceName> <instance>NBI-HLM-v90-LIB0</instance> </destination> <sessionID>b0a354ec-6249-4cf6-ac7d-f73bac6918d4</sessionID> <fault>true</fault> <exception> <detailMessage>GET https://10.48.55.29/elm-resources returned a response status of 404 Not Found</detailMessage> <stackTrace> <trace>com.cisco.hcs.HLM.core.HLMCoreAgent$null0_CreateELMRequest.fire(HLMCoreAgent.java:363)</trace> <trace>com.cisco.hcs.hcsagent.agent.AgentFSM.fireTransition(AgentFSM.java:554)</trace> <trace>com.cisco.hcs.hcsagent.agent.AgentFSM.fire(AgentFSM.java:445)</trace> <trace>com.cisco.hcs.hcsagent.agent.Agent.process(Agent.java:324)</trace> <trace>com.cisco.hcs.hcsagent.agent.AgentTask.run(AgentTask.java:75)</trace> <trace>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)</trace> <trace>java.util.concurrent.FutureTask.run(FutureTask.java:262)</trace> <trace>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)</trace> <trace>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)</trace> <trace>java.lang.Thread.run(Thread.java:745)</trace> </stackTrace> <suppressedExceptions class="java.util.Collections$UnmodifiableRandomAccessList" resolves-to="java.util.Collections$UnmodifiableList"> <c class="list"/> <list reference="../c"/> </suppressedExceptions> </exception> <Fork>false</Fork> <requeueCount>0</requeueCount> </com.cisco.hcs.HLM.jms.CreateELMResponse> --------- 2016-09-02 17:01:08,618 DEBUG [63414] Agent: Sent message to SDRUI(NBI-HLM-v90-LIB0) 2016-09-02 17:01:08,618 DEBUG [63414] HLMCoreAgent:CreateELMResponse message was sent. 2016-09-02 17:01:11,625 INFO [63414] AgentFSM::currentState changed from null0 to null0 The PLM API services must be activeted in order to fix the issue. Run the following commands from PLM CLI. admin:utils service activate Cisco Prime LM Resource API Service Manager is running Cisco Prime LM Resource API[STARTED] admin:utils service activate Cisco Prime LM Resource Legacy API Service Manager is running Cisco Prime LM Resource Legacy API[STARTED]
Verify
Run the command from PLM CLI.
admin utils service list Cisco Prime LM Admin[STARTED] Cisco Prime LM DB[STARTED] Cisco Prime LM Resource API[STARTED] Cisco Prime LM Resource Legacy API[STARTED]
HLM logs show that the PLM runs in demo mode.
11 | 2016-09-02 05:21:07.615 | [63406] | SdlSig | CreateELMRequest | null0 | HLM(HLMCORE) | SDRUI(NBI-HLM-v90-LIB0) | null 2016-09-02 17:21:07,615 INFO [63406] HLMCoreAgent:null0_CreateELMRequest is fired... 2016-09-02 17:21:07,615 INFO [63406] HLMCoreAgent:null0_CreateELMRequest recieved 2016-09-02 17:21:07,615 INFO [63406] HLMCoreAgent:elmName PLM2 2016-09-02 17:21:07,615 INFO [63406] HLMCoreAgent:elmHost 10.48.55.29 2016-09-02 17:21:07,615 INFO [63406] HLMCoreAgent:elmUser administrator 2016-09-02 17:21:07,615 INFO [63406] HLMCoreAgent:elmPass ****** 2016-09-02 17:21:07,615 INFO [63406] HLMCoreAgent:deploymentMode HCS 2016-09-02 17:21:07,615 INFO [63406] HLMCoreAgent:null0_CreateELMRequest opening REST connection to supplied LM 2016-09-02 17:21:07,615 INFO [63406] ELMApi: init, connecting to 10.48.55.29 2016-09-02 17:21:07,615 DEBUG [63406] Got the sslCtx 2016-09-02 17:21:07,615 INFO [63406] Initialized the context 2016-09-02 17:21:07,687 INFO [63406] ELMApi: initialized with connection timeout, connected to 10.48.55.29 2016-09-02 17:21:07,687 INFO [63406] HLMCoreAgent:null0_CreateELMRequest validating connection to LM 2016-09-02 17:21:07,687 DEBUG [63406] ELMApi: validateConnection Called 2016-09-02 17:21:07,795 DEBUG [63406] ELMApi: connection Validated 2016-09-02 17:21:07,795 DEBUG [63406] ELMApi: getELMReg Called 2016-09-02 17:21:07,846 INFO [63406] HLMCoreAgent:Found LM to be in Demo Mode 2016-09-02 17:21:07,846 DEBUG [63406] HLMCoreAgent: CreateELMRequest closing the connection 2016-09-02 17:21:07,847 DEBUG [63406] Agent: sending to [SDRUI] ------- <com.cisco.hcs.HLM.jms.CreateELMResponse> <messageType>CreateELMResponse</messageType> <source> <serviceName>HLM</serviceName> <instance>HLMCORE</instance> </source> <destination> <serviceName>SDRUI</serviceName> <instance>NBI-HLM-v90-LIB0</instance> </destination> <sessionID>ae3b7f12-d5cb-4db4-bc37-291833d00f83</sessionID> <fault>true</fault> <exception> <detailMessage>ELM is in demo license.</detailMessage> <stackTrace> <com.cisco.hcs.HLM.jms.CreateELMResponse> <messageType>CreateELMResponse</messageType> <source> <serviceName>HLM</serviceName> <instance>HLMCORE</instance> </source> <destination> <serviceName>SDRUI</serviceName> <instance>NBI-HLM-v90-LIB0</instance> </destination> <sessionID>ab8cbfc5-cbe5-4c3f-8c33-324517d69d7d</sessionID> <fault>true</fault> <exception> <detailMessage>ELM is in demo license.</detailMessage> <stackTrace>
Add HCS licenses into PLM and re-attempt to add the PLM.
Verify
Login to the PLM user interface and click on About. After you add the HCS licenses, the PLM shows up a valid registritation ID as shown in the image:
HLM is able to contact the PLM but PLM replies with 401 message.
12 | 2016-09-02 05:22:51.594 | [63414] | SdlSig | CreateELMRequest | null0 | HLM(HLMCORE) | SDRUI(NBI-HLM-v90-LIB0) | null 2016-09-02 17:22:51,594 INFO [63414] HLMCoreAgent:null0_CreateELMRequest is fired... 2016-09-02 17:22:51,594 INFO [63414] HLMCoreAgent:null0_CreateELMRequest recieved 2016-09-02 17:22:51,595 INFO [63414] HLMCoreAgent:elmName PLM2 2016-09-02 17:22:51,595 INFO [63414] HLMCoreAgent:elmHost 10.48.55.29 2016-09-02 17:22:51,595 INFO [63414] HLMCoreAgent:elmUser administrator 2016-09-02 17:22:51,595 INFO [63414] HLMCoreAgent:elmPass ****** 2016-09-02 17:22:51,595 INFO [63414] HLMCoreAgent:deploymentMode HCS 2016-09-02 17:22:51,595 INFO [63414] HLMCoreAgent:null0_CreateELMRequest opening REST connection to supplied LM 2016-09-02 17:22:51,595 INFO [63414] ELMApi: init, connecting to 10.48.55.29 2016-09-02 17:22:51,595 DEBUG [63414] Got the sslCtx 2016-09-02 17:22:51,595 INFO [63414] Initialized the context 2016-09-02 17:22:51,673 INFO [63414] ELMApi: initialized with connection timeout, connected to 10.48.55.29 2016-09-02 17:22:51,673 INFO [63414] HLMCoreAgent:null0_CreateELMRequest validating connection to LM 2016-09-02 17:22:51,673 DEBUG [63414] ELMApi: validateConnection Called 2016-09-02 17:22:53,546 ERROR [63414] HLMCoreAgent:CreateELMRequest Exception: GET https://10.48.55.29/elm-resources returned a response status of 401 Unauthorized 2016-09-02 17:22:53,548 DEBUG [63414] Agent: sending to [SDRUI] ------- <com.cisco.hcs.HLM.jms.CreateELMResponse> <messageType>CreateELMResponse</messageType> <source> <serviceName>HLM</serviceName> <instance>HLMCORE</instance> </source> <destination> <serviceName>SDRUI</serviceName> <instance>NBI-HLM-v90-LIB0</instance> </destination> <sessionID>06e919ba-1c30-47bb-b72d-a4a0a1ca049c</sessionID> <fault>true</fault> <exception> <detailMessage>GET https://10.48.55.29/elm-resources returned a response status of 401 Unauthorized</detailMessage> <stackTrace> <trace>com.cisco.hcs.HLM.core.HLMCoreAgent$null0_CreateELMRequest.fire(HLMCoreAgent.java:363)</trace>
Add the the PLM OS Admin User Id and Password.
HLM is able to contact the PLM, however PLM replies back with an incompatible version.
2016-09-05 11:53:19,131 INFO [63394] HLMCoreAgent:null0_CreateELMRequest is fired... 2016-09-05 11:53:19,131 INFO [63394] HLMCoreAgent:null0_CreateELMRequest recieved 2016-09-05 11:53:19,132 INFO [63394] HLMCoreAgent:elmName PLM11 2016-09-05 11:53:19,132 INFO [63394] HLMCoreAgent:elmHost 10.48.53.120 2016-09-05 11:53:19,132 INFO [63394] HLMCoreAgent:elmUser admin 2016-09-05 11:53:19,132 INFO [63394] HLMCoreAgent:elmPass ****** 2016-09-05 11:53:19,132 INFO [63394] HLMCoreAgent:deploymentMode HCS 2016-09-05 11:53:19,132 INFO [63394] HLMCoreAgent:null0_CreateELMRequest opening REST connection to supplied LM 2016-09-05 11:53:19,132 INFO [63394] ELMApi: init, connecting to 10.48.53.120 2016-09-05 11:53:19,132 DEBUG [63394] Got the sslCtx 2016-09-05 11:53:19,132 INFO [63394] Initialized the context 2016-09-05 11:53:19,206 INFO [63394] ELMApi: initialized with connection timeout, connected to 10.48.53.120 2016-09-05 11:53:19,206 INFO [63394] HLMCoreAgent:null0_CreateELMRequest validating connection to LM 2016-09-05 11:53:19,206 DEBUG [63394] ELMApi: validateConnection Called 2016-09-05 11:53:19,457 DEBUG [63394] ELMApi: connection Validated 2016-09-05 11:53:19,457 DEBUG [63394] ELMApi: getELMReg Called 2016-09-05 11:53:19,560 DEBUG [63394] ELMApi: getELMVer Called 2016-09-05 11:53:19,667 DEBUG [63394] HLMCoreAgent:null0_createELMRequest LM PLM11 version is 11.0.1.10000-2 2016-09-05 11:53:19,667 DEBUG [63394] hlmSDRUtil.createELM 2016-09-05 11:53:19,669 DEBUG [63394] hlmSDRUtil.getHCMFDeploymentMode 2016-09-05 11:53:19,669 DEBUG [63394] hlmSDRUtil.beginTransaction 2016-09-05 11:53:19,670 DEBUG [63394] hlmSDRUtilcommitTransaction 2016-09-05 11:53:19,671 DEBUG [63394] hlmSDRUtil.beginTransaction 2016-09-05 11:53:19,672 INFO [63394] LM version is 11.0.1.10000-2 2016-09-05 11:53:19,673 ERROR [63394] hlmSDRUtil.createELM: SDR exception(SDRSyncSession): null 2016-09-05 11:53:19,673 DEBUG [63394] hlmSDRUtil.rollbackTransaction 2016-09-05 11:53:19,673 ERROR [63394] HLMCoreAgent:CreateELMRequest Exception: Attribute TypeELMVersion is required.
Ensure that the PLM release that is added is suported by the HCM-F and is referred to the HCS compatiblity matrix.