Troubleshoot License Sync on Catalyst SD-WAN Manager through On-prem Reporting Mode

Introduction

This document describes how to troubleshoot an error encountered while syncing license on Catalyst SD-WAN Manager through On-prem reporting mode.

Requirements

For scenarios in which Catalyst SD-WAN Manager is not connected directly to the internet, using a proxy server can provide access to internet-based services, such as Cisco SSM, or to a local on-prem SSM.

Minimum release: Catalyst SD-WAN Manager Release 20.9.1

Cisco Smart Software Manager on-prem (SSM on-prem) is a Cisco Smart Licensing solution that enables you to administer licenses from a server on your premises, instead of having to connect directly to Cisco SSM. The solution involves setting up a Cisco SSM on-prem license server, which synchronizes its license database with Cisco SSM periodically and functions similarly to Cisco SSM, while operating locally.

Catalyst SD-WAN Manager supports management of licenses using a Cisco SSM on-prem server, using a mode called on-prem. On-prem mode is useful for organizations that use Cisco SSM on-prem to accommodate a strict security policy that does not permit network devices to communicate with Cisco SSM by direct internet connection.

When operating in on-prem mode, Catalyst SD-WAN Manager synchronizes license information with the Cisco SSM on-prem license server every 24 hours. During this synchronization, Catalyst SD-WAN Manager receives any updates to available licenses and it sends license usage reports to the Cisco SSM on-prem license server. You can synchronize licenses at any time.

Benefits of Using Cisco Smart Software Manager On-Prem

Organizations whose security policies, or other circumstances, require that Catalyst SD-WAN Manager not be connected to the internet have two options for managing licenses for Smart License Using Policy:

• Use offline mode, which requires transferring files manually between Catalyst SD-WAN Manager and Cisco SSM.
• Use a Cisco SSM on-prem server that is accessible through a local area connection to Catalyst SD-WAN Manager.

Both of these methods address the need to transfer license information between Cisco SSM and Catalyst SD-WAN Manager. Wherever it is possible to use the on-prem mode, this mode provides the significant benefit of reducing the maintenance overhead of transferring files manually between Catalyst SD-WAN Manager and Cisco SSM, as is necessary for offline mode.

Error

While syncing the Smart credentials from Catalyst SD-WAN Manager GUI, we get this error:

Failed to authenticate Smart Account credentials.: Failed to authenticate user - 'admin'. {"error":"invalid_client","error_message":"Grant not found. Ensure that the given grant details are correct."}

Troubleshooting Aprroach

• vManage must be on code 20.9.1 or later.
• Check logs on the Catalyst SD-WAN Manager (vmanage-server.logs) while putting Smart Account Credentials on the Catalyst SD-WAN Manager Licence Management Section.
• Make sure correct Client ID and Secret-key shared by on-prem SSM team.
• TCPDUMP on the vManage for CSSM Server IP
• Verify DNS is correctly configured on the Catalyst SD-WAN Manager and able to ping cloudsso.cisco.com
• Involve on-prem SSM team and request SSM team to debug at on-prem server end.

Catalyst SD-WAN Manager IP: 10.66.76.81 / 192.168.10.1

CSSM Server IP: 10.106.66.55

TCPDump on the vManage for SSM server IP:

um8_vManage# tcpdump vpn 0 interface eth0 options "host 10.106.66.55 -nn -vv"

tcpdump -p -i eth0 -s 128 host 10.106.66.55 -nn -vv in VPN 0

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 128 bytes

12:15:06.407513 IP (tos 0x0, ttl 64, id 24618, offset 0, flags [DF], proto TCP (6), length 52)

    192.168.10.1.57886 > 10.106.66.55.8443: Flags [S], cksum 0xfadb (incorrect -> 0xdf91), seq 746386211, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0

12:15:06.651698 IP (tos 0x20, ttl 44, id 0, offset 0, flags [DF], proto TCP (6), length 52)

    10.106.66.55.8443 > 192.168.10.1.57886: Flags [S.], cksum 0x1b34 (correct), seq 2758352947, ack 746386212, win 29200, options [mss 1380,nop,nop,sackOK,nop,wscale 7], length 0

12:15:06.651768 IP (tos 0x0, ttl 64, id 24619, offset 0, flags [DF], proto TCP (6), length 40)

    192.168.10.1.57886 > 10.106.66.55.8443: Flags [.], cksum 0xfacf (incorrect -> 0xcce1), seq 1, ack 1, win 229, length 0

12:15:06.654592 IP (tos 0x0, ttl 64, id 24620, offset 0, flags [DF], proto TCP (6), length 212)

    192.168.10.1.57886 > 10.106.66.55.8443: Flags [P.], seq 1:173, ack 1, win 229, length 172

12:15:06.899695 IP (tos 0x0, ttl 41, id 44470, offset 0, flags [DF], proto TCP (6), length 40)

    10.106.66.55.8443 > 192.168.10.1.57886: Flags [.], cksum 0xcc2d (correct), seq 1, ack 173, win 237, length 0

12:15:06.911484 IP (tos 0x0, ttl 41, id 44471, offset 0, flags [DF], proto TCP (6), length 1420)

    10.106.66.55.8443 > 192.168.10.1.57886: Flags [.], seq 1:1381, ack 173, win 237, length 1380

12:15:06.911542 IP (tos 0x0, ttl 41, id 44472, offset 0, flags [DF], proto TCP (6), length 254)

    10.106.66.55.8443 > 192.168.10.1.57886: Flags [P.], seq 1381:1595, ack 173, win 237, length 214

12:15:06.911573 IP (tos 0x0, ttl 64, id 24621, offset 0, flags [DF], proto TCP (6), length 40)

    192.168.10.1.57886 > 10.106.66.55.8443: Flags [.], cksum 0xfacf (incorrect -> 0xc6bb), seq 173, ack 1381, win 251, length 0

12:15:06.911598 IP (tos 0x0, ttl 64, id 24622, offset 0, flags [DF], proto TCP (6), length 40)

    192.168.10.1.57886 > 10.106.66.55.8443: Flags [.], cksum 0xfacf (incorrect -> 0xc5cf), seq 173, ack 1595, win 273, length 0

12:15:06.923929 IP (tos 0x0, ttl 64, id 24623, offset 0, flags [DF], proto TCP (6), length 234)

    192.168.10.1.57886 > 10.106.66.55.8443: Flags [P.], seq 173:367, ack 1595, win 273, length 194

On-prem Server Logs:

[root@SSM-On-Prem log]# tail -f messages

Jan 13 11:13:36 SSM-On-Prem chronyd[1319]: Source 172.20.226.229https://172.20.226.229 replaced with 172.30.5.123https://172.30.5.123

Jan 13 11:14:09 SSM-On-Prem b09c1e3b5d81: 1:M 13 Jan 2023 11:14:09.049 * 100 changes in 300 seconds. Saving...

Jan 13 11:14:09 SSM-On-Prem b09c1e3b5d81: 1:M 13 Jan 2023 11:14:09.050 * Background saving started by pid 4617

Jan 13 11:14:09 SSM-On-Prem b09c1e3b5d81: 4617:C 13 Jan 2023 11:14:09.052 * DB saved on disk

Jan 13 11:14:09 SSM-On-Prem b09c1e3b5d81: 4617:C 13 Jan 2023 11:14:09.053 * RDB: 0 MB of memory used by copy-on-write

Jan 13 11:14:09 SSM-On-Prem b09c1e3b5d81: 1:M 13 Jan 2023 11:14:09.150 * Background saving terminated with success

Jan 13 11:14:46 SSM-On-Prem 1a1fca641d0a: Redis#exists(key) will return an Integer in redis-rb 4.3. exists? returns a boolean, you should use it instead. To opt-in to the new behavior now you can set Redis.exists_returns_integer =  true. To disable this message and keep the current (boolean) behaviour of 'exists' you can set Redis.exists_returns_integer = false, but this option will be removed in 5.0. (/usr/local/lib/ruby/gems/2.6.0/gems/redis-session-store-0.11.1/lib/redis-session-store.rb:70:in `session_exists?')

Jan 13 11:14:46 SSM-On-Prem 1a1fca641d0a: [active_model_serializers] Rendered UserSerializer with ActiveModelSerializers::Adapter::Attributes (3.0ms)

Jan 13 11:14:46 SSM-On-Prem 1a1fca641d0a: method=GET path=/sessions/get_user format=json controller=SessionsController action=get_user status=200 duration=24.86 view=3.23 db=7.98 params={"controller"=>"sessions", "action"=>"get_user", "session"=>{}} session_id=[FILTERED] uid=admin time=2023-07-13T11:14:46Z http_referer=https://172.20.85.137:8443/admin/ client_ip=10.110.35.124https://10.110.35.124 user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/10.120.0.0 Safari/537.36

Jan 13 11:14:46 SSM-On-Prem 504f06c0d581: 10.110.35.124https://10.110.35.124 - - [13/Jan/2023:11:14:46 +0000] "GET /backend/sessions/get_user HTTP/1.1" 200 271 https://172.20.85.137:8443/admin/ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/10.120.0.0 Safari/537.36" "-"

Jan 13 11:17:01 SSM-On-Prem 504f06c0d581: 2023/07/13 11:17:01 [error] 47#47: *1576 connect() failed (111: Connection refused) while connecting to upstream, client: 10.66.76.85, server: , request: "POST /backend/oauth/token?grant_type=password&client_id=jF6qntGFm429n2B7tlgJPNHueoyjjn8p6zAn7BhYLUhjAY5BZlTUV9Q0r_Zk7uBf&client_secret=d1b_ZV4QWs4UzB00_YYHz0Ek_qxcvywN4d2GD40LQOyKfMOV6MoKmVchUBX3GsGx&username=admin&password=CiscoLab%21234567 HTTP/1.1", upstream: http://[fd00:dead:beef::5]:3000/oauth/token?grant_type=password&client_id=jF6qntGFm429n2B7tlgJPNHueoyjjn8p6zAn7BhYLUhjAY5BZlTUV9Q0r_Zk7uBf&client_secret=d1b_ZV4QWs4UzB00_YYHz0Ek_qxcvywN4d2GD40LQOyKfMOV6MoKmVchUBX3GsGx&username=admin&password=CiscoLab%21234567, host: "172.20.85.137:8443"

Jan 13 11:17:01 SSM-On-Prem 504f06c0d581: 2023/07/13 11:17:01 [warn] 47#47: *1576 upstream server temporarily disabled while connecting to upstream, client: 10.66.76.85, server: , request: "POST /backend/oauth/token?grant_type=password&client_id=jF6qntGFm429n2B7tlgJPNHueoyjjn8p6zAn7BhYLUhjAY5BZlTUV9Q0r_Zk7uBf&client_secret=d1b_ZV4QWs4UzB00_YYHz0Ek_qxcvywN4d2GD40LQOyKfMOV6MoKmVchUBX3GsGx&username=admin&password=CiscoLab%21234567 HTTP/1.1", upstream: http://[fd00:dead:beef::5]:3000/oauth/token?grant_type=password&client_id=jF6qntGFm429n2B7tlgJPNHueoyjjn8p6zAn7BhYLUhjAY5BZlTUV9Q0r_Zk7uBf&client_secret=d1b_ZV4QWs4UzB00_YYHz0Ek_qxcvywN4d2GD40LQOyKfMOV6MoKmVchUBX3GsGx&username=admin&password=CiscoLab%21234567, host: "172.20.85.137:8443"

Jan 13 11:17:01 SSM-On-Prem 1a1fca641d0a: [active_model_serializers] Rendered ActiveModel::Serializer::Null with Hash (0.09ms)

Jan 13 11:17:01 SSM-On-Prem 1a1fca641d0a: method=POST path=/oauth/token format=json controller=Doorkeeper::OauthTokensController action=create status=403 duration=4.21 view=0.53 db=1.12

Jan 13 11:17:01 SSM-On-Prem 504f06c0d581: 10.66.76.85https://10.66.76.85 - - [13/Jan/2023:11:17:01 +0000] "POST /backend/oauth/token?grant_type=password&client_id=jF6qntGFm429n2B7tlgJPNHueoyjjn8p6zAn7BhYLUhjAY5BZlTUV9Q0r_Zk7uBf&client_secret=d1b_ZV4QWs4UzB00_YYHz0Ek_qxcvywN4d2GD40LQOyKfMOV6MoKmVchUBX3GsGx&username=admin&password=CiscoLab%21234567 HTTP/1.1" 403 121 "-" "Apache-HttpClient/4.5.11 (Java/11.0.7)" "-"

Jan 13 11:17:14 SSM-On-Prem 1a1fca641d0a: [INFO] Session expiring outcome=success

Logs on the vManage while putting Smart Accounts details in vManage Licence Management Section:

13-Jan-2023 17:29:02,775 IST INFO  [um8_vManage] [SmartLicensingIntegrationManager] (default task-24) |default| user is using in On Prem mode

13-Jan-2023 17:29:02,776 IST INFO  [um8_vManage] [SmartLicensingIntegrationManager] (default task-24) |default| Authenticating on-prem server

13-Jan-2023 17:29:02,780 IST INFO  [um8_vManage] [AbstractSettingsManager] (default task-24) |default| Found smart licensing mode is onprem

13-Jan-2023 17:29:02,781 IST INFO  [um8_vManage] [SmartLicensingUtil] (default task-24) |default| intializing client Map {clientUrl=172.20.85.137https://172.20.85.137, client_secret=d1b_ZV4QWs4UzB00_YYHz0Ek_qxcvywN4d2GD40LQOyKfMOV6MoKmVchUBX3GsGx, client_id=jF6qntGFm429n2B7tlgJPNHueoyjjn8p6zAn7BhYLUhjAY5BZlTUV9Q0r_Zk7uBf}

13-Jan-2023 17:29:02,781 IST INFO  [um8_vManage] [SmartLicensingUtil] (default task-24) |default| Getting onPrem server details

13-Jan-2023 17:29:02,793 IST INFO  [um8_vManage] [RestAPIClient] (default task-24) |default| RestAPI proxy host

13-Jan-2023 17:29:02,793 IST INFO  [um8_vManage] [RestAPIClient] (default task-24) |default| RestAPI proxy port

13-Jan-2023 17:29:02,798 IST INFO  [um8_vManage] [SmartLicensingUtil] (default task-24) |default| URL backend/oauth/token?grant_type=password&client_id=jF6qntGFm429n2B7tlgJPNHueoyjjn8p6zAn7BhYLUhjAY5BZlTUV9Q0r_Zk7uBf&client_secret=d1b_ZV4QWs4UzB00_YYHz0Ek_qxcvywN4d2GD40LQOyKfMOV6MoKmVchUBX3GsGx&username=ptundalw&password=Carnation%2321

13-Jan-2023 17:29:02,798 IST INFO  [um8_vManage] [SmartLicensingUtil] (default task-24) |default| Query Smart Account: backend/oauth/token?grant_type=password&client_id=jF6qntGFm429n2B7tlgJPNHueoyjjn8p6zAn7BhYLUhjAY5BZlTUV9Q0r_Zk7uBf&client_secret=d1b_ZV4QWs4UzB00_YYHz0Ek_qxcvywN4d2GD40LQOyKfMOV6MoKmVchUBX3GsGx&username=ptundalw&password=*******

13-Jan-2023 17:29:03,490 IST ERROR [um8_vManage] [RestAPIClient] (default task-24) |default| Failed to process POST request uri: backend/oauth/token?grant_type=password&client_id=jF6qntGFm429n2B7tlgJPNHueoyjjn8p6zAn7BhYLUhjAY5BZlTUV9Q0r_Zk7uBf&client_secret=d1b_ZV4QWs4UzB00_YYHz0Ek_qxcvywN4d2GD40LQOyKfMOV6MoKmVchUBX3GsGx&username=ptundalw&password=******* . Code: 403 Message: {"error":"invalid_client","error_message":"Grant not found. Ensure that the given grant details are correct."}

13-Jan-2023 17:29:03,491 IST ERROR [um8_vManage] [SmartLicensingUtil] (default task-24) |default| Failed to authenticate user - 'ptundalw'.

13-Jan-2023 17:29:03,491 IST ERROR [um8_vManage] [SmartLicensingIntegrationRestfulResource] (default task-24) |default| Smart Account User Authentication failed.

Note: We are getting error 403 while syncing the smart account from vManage GUI which indicates that the server understands the request but refuses to authorize it.

Workaround

1. Login to On Prem Server.
2. Navigate to API Tool Kit.
3. Select "Resource Owner Grant", Enter the details as Name and save.
   
   

4. Select the saved record (mentioned in previous snapshot) and check Client ID and Client Secret.

5. Share and enter the shared Client ID and Client Secret in Catalyst SD-WAN Manager portal.
6. Go to "Sync Licenses and Refresh Devices" in vManage and use the same on prem credentials with which you logged in to generate client ID and client secret.