Field Notice: FN - 72290 - Cisco IOS XR: QuoVadis Root CA 2 Decommission Might Affect Smart Licensing and Smart Call Home Functionality - Workaround Provided

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.1	21-Mar-22	Updated the Workaround/Solution Section and Added the Additional Information Section
1.0	24-Feb-22	Initial Release

Products Affected

Affected OS Type	Affected Software Product	Affected Release	Affected Release Number	Comments
NON-IOS	IOS XR Software	7	7.0.0, 7.0.1, 7.0.2, 7.1.1, 7.1.2, 7.1.3, 7.2.0, 7.2.1, 7.2.2, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.4.1, 7.5.1
NON-IOS	IOS XR Software	6	6.0.0, 6.0.1, 6.0.2, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.2.1, 6.2.2, 6.2.3, 6.3.1, 6.3.2, 6.3.3, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.5.1, 6.5.2, 6.5.3, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.7.1, 6.7.2, 6.7.3, 6.7.4, 6.8.1

Defect Information

Defect ID	Headline
CSCvx00476	QuoVadis root CA decommission on all

Problem Description

For affected versions of the Cisco IOS^® XR products, some Secure Socket Layer (SSL) certificates issued from the QuoVadis root certificate authority (CA) trust chain before March 31, 2021 cannot be renewed from this CA. Once those certificates expire on devices or are removed from the Cisco cloud servers, functions such as Smart Licensing and Smart Call Home will fail to establish secure connections to Cisco and might not operate properly.

Background

The QuoVadis Root CA 2 Public Key Infrastructure (PKI) used by Cisco IOS XR products to issue SSL certificates is subject to an industry-wide issue that affects revocation abilities. Due to this issue, no new QuoVadis Root CA 2 certificates will be issued or renewed by Cisco after March 31, 2021. This affects certificate renewals on devices, Cisco cloud servers, and third-party services.

Certificates issued before the QuoVadis Root CA 2 was decommissioned will continue to be valid. However, the certificates will not renew when they expire on either the device or the Cisco cloud server. This will cause functions such as Smart Licensing and Smart Call Home to fail to establish secure connections to Cisco cloud servers.

This table shows a summary of the QuoVadis Root CA 2 certificate expiration dates for affected Cisco services.

Cisco Cloud Server	QuoVadis Certificate Expiration Date	Affected Services
tools.cisco.com	February 5, 2022	Smart Licensing Smart Call Home
smartreceiver.cisco.com	January 26, 2023	Smart Licensing

Problem Symptom

Expiration of the QuoVadis Root CA 2 certificates affects these services with the associated symptoms.

Affected Services	Symptoms for Affected Services
Smart Licensing	Failure to connect to the server (Details are provided in this section)
Smart Call Home	Failure to connect to the server and the Call-Home HTTP request fails

 

For Cisco IOS XR software, affected devices will be unable to connect to the Smart Licensing and Smart Call Home services hosted by Cisco. Smart licenses might fail entitlement and reflect an Out of Compliance status.

The features that use Smart Licensing will continue to function for up to one year after the last successful secure connection.

Some Smart Licensing symptoms are:

• The device might indicate a failure to communicate with the Smart Licensing server within 30 days from the last successful connection.
• The device will show the "Authorization Expired" state if there is no communication with the Smart Licensing server within 90 days.
• The device will show the "Unregistered" state if there is no communication with the Smart Licensing server after one year and the licensed features usage become suspended.

Smart Licensing Probable Symptoms

If Smart License registration fails, these syslog messages will be seen:

RP/0/RP0/CPU0:Feb 22 10:00:07.962 UTC: smartlicserver[127]: %LICENSE-SMART_LIC-3-COMM_FAILED : Communications failure with the Cisco Smart Software Manager (CSSM) : Fail to send out Call Home HTTP message
RP/0/RP0/CPU0:Feb 22 10:00:07.962 UTC: smartlicserver[127]: %LICENSE-SMART_LIC-3-AGENT_REG_FAILED : Smart Agent for Licensing Registration with the Cisco Smart Software Manager (CSSM) failed: Fail to send out Call Home HTTP message

Smart Call Home Symptoms

This error log might be observed in the affected device:

RP/0/RP0/CPU0:Feb 22 09:57:36.246 UTC: http_client[255]: %SECURITY-XR_SSL-3-CERT_VERIFY_ERR_2_PARAM : SSL certificate verify error: Peer certificate verification failed - no trusted cert 'Crypto Engine' detected the 'warning' condition 'Invalid trustpoint or trustpoint not exist'

If the download fails, this message might be seen:

RP/0/RP0/CPU0:Feb 22 09:57:24.025 UTC: cepki[279]: %SECURITY-PKI-6-ERR_1_PARAM : Download failed, HTTP returned an error

Note: Offline licensing, such as Permanent License Reservation (PLR) and Specific License Reservation (SLR), is not affected by the certificate change on the Smart Licensing server.

For additional information, refer to the Cisco Smart Licensing Guide and the Cisco IOS XR Platform Guide for your specific version of Cisco IOS XR software.

This log shows the license authorization failure:

RP/0/RP0/CPU0:ios#show license status
Wed Feb 23 06:08:07.562 UTC
 
Smart Licensing is ENABLED
 
Utility:
  Status: DISABLED
 
Data Privacy:
  Sending Hostname: yes
    Callhome hostname privacy: DISABLED
    Smart Licensing hostname privacy: DISABLED
  Version privacy: DISABLED
 
Transport:
  Type: Callhome
 
Registration:
  Status: REGISTERED
  Smart Account: BU Production Test 1
  Virtual Account: NCS550 Testing
  Export-Controlled Functionality: ALLOWED
  Initial Registration: SUCCEEDED on Feb 23 2022 05:56:41 UTC
  Last Renewal Attempt: None
  Next Renewal Attempt: Aug 22 2022 05:56:40 UTC
  Registration Expires: Feb 23 2023 05:51:36 UTC 

License Authorization:
  Status: OUT OF COMPLIANCE on Feb 23 2022 05:56:50 UTC
  Last Communication Attempt: FAILED on Feb 23 2022 06:08:07 UTC
    Failure reason: Fail to send out Call Home HTTP message
  Next Communication Attempt: Feb 23 2022 06:08:36 UTC
  Communication Deadline: May 24 2022 05:56:21 UTC
 
Export Authorization Key:
  Features Authorized:
    <none> 

Miscellaneous:
  Custom Id: <empty>
RP/0/RP0/CPU0:ios#

Workaround/Solution

Cisco has migrated from the QuoVadis Root CA 2 to the IdenTrust Commercial Root CA 1 for SSL certificates. Cisco recommends a manual certificate update to add the new IdenTrust Commercial Root CA 1 certificate to the Cisco IOS XR products.

Manual Certificate Update

In order to check the built-in and downloaded certificates installed on your device, enter the show crypto ca trustpool command.

Example:

RP/0/RP0/CPU0:ios#show crypto ca trustpool
Wed Apr  1 10:25:52.995 UTC

Trustpool: Built-In
==================================================
CA certificate
  Serial Number  : 5F:F8:7B:28:2B:54:DC:8D:42:A3:15:B5:68:C9:AD:FF
  Subject:
        CN=Cisco Root CA 2048,O=Cisco Systems
  Issued By      :
        CN=Cisco Root CA 2048,O=Cisco Systems
  Validity Start : 20:17:12 UTC Fri May 14 2004
  Validity End   : 20:25:42 UTC Mon May 14 2029
  SHA1 Fingerprint:
         DE990CED99E0431F60EDC3937E7CD5BF0ED9E5FA

Trustpool: Built-In
==================================================
CA certificate
  Serial Number  : 2E:D2:0E:73:47:D3:33:83:4B:4F:DD:0D:D7:B6:96:7E
  Subject:
        CN=Cisco Root CA M1,O=Cisco
  Issued By      :
        CN=Cisco Root CA M1,O=Cisco
  Validity Start : 21:50:24 UTC Tue Nov 18 2008
  Validity End   : 21:59:46 UTC Fri Nov 18 2033
  SHA1 Fingerprint:
         45AD6BB499011BB4E84E84316A81C27D89EE5CE7

Trustpool: Built-In
==================================================
CA certificate
  Serial Number  : 3C:91:31:CB:1F:F6:D0:1B:0E:9A:B8:D0:44:BF:12:BE
  Subject:
        OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
  Issued By      :
        OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
  Validity Start : 00:00:00 UTC Mon Jan 29 1996
  Validity End   : 23:59:59 UTC Wed Aug 02 2028
  SHA1 Fingerprint:
         A1DB6393916F17E4185509400415C70240B0AE6B

Trustpool: Built-In
==================================================
CA certificate
  Serial Number  : 05:09
  Subject:
        CN=QuoVadis Root CA 2,O=QuoVadis Limited,C=BM
  Issued By      :
        CN=QuoVadis Root CA 2,O=QuoVadis Limited,C=BM
  Validity Start : 18:27:00 UTC Fri Nov 24 2006
  Validity End   : 18:23:33 UTC Mon Nov 24 2031
  SHA1 Fingerprint:
         CA3AFBCF1240364B44B216208880483919937CF7

 

Manual Certificate Update Options

The three options for a manual certificate update are described in this section.

Option 1. Download a Bundle of Certificates From the Default Location

The bundle of certificates can be downloaded from http://www.cisco.com/security/pki/trs/ios.p7b by default. If a device is not able to resolve the domain name, it will not download the pool of certificates. This bundle of certificates has the Smart Licensing root certificates.

Initiate the download with this command:

crypto ca trustpool import url http://www.cisco.com/security/pki/trs/ios.p7b

 

Option 2. Download the Bundle of Certificates by Hosting it Locally

Cisco IOS XR devices support trustpool policy configuration which allows you to change the default CA Bundle URL. This configuration command can be used to change the CA bundle policy.

crypto ca trustpool policy
    cabundle url <local customer defined url>

Download the CA bundle file from the default location in order to host it locally in an HTTP server. This can be leveraged in case of a connectivity issue.

Enter this command in order to display the trustpool policy:

show crypto ca trustpool policy

 

Option 3. File-Based Download

Customers who are not able to authenticate the certificate with options 1 or 2 can use the file-based approach. Download the file from the default URL (http://www.cisco.com/security/pki/trs/ios.p7b) and copy it to the "tmp" directory. You can then enter this command to authenticate the CA certificates to the device.

crypto ca trustpool import url /tmp/pki_bundle_0.p7b

Note: Cisco IOS XR does not support customer-signed CA bundle of certificates. It must be signed by the Cisco M1 root certificate, and it is managed by the Cisco Information Security team.

Enter this command for authorization:

RP/0/RP0/CPU0:ios#license smart renew auth

Check the license status:

RP/0/RP0/CPU0:ios#show license status
Wed Feb 23 06:02:34.825 UTC
 
Smart Licensing is ENABLED
 
Utility:
  Status: DISABLED
 
Data Privacy:
  Sending Hostname: yes
    Callhome hostname privacy: DISABLED
    Smart Licensing hostname privacy: DISABLED
  Version privacy: DISABLED
 
Transport:
  Type: Callhome
 
Registration:
  Status: REGISTERED
  Smart Account: BU Production Test 1
  Virtual Account: NCS550 Testing
  Export-Controlled Functionality: ALLOWED
  Initial Registration: SUCCEEDED on Feb 23 2022 05:56:41 UTC
  Last Renewal Attempt: None
  Next Renewal Attempt: Aug 22 2022 05:56:40 UTC
  Registration Expires: Feb 23 2023 05:51:36 UTC
 
License Authorization:
  Status: OUT OF COMPLIANCE on Feb 23 2022 05:56:50 UTC
  Last Communication Attempt: SUCCEEDED on Feb 23 2022 06:01:22 UTC
  Next Communication Attempt: Feb 23 2022 18:01:21 UTC
  Communication Deadline: May 24 2022 05:56:20 UTC
 
Export Authorization Key:
  Features Authorized:
    <none>
 
Miscellaneous:
  Custom Id: <empty>
RP/0/RP0/CPU0:ios#

Additional Information

Cisco has created a web page to provide customers and partners with additional information on this issue. Consult the QuoVadis Root CA 2 Decommission page for a full list of products affected, associated Field Notices, and frequently asked questions.

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email or telephone

Receive Email Notification For New Field Notices

My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.