Field Notice: FN - 72428 - Expired PKI Certificate on DNA Center Causes AI Network Analytics Features to Fail - Software Upgrade Recommended

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.0	25-Jul-22	Initial Release

Products Affected

Affected OS Type	Affected Software Product	Affected Release	Affected Release Number	Comments
NON-IOS	DNA Center Software	2	2.1.2.7, 2.1.2.8, 2.2.1.0, 2.2.2.0, 2.2.2.3, 2.2.2.4, 2.2.2.5, 2.2.2.6, 2.2.2.7, 2.2.2.8, 2.2.3.0, 2.2.3.3	Note: versions prior to 2.1.2.7 are omitted because the problem described in this field notice is superseded by https://www.cisco.com/c/en/us/support/docs/field-notices/723/fn72359.html in those earlier versions.

Defect Information

Defect ID	Headline
CSCwa78814	Cisco AI Cloud agent client certificate renewal

Problem Description

Affected versions of Cisco Digital Network Architecture (DNA) Center include a PKI (Public Key Infrastructure) certificate for Cisco AI Network Analytics that does not automatically renew and cannot be manually renewed by the user. Once this certificate expires, Cisco DNA Center will fail to establish a secure connection with the Cisco AI Network Analytics cloud. As a result, all Cisco DNA Center AI Analytics and AI Endpoint Analytics features will no longer function.  This problem is not applicable to Airgap versions of Cisco DNA Center.

Background

In Cisco DNA Center, the AI Network Analytics agent communicates with the Cisco AI Network Analytics cloud via Transport Layer Security (TLS) sessions authenticated by PKI X.509 certificates. The agent client certificate is stored on Cisco DNA Center and has a three year expiration. Affected versions of Cisco DNA Center do not have the capability to renew the agent client certificate without manual intervention from authorized Cisco personnel. Once the certificate expires, the agent can no longer communicate with the cloud and all AI Network Analytics features cease to function.

Note: Cisco AI Network Analytics agent client certificates signed after June 2021 have a one year expiration.

The features shown here are impacted by this problem:

• Cisco AI Network Analytics
  • AI Driven Issues
  • Network Heatmaps
  • Network Trends and Insights
  • Network Comparisons
  • Peer Comparisons
• Cisco AI Endpoint Analytics
  • Smart Grouping
  • Download/Upgrade of AI Spoofing Detection Models
• Cisco AI Enhanced Radio Resource Management (RRM)

There are two ways you can encounter an expired agent client certificate:

1. The client certificate on an affected system passes its expiration date.
2. A recovery operation is performed on any system (including fixed software versions) using a stale backup configuration file that contains an expired client certificate.

Problem Symptom

Affected systems with an unexpired Cisco AI Analytics agent client certificate will not exhibit any failure symptoms. There is no proactive notification on the Cisco DNA Center user interface to indicate an agent client certificate that is about to expire. Follow the instructions shown here in order to determine the agent client certificate expiration date on your system.

How to Find the Cisco AI Network Analytics Agent Client Certificate Expiration Date

The Cisco AI Analytics agent’s client certificate is stored along with the AI Network Analytics configuration. Use the Cisco DNA Center CLI commands shown here in order to display the expiration date:

export MAGLEV_TOKEN=$(cat ~/.maglevconf | grep token | awk -F '= ' '{print $2}')

curl -s -X GET -H "X-Auth-Token:$MAGLEV_TOKEN" -k http://kairos-agent.ai-network-analytics.svc.cluster.local:8089/api/v1/config | jq -r '."client-cert"' | openssl x509 -text | grep "Not After"

The example shown here shows the result returned. The certificate expiration date is highlighted for clarity.

$ curl -s -X GET -H "X-Auth-Token:$MAGLEV_TOKEN" -k http://kairos-agent.ai-network-analytics.svc.cluster.local:8089/api/v1/config | jq -r '."client-cert"' | openssl x509 -text | grep "Not After"   

Not After : Feb 26 13:03:00 2023 GMT

Symptoms of an Expired Agent Client Certificate

Systems with an expired Cisco AI Network Analytics agent client certificate produce symptoms in common with other certificate and connectivity problems. Like with all of those problem types, the secure connection from Cisco DNA Center to the Cisco AI Analytics cloud will fail. The connection failure can be observed from the Cisco DNA Center user interface and in logs. If any of the symptoms shown here occur on your system, use the certificate expiration date instructions in the previous section in order to determine if the failure is caused by an expired agent client certificate.

• AI Analytics Trends and Insights

Example: Heatmap

Choose Cisco DNA Center UI > Assurance > Trends and Insights > Network Heatmap.

Oops! There is an error fetching data.

• AI Analytics agent logs

The agent logs can be collected through these steps.

1. Run this command on the Cisco DNA Center Appliance CLI.

   $ magctl service logs -a ai-network-analytics kairos-agent

2. Open the Cisco DNA Center Root Cause Analysis (RCA) bundle. The logs will show errors similar to this example:

{
    "component": "apiproxy",
    ...
    "level": "warning",
    "method": "POST",
    "msg": "api proxy: request failed",
    "object": "ProxyContext",
    "package": "proxy",
    "resBody": "<html>\r\n<head><title>400 The SSL certificate error</title></head>\r\n<body>\r\n<center><h1>400 Bad Request</h1></center>\r\n<center>The SSL certificate error</center>\r\n</body>\r\n</html>\r\n",
    "status": 400,
    ...
}

Workaround/Solution

There are two solution options. The choice depends on whether the agent client certificate has expired.

Solution for Systems with an Unexpired Certificate

A software fix is available for Cisco DNA Center. The fix implements an automatic X.509 certificate renewal process for the agent client certificate in order to prevent expiration. Cisco recommends customers upgrade their affected Cisco DNA Center appliances to one of these versions with the fix:

• Cisco DNA Center version 2.2.2.9 or later
• Cisco DNA Center version 2.2.3.4 or later
• Cisco DNA Center version 2.3.2.1 or later

Note: The software fix will prevent the expiration of a Cisco AI Network Analytics agent client certificate, but it cannot renew a certificate that has already expired. If your agent client certificate has already expired, you must choose the solution for an expired certificate as shown in the next section.

Solution for Systems with an Expired Certificate

If the agent client certificate has expired on your system, the certificate must be renewed in order to restore Cisco AI Network Analytics features. The certificate can only be renewed via a manual process performed on your system and must be performed by authorized Cisco personnel. Contact the Cisco Technical Assistance Center (TAC) for assistance with the certificate renewal.

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email or telephone

Receive Email Notification For New Field Notices

My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.