Field Notice: FN74116 - Risk of Configuration Loss in Cisco Application Policy Infrastructure Controllers During Power Event - Software Upgrade Recommended

Cisco Application Policy Infrastructure Controllers (APICs) are vulnerable to data loss during any ungraceful reload, such as a power outage or physical removal of power. This may result in the Cisco APICs coming back in a cluster diverged state or the configuration being lost upon power up.

A configuration backup with a known AES encryption key can be used to restore any missing configuration to the Cisco APICs after an ungraceful reload.

It is strongly recommended that all Cisco Application Centric Infrastructure (ACI) operators have recent configuration backups available on a system outside of the Cisco APIC and that all encryption passwords are clearly documented.

This problem exists for all Cisco APIC appliances that are running an affected 5.2 or 6.0 software release and experience a physical loss of power. Software reboots or power cycles through the Cisco Integrated Management Controller (IMC) are not affected.

One or more of the following symptoms may be observed after an ungraceful reload or power outage:

• User configuration is missing on Cisco APICs and switches.
• Data Management Engine (DME) services fail with a fatal error that indicates that it could not open or read the clone database.
  • Example: Cannot create or open DB: /var/run/mgmt/db/ifc_plgnhandler/S24_R3_clone/ifc_plgnhandler.db, rv = 26
• The Cisco APIC cluster is partially diverged.

Solution

Upgrade the Cisco ACI fabric to a fixed software release prior to any power event. Software is available from the Software Center on Cisco.com.

If a power outage or ungraceful reload has occurred, call the Cisco technical Assistance Center (TAC) to perform the following steps:

1. Remove any empty or corrupted clone directories from the filesystem. This must be performed as a root user.
2. Restart the affected processes and ensure that the Cisco APIC pulls a working database copy from another Cisco APIC.

To determine whether a Cisco APIC is affected by the software defect, check the current software release using the show version command, as shown in the following example:

apic1# show version
 Role        Pod         Node        Name                      Version
 ----------  ----------  ----------  ------------------------  --------------------
 controller  1           1           apic1                     6.0(4c)
 controller  1           2           apic2                     6.0(4c)
 controller  1           3           apic3                     6.0(4c)

A Cisco APIC is affected by the software defect if the cluster is diverged and there are log lines that show last term 0 last index 0. These two conditions can be verified with the following commands:

apic1# show faults code F0321 controller
Code            : F0321
Severity        : critical
Last Transition : 2024-01-23T15:44:27.585-05:00
Lifecycle       : raised
DN              : topology/pod-1/node-1/av/node-1/fault-F0321
Description     : Controller 1 is unhealthy because: Data Layer Partially Diverged
                 
apic1# zgrep "last term = 0 last index = 0" /var/log/dme/log/svc_ifc_*
svc_ifc_policymgr.bin.log.60.gz:5124||2024-01-23T15:42:21.784791378+00:00||instance||INFO||||MIT at /var/run/mgmt/db/ifc_policymgr/S8_R1_clone/ifc_policymgr.db last term = 0 last index = 0||../common/src/framework/./core/shard/Instance.cc||1194
svc_ifc_policymgr.bin.log.60.gz:14609||2024-01-23T15:43:05.266039342+00:00||instance||INFO||||MIT at /var/run/mgmt/db/ifc_policymgr/S8_R1_clone/ifc_policymgr.db last term = 0 last index = 0||../common/src/framework/./core/shard/Instance.cc||1194
svc_ifc_policymgr.bin.log.60.gz:14654||2024-01-23T15:43:19.863083592+00:00||instance||INFO||||MIT at /var/run/mgmt/db/ifc_policymgr/S8_R1_clone/ifc_policymgr.db last term = 0 last index = 0||../common/src/framework/./core/shard/Instance.cc||1194