Field Notice: FN74201 - Cisco UCS Manager Becomes Inaccessible After Upgrade with Integrated S3260 Server - Software Upgrade Recommended

Upgrading to Cisco UCS Manager Release 4.3(4a), 4.3(4b), or 4.3(4c) will fail, and the GUI may become inaccessible. This will occur if a Cisco UCS S3260 Storage Server that is integrated into UCS Manager contains a local logical unit number (LUN) configuration as part of the storage profile.

If an S3260 server with a storage profile that contains a local LUN or LUN set configuration is applied prior to an upgrade, the upgrade will fail to activate UCS Manager, and UCS Manager will become unresponsive or inaccessible.

If a storage profile that contains a local LUN or LUN set configuration is added to an S3260 server that is running Cisco UCS Manager Release 4.3(4a), 4.3(4b), or 4.3(4c), UCS Manager will become unresponsive or inaccessible.

The data management engine (DME) maintains a database of managed virtual objects (MOs), such as service profile mappings or MAC addresses. All identifiers, policies, profiles, and other data within the UCS Manager have associated MOs. Certain checks are in place to ensure that the database is error-free and that all MOs have valid data. Corrupted or null database entries can cause failures of the DME. Cisco UCS Manager releases 4.3(4a), 4.3(4b), and 4.3(4c) introduce a condition that can cause a database fault when a local storage policy is associated to an S3260 server.

An upgrade to Cisco UCS Manager Release 4.3(4a), 4.3(4b), or 4.3(4c) will fail, and the UCS Manager DME will fail. All commands that require UCS Manager will fail to execute. In most cases, the UCS Manager will remain inaccessible and require recovery. 

If a local LUN is added to a storage profile of an S3260 server after it has been upgraded to an affected release, the internal DME process will fail, and the UCS Manager will become inaccessible. A user will need to contact the Cisco Technical Assistance Center (TAC) for recovery steps.

If a system has encountered this issue, the cluster state in the CLI will show HA NOT READY, as shown in this example:

UCS-A# connect localUCS-A(local-mgmt)

# show cluster extended-state
Cluster Id: 0xeb576646312e44ae-0xa5090b9636da34f3

Start time: Sat Aug 10 22:13:24 2024
Last election time: Sat Aug 10 22:26:02 2024

B: UP, INAPPLICABLE, (Management services: DOWN)
A: UP, INAPPLICABLE, (Management services: DOWN)

B: memb state UP, lead state INAPPLICABLE, mgmt services state: DOWN
A: memb state UP, lead state INAPPLICABLE, mgmt services state: DOWN
heartbeat state PRIMARY\_OK

INTERNAL NETWORK INTERFACES:
eth3, UP
eth4, UP

HA NOT READY
Management services are unresponsive on local Fabric Interconnect

The output of show pmon state will show the DME as failed.

UCS-A(local-mgmt)
# show pmon state
SERVICE NAME             STATE     RETRY(MAX)    EXITCODE    SIGNAL    CORE
------------             -----     ----------    --------    ------    ----
svc_sam_controller     running           0(4)           0         0      no
svc_sam_dme            failed          321(4)           0        11      yes  <-------
svc_sam_dcosAG         running           0(4)           0         0      no
svc_sam_bladeAG        running           0(4)           0         0      no
svc_sam_portAG         running           0(4)           0         0      no
svc_sam_statsAG        running           0(4)           0         0      no
svc_sam_hostagentAG    running           0(4)           0         0      no
svc_sam_nicAG          running           0(4)           0         0      no
svc_sam_licenseAG      running           0(4)           0         0      no
svc_sam_extvmmAG       running           0(4)           0         0      no
httpd.sh               running           0(4)           0         0      no
httpd_cimc.sh          running           0(4)           0         0      no
svc_sam_sessionmgrAG   running           0(4)           0         0      no
svc_sam_pamProxy       running           0(4)           0         0      no
dhcpd                  running           0(4)           0         0      no
sam_core_mon           running           0(4)           0         0      no
svc_sam_netSnmpAG      running           0(4)           0         0      no
svc_sam_rsdAG          running           0(4)           0         0      no
svc_sam_svcmonAG       running           0(4)           0         0      no
ucsm_tftpdv6           running           0(4)           0         0      no
svc_sam_samcproxy      running           0(4)           0         0      no
svc_sam_samcstatsproxy   running           0(4)           0         0      no 

Prior to upgrading to Cisco UCS Manager Release 4.3(4a), 4.3(4b), or 4.3(4c), detach the storage profile that contains a local LUN object from the S3260 server profile. This action will delete the MO association between the service profile and the local LUN but will not delete the LUN from the server itself. The deployed server will be able to access local LUNs even though the MO for the LUN is no longer present in the database. 

• Do not reattach the storage profile to an S3260 server after a successful upgrade to an affected release. Associating a local LUN policy to the S3260 server on an affected release will render the UCS Manager unmanageable. 
• An unsuccessful upgrade should automatically fail and roll back to the previous UCS Manager release.
• If a UCS domain is affected by CSCwm49499 after an upgrade and the UCS Manager has been rendered unmanageable, contact Cisco TAC for assistance in recovering the domain.

Download the latest release of Unified Computing System (UCS) Infrastructure Software Bundle from the Cisco Software Center.

This issue affects any UCS domain that uses UCS Manager and has UCS S3260 servers with a specific storage profile configuration. The presence of a local LUN or LUN set configuration will trigger this UCS Manager failure.

To encounter this software issue, a UCS domain must have S3260 servers with local LUN configurations. If a UCS domain does not have integrated S3260 servers, it is not affected by this issue.

Standalone or Intersight Managed S3260 servers are not affected by this issue.

To determine if a serial number is affected, see the Serial Number Validation section of this field notice.