Troubleshoot a UCS C-Series mLOM Card that Cannot Communicate with the CIMC

Introduction

This document describes how to troubleshoot a Cisco UCS C-Series Rack Server mLOM card that is unable to communicate with the CIMC.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on these software and hardware versions:

• Cisco Unified Computing System (UCS) C220-M5 running firmware version 4.0(4i)
• Cisco UCS VIC 1387 running firmware version 4.3(3b) (This is the modular LAN On Motherboard (mLOM) adapter)

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

The mLOM is installed in the Cisco UCS controller, compute, Object Storage Disk (OSD) Compute, and ceph nodes.

A virtual interface card (VIC) is the adapter that serves as the mLOM card in the UCS server. 

The mLOM VIC adapter is a field replaceable unit (FRU) that is placed in controller, compute, and storage servers, based on Undercloud and Overcloud networking requirements.

• The controller server has one mLOM with 2x10GigE ports, 1 LOM with 2x1Gb ports, and 1Gb Cisco Integrated Management Controller (CIMC) port.

• The compute server has one mLOM with 2x10GigE ports, 1 LOM with 2x1Gb ports, 2 PCIe cards in slots 1 and 4 with 2x10GigE, and 1 GigE CIMC.

• The OSD compute server has one mLOM with 2x10GigE ports, 1 LOM with 2xGb ports, and 2 PCIe cards in slots 1 and 4 with 2x10GigE ports 10GigE, 1xGb CIMC ports.

• The storage server has one mLOM with 2x10GigE ports, 1 LOM with 2xGb ports, and 1Gb CIMC port.

The mLOM interfaces serve Overcloud provisioning, VNF Management, and Orchestration, by providing VIM networking interfaces for various networks. These networks can include external floating IP networks, internal API networks, storage networks, Storage Management networks, and tenant networks.

Troubleshoot communication issues between the mLOM and the CIMC in order to help maintain functions within these networks.

Symptoms

Symptom 1. The CIMC GUI Displays “Unable to communicate with the Adapter Card – mLOM"

When accessing Networking/Adapter Card - mLOM in the UCS CIMC, the GUI presents the message “Unable to communicate with the Adapter Card – mLOM. Reload the WebUI to get the latest data.”

Symptom 2. The mLOM Does Not Appear in the UCS Adapter List

The mLOM entry is absent from the Adapter List in the UCS Tech Support file labeled tech_support.frupids, located in the UCS Tech Support file path <TMP > tech_support.frupids>.

Under normal circumstances, the mLOM appears in the UCS Tech Support tech_support_frupids Adapter List.

AdapterList:
…
Slot: MLOM
Description: Cisco UCS VIC1387 Virtual Interface Card - Dual Port 40Gb QSFP+ MLOM
PID: UCSC-MLOM-C40Q-03
CPN: xxxxx
powMin: xxxxx
powMax: xxxxx
Vendor: xxxxx
Device: xxxxx
SubVendor: xxxxx
SubDevice: xxxxx

Symptom 3. The mLOM Does Not Appear Within CIMC CLI Scope Chassis outputs for "scope adapter mlom"

C220-node/chassis # scope adapter mlom
Error: Managed object does not exist, use show command to list valid targets

Symptom 4. The mLOM is Not Detected by the IPMI Sensors

The mLOM card is not detected by the Intelligent Platform Management Interface (IPMI) sensors, per the UCS Tech Support file <UCS Tech support > tmp > tech_support>.

Running: dumpIPMISensors
Querying All IPMI Sensors:
Sensor Name | Reading | Unit | Status | LNR | LC | LNC | UNC | UC | UNR |
=================|=========|==============|========|=========|=========|=========|=========|=========|=========|
MLOM_PRSNT | disc -> | discrete | 0x0280 | na | na | na | na | na | na |
MLOM_TEMP | 38.000 | degrees C | OK | na | na | na | na | 90.000 | 95.000 |

mLOM Card Partial Detection

The CIMC observes server inventory through sensors and connections on the server motherboard, but the server operating system observes hardware through server data channels. Since the CIMC and the server operating system inventory use different physical paths in order to monitor the mLOM card, the mLOM card can be partially detected or CIMC inventory data can show different results than the server operating system.

• The mLOM can appear in the Field Replaceable Unit (FRU) Record <UCS Tech Support > TMP > tech_support.frupids>.

====== Dumping IPMI FRU Records ======
…
FRU Device Description : FRU_MLOM (ID
Board Mfg : Cisco Systems Inc
Board Product : UCSC-MLOM-C40Q-03
Board Serial : FCH2328764C
Board Part Number : 73-17793-06
Board Extra : A03V04
Board Extra : 0000000000
OEM (0xUnknown (0x9)) Record

• The mLOM presence can appear to have been asserted in the UCS Tech Support SEL Decodes <UCS Tech Support > var > sel_decode>.

Id: 757
severity: Normal
dateTime: 2023-12-29 11:08:15 EST
dateTimeOrder: 00028
description: "FRU_MLOM MLOM_PRSNT: Presence sensor for FRU_MLOM, Device Inserted / Device Present was asserted"

• The BIOS Tech log <UCS Tech Support > mnt > jffs2 > BIOS > bt > BiosTech.log> can show that the mLOM card was detected previously.

7:2023 Dec 29 11:04:33 EST:mLomPresent = TRUE
7:2023 Dec 29 11:04:33 EST:mLomSku = mLOM-x8

• The mLOM can show in the UCS Tech Support Inventoy Adapter list and FRU Details list <UCS Tech Support > var > inventory-all>.

Adapter List
slot-number: MLOM
serial-number: FCH2328764C
card-type: 81
asic-type: 1
product-id: UCSC-MLOM-C40Q-03
asic-name: cruz
hw-part-no: 73-17793-06
hw-revision: 3
i2cLogicalBus: 112
new-card: no
active: no
standby-power: no
overtemp-condition: no
fan-running: no
ncsiPackageId: 1
eth-interface: eth0
bmc-mac0: 2c:f8:9b:29:7d:de
bmc-mac1: 2c:f8:9b:29:7d:df
cruz-mac: 2c:f8:9b:29:7d:c2
ipAddress: 127.16.3.1
remoteIPAddress: 127.16.3.2
virtual-eth-if: eth0_m3.4043
actions-blocked: yes
capabilities: 0x1
diagnostic-mode: no
UCSM-mode: Standalone
description:
...
FRU Details
FRU_MLOM(ID8)
Board Mfg : Cisco Systems Inc
Board Product : UCSC-MLOM-C40Q-03
Board Serial : FCH2328764C
Board Part Number : 73-17793-06
Board Extra : A03V04
Board Extra : 0000000000
OEM (0xUnknown (0x9)) Record

• The mLOM can appear in the CIMC CLI mLOM adapter scope <show detail> outputs.

C220-WZP2204006C /chassis # scope adapter MLOM
C220-WZP2204006C /chassis/adapter # show detail
PCI Slot MLOM:
Product Name: N/A
Serial Number: FCH22127H6J
Product ID: UCSC-MLOM-C40Q-03
Adapter Hardware Revision: 3
Current FW Version: N/A
VNTAG: N/A
FIP: N/A
LLDP: N/A
PORT CHANNEL: N/A
Configuration Pending:
Cisco IMC Management Enabled: no
VID: N/A
Vendor: N/A
Description:
Bootloader Version: N/A
FW Image 1 Version: N/A
FW Image 1 State: N/A
FW Image 2 Version: N/A
FW Image 2 State: N/A
FW Update Status: N/A
FW Update Error: N/A
FW Update Stage: N/A
FW Update Overall Progress: N/A

Troubleshooting Steps

Between each step, check if the mLOM card can be seen in the scope chassis outputs of <show pci-adapter>.

Note: 'Not-Loaded' can still appear in these outputs due to the server state, rather than any issue in communication with mLOM card.

C220-node# scope chassis
C220-node /chassis # show pci-adapter
Slot Vendor ID Device ID SubVendor ID SubDevice ID Firmware Version Product Name Option ROM Status
---------- ---------- ---------- ------------- ------------- ------------- ------------------------- ----------
MLOM 0x1137 0x0042 0x1137 0x015d 4.3(3b) Cisco UCS VIC 1387 MLOM Not-Loaded <<<<<<<<<<<<
MRAID 0x1000 0x0014 0x1137 0x020e 51.10.0-3151 Cisco 12G Modular Raid... Not-Loaded
L 0x8086 0x1563 0x1137 0x01a3 0x80000EC9... Intel X550 LOM Not-Loaded

Step 1. Check Physical Connections

Trace the cables from the two ports on the mLOM to their network switch in order to ensure that they are undamaged. Replace cables and Small Form-factor Pluggable (SFP) if necessary.

- Disconnect and reconnect the SFPs on the Server end and assess for mLOM connectivity..

- Disconnect and reconnect the SFPs on the mLOM card and assess for mLOM connectivity.

Step 2. Confirm That the BIOS Has All Ports Enabled. If They Are Not Enabled, Enable the Ports.

Step 2.1. Access the BIOS on the server by rebooting the server and clicking F2 in order to enter the Aptio Setup Utility.

Step 2.2. In the BIOS menu, navigate to Advanced LOM and PCI options, and ensure that All Onboard LOM Ports are enabled.

Step 2.3. Navigate to PCIe Slots Inventory Details. The Slot mLOM card details must be shown there. For example, Missing Slot mLOM.

Step 3. Reboot the CIMC

A CIMC reboot only reboots the CIMC management plane. The server data plane is not impacted.

Step 3.1. Enter the CIMC command mode, Server# scope cimc.

Step 3.2. Reboot the CIMC using Server /cimc # reboot.

Step 4. Run the Cisco UCS Host Upgrade Utility (HUU) to Reflash the UCS

Step 4.1. Gather their firmware version using the <show brief> command.

Running: showBrief
Fri Dec 29 11:13:29 EST 2023
BMC Version Info
ver: 4.0(4i) <<<<<<<< Firmware Version

Step 4.2. From the Cisco Software Download website, download the appropriate Cisco UCS Host Upgrade Utility for the version of UCS firmware.
https://software.cisco.com/download/home

For example, for a UCS C220 M5 with version 4.0(4i), the correct utility version is ucs-c220m5-huu-4.0.4i.iso.

https://software.cisco.com/download/home/286318809/type/283850974/release/4.0(4i)

Step 4.3. Reflash the UCS firmware using the procedure listed in the 'Updating Firmware Using the Update All Option' section in the 'Cisco Host Upgrade Utility User Guide' for the same model Cisco UCS server.

For example, for a UCS C220 M5 model server, the guide to update the firmware is found at: https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/c/sw/lomug/2-0-x/3_0/b_huu_3_0_1/b_huu_2_0_13_chapter_011.html.

• If the HUU produces a prompt appears indicating “Running and packaged versions are same, skip update.”, during the reflash procedure, the correct firmware is already operating in this UCS server.

• Check if the Cisco HUU shows the presence of the mLOM card. Here is an example of the Cisco HUU utility not showing the mLOM card.

Step 5. Physically Reseat the mLOM Adapter Card

Physically reseat the mLOM adapter card, using the procedure described in the 'Replacing an mLOM Card' in the 'Server Installation and Service Guide' for the same model Cisco UCS server.

For example, for a UCS C220 M5 model server, the guide to reseat the mLOM is found at: https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/c/hw/C220M5/install/C220M5/C220M5_chapter_010.html#task_zz5_h1m_3z.

Conclusion

If physical connections and BIOS ports have been verified, a hard reboot on the CIMC has been performed, the server firmware has been reflashed with the correct firmware, and the mLOM has been physically reseated, but the mLOM card does not appear in the outputs of <show pci-adapter>, a mLOM hardware failure can be concluded. In such a circumstance, replace the mLOM card.

Related Information

• Replacing an mLOM Card - Cisco UCS C220 M5 Server Installation and Service Guide
• Ultra-M Solution Guide with CVIM, Release 6.2.bx
• Technical Support & Documentation - Cisco Systems