Troubleshoot IMM Network on UCS Domain with API Explorer and NXOS

Introduction

This document describes the analysis of network connectivity or life of the packet for a Unified Computing System (UCS) Domain in Intersight Managed Mode and identifies the internal connection for servers with the API Explorer and NXOS commands.

Contributed by Luis Uribe, Cisco TAC engineer.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Intersight
• Physical Network Connectivity
• Application Programming Interface (API)

Components used

The information in this document is based on these software and hardware versions:

• Cisco UCS 6454 Fabric Interconnect, firmware 4.2(1e)
• UCSB-B200-M5 blade server, firmware 4.2(1a)
• Intersight software as a service (SaaS)

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

The connection between Fabric interconnects and Virtual Network Interface (vNICs) is established through virtual circuits, called Virtual Interface (VIF). Such VIF's are pinned to uplinks and allow communication with the upstream network

In Intersight Managed Mode there is no command that maps the virtual interfaces with each server such as show service-profile circuit. API Explorer/NXOS commands can be used in order to determine the relationship of the internal circuits created within the UCS domain.

API Explorer

API explorer is available from the Graphical User Interface (GUI) of either of the fabric interconnects (primary or subordinate). Once logged into the console, navigate to Inventory, select the server and then click on Launch API Explorer.

The API Explorer contains an API Reference, which lists the available calls. It also includes a representational state transfer (REST) Client interface to test API calls.

Identify VIF through API calls

You can use a set of API calls to determine which VIF corresponds to each virtual vNIC. This allows you to troubleshoot NXOS more effectively.

For the purpose of this document, navigation with API calls is done through these items: Chassis, Server, Network Adapter, vNIC/vHBA.

Retrieve Chassis ID

Copy the Chassis ID for the API call.

/redfish/v1/Chassis/FLM2402001F

Retrieve the network adapter ID

Copy the Network ID for the next API call.

/redfish/v1/Chassis/FLM2402001F/NetworkAdapters/UCSB-MLOM-40G-04_FCH23527C67

Retrieve vNIC ID

Copy the Network adapter(s) ID.

/redfish/v1/Chassis/FLM2402001F/NetworkAdapters/UCSB-MLOM-40G-04_FCH23527C67/NetworkDeviceFunctions/Vnic-A
/redfish/v1/Chassis/FLM2402001F/NetworkAdapters/UCSB-MLOM-40G-04_FCH23527C67/NetworkDeviceFunctions/Vnic-B

Retrieve the VIF ID of the corresponding vNIC

In this case, vNIC-A is mapped to VIF 800. From here, NXOS commands contain this virtual interface.

Identifying VIF with NXOS and Grep Filters

If API Explorer is not available, or you have no access to the GUI, CLI commands can be used to retrieve VIF information.

Note: You must know the Server Profile in order to use these commands.

UCS-TS-MXC-P25-6454-IMM-A(nx-os)# show run interface | grep prev 1 IMM-Server-1
switchport trunk allowed vsan 1
switchport description SP IMM-Server-1, vHBA vhba-a, Blade:FLM2402001F
--
interface Vethernet800
description SP IMM-Server-1, vNIC Vnic-A, Blade:FLM2402001F
--
interface Vethernet803
description SP IMM-Server-1, vNIC Vnic-b, Blade:FLM2402001F
--
interface Vethernet804
description SP IMM-Server-1, vHBA vhba-a, Blade:FLM2402001F

NXOS Troubleshoot

Once the vNIC has been mapped to the correspondent Vethernet, analysis can be done on NXOS with the same commands used to troubleshoot physical interfaces.

Notation for vNICs is veth - Vethernet.

show interface brief shows Veth800 in downstate with ENM Source Pin Failure as the reason.

UCS-TS-MXC-P25-6454-IMM-A# connect nxos

UCS-TS-MXC-P25-6454-IMM-A(nx-os)# show interface brief | grep -i Veth800
Veth800 1 virt trunk down ENM Source Pin Fail auto

show interface shows Vethernet 800 is in an initializing state.

UCS-TS-MXC-P25-6454-IMM-A(nx-os)# show interface Vethernet 800
Vethernet800 is down (initializing)
    Port description is SP IMM-Server-1, vNIC Vnic-A, Blade:FLM2402001F
    Hardware is Virtual, address is 0000.abcd.dcba
    Port mode is trunk
    Speed is auto-speed
    Duplex mode is auto
  300 seconds input rate 0 bits/sec, 0 packets/sec
  300 seconds output rate 0 bits/sec, 0 packets/sec
  Rx
    0 unicast packets  0 multicast packets  0 broadcast packets
    0 input packets  0 bytes
    0 input packet drops
  Tx
    0 unicast packets  0 multicast packets  0 broadcast packets
    0 output packets  0 bytes
    0 flood packets
    0 output packet drops


UCS-TS-MXC-P25-6454-IMM-A(nx-os)# show running-config interface Vethernet 800

!Command: show running-config interface Vethernet800
!Running configuration last done at: Mon Sep 27 16:03:46 2021
!Time: Tue Sep 28 14:35:22 2021

version 9.3(5)I42(1e) Bios:version 05.42 

interface Vethernet800
  description SP IMM-Server-1, vNIC Vnic-A, Blade:FLM2402001F
  no lldp transmit
  no lldp receive
  no pinning server sticky
  pinning server pinning-failure link-down
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 1,470
  hardware vethernet mac filtering per-vlan
  bind interface port-channel1280 channel 800
  service-policy type qos input default-IMM-QOS
  no shutdown

A VIF needs to be pinned to an uplink interface, in this scenario show pinning border interface does not display the Vethernet pinned to any uplink.

UCS-TS-MXC-P25-6454-IMM-A(nx-os)# show pinning border-interfaces 

--------------------+---------+----------------------------------------
Border Interface     Status    SIFs                                    
--------------------+---------+----------------------------------------
Eth1/45              Active    sup-eth1         
Eth1/46              Active    Eth1/1/33  

This indicates that uplinks require additional configuration. This output corresponds to the show running configuration of Ethernet Uplink 1/46.

UCS-TS-MXC-P25-6454-IMM-B(nx-os)# show running-config interface ethernet 1/45

!Command: show running-config interface Ethernet1/45
!No configuration change since last restart
!Time: Wed Sep 29 05:15:21 2021

version 9.3(5)I42(1e) Bios:version 05.42 

interface Ethernet1/45
  description Uplink
  pinning border
  switchport mode trunk
  switchport trunk allowed vlan 69,470
  no shutdown

 show mac address-table details that Veth800 uses VLAN 1 which is not present on the uplinks.

UCS-TS-MXC-P25-6454-IMM-A(nx-os)# show mac address-table 
Legend: 
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
*    1     0025.b501.0036   static   -         F      F    Veth800

On a UCS Domain, the VLAN in use must be included on the vNIC and the uplinks as well. VLAN policy configures the VLANs on the fabric interconnects. The image shows the configuration of this UCS domain.

VLAN 1 is not present on the policy so it must be added. 

Select Edit Policy in order to allow connectivity. This change requires the deployment of the UCS Domain Profile.

VLAN assignment can be verified by CLI:

UCS-TS-MXC-P25-6454-IMM-A(nx-os)# show running-config interface ethernet 1/45

!Command: show running-config interface Ethernet1/45
!Running configuration last done at: Wed Sep 29 07:50:43 2021
!Time: Wed Sep 29 07:59:31 2021

version 9.3(5)I42(1e) Bios:version 05.42 

interface Ethernet1/45
  description Uplink
  pinning border
  switchport mode trunk
  switchport trunk allowed vlan 1,69,470
  udld disable
  no shutdown

UCS-TS-MXC-P25-6454-IMM-A(nx-os)# 

Now that the necessary VLAN(s) is added, the same set of commands can be used in order to verify connectivity on Vethernet800:

UCS-TS-MXC-P25-6454-IMM-A(nx-os)# show interface brief | grep -i Veth800
Veth800       1      virt trunk  up      none                     auto 


UCS-TS-MXC-P25-6454-IMM-A(nx-os)# show interface Vethernet 800
Vethernet800 is up
    Port description is SP IMM-Server-1, vNIC Vnic-A, Blade:FLM2402001F
    Hardware is Virtual, address is 0000.abcd.dcba
    Port mode is trunk
    Speed is auto-speed
    Duplex mode is auto
  300 seconds input rate 0 bits/sec, 0 packets/sec
  300 seconds output rate 0 bits/sec, 0 packets/sec
  Rx
    0 unicast packets  1 multicast packets  6 broadcast packets
    7 input packets  438 bytes
    0 input packet drops
  Tx
    0 unicast packets  25123 multicast packets  137089 broadcast packets
    162212 output packets  11013203 bytes
    0 flood packets
    0 output packet drops




UCS-TS-MXC-P25-6454-IMM-A(nx-os)# show running-config interface Vethernet 800

!Command: show running-config interface Vethernet800
!Running configuration last done at: Wed Sep 29 07:50:43 2021
!Time: Wed Sep 29 07:55:51 2021

version 9.3(5)I42(1e) Bios:version 05.42 

interface Vethernet800
  description SP IMM-Server-1, vNIC Vnic-A, Blade:FLM2402001F
  no lldp transmit
  no lldp receive
  no pinning server sticky
  pinning server pinning-failure link-down
  switchport mode trunk
  switchport trunk allowed vlan 1,69,470
  hardware vethernet mac filtering per-vlan
  bind interface port-channel1280 channel 800
  service-policy type qos input default-IMM-QOS
  no shutdown

 

Veth800 is listed on the pinned interfaces to the uplink Ethernet interfaces:

UCS-TS-MXC-P25-6454-IMM-A(nx-os)# show pinning border-interfaces 

--------------------+---------+----------------------------------------
Border Interface     Status    SIFs                                    
--------------------+---------+----------------------------------------
Eth1/45              Active    sup-eth1 Veth800 Veth803          
Eth1/46              Active    Eth1/1/33                         

Total Interfaces : 2 

UCS-TS-MXC-P25-6454-IMM-A(nx-os)# 

VIFs are now ready to transmit traffic to the upstream network.

Related Information

• Domain Profiles in Intersight
• Server Profiles in Intersight
• Domain Policies in Intersight
• Technical Support & Documentation - Cisco Systems