Configuring Direct Connect

This chapter includes the following sections:

Direct Connect Mode

Beginning with release 3.1, Cisco UCS Manager supports an option to connect the C-Series Rack-Mount Server directly to the Fabric Interconnects. You do not need the Fabric Extenders. This option enables Cisco UCS Manager to manage the C-Series Rack-Mount Servers using a single cable for both management traffic and data traffic.


Note

After connecting the server with the FI, when the server is discovered, make sure to update the C-Series software bundle available for Cisco UCS Manager, release 3.1 or later.

While using Cisco VIC 1385 and 1387 in a 10G connection you would a require a 40G to 10G QSFP convertor but you cannot use the 4x10 Breakout Small Form-Factor Pluggable (SFP) cables and 4x10 Active Optical (OAC) cables.

When you use direct connect mode, all Cisco UCS Managed adapters must be connected to the server ports on the Fabric Interconnects. Make sure you have server firmware as listed in the Requirements for Direct Connect Mode Integration with Cisco UCS Manager. If not, upgrade your server firmware before integrating the server with Cisco UCS Manager. See Upgrading the C-Series Server Firmware.

VIF Scalability

For the maximum number of virtual interfaces allowed on each adapter in the server that is directly connected to the Fabric Interconnect, see

Cisco UCS Configuration Limits for Cisco UCS Manager Guide

Requirements for Direct Connect Mode Integration with Cisco UCS Manager

Requirements for Cluster Setup

You must have the following items to create a direct connect integration for C-Series Rack-Mount Server with Cisco UCS Manager in cluster setup:

  • A Cisco UCS system that is running Cisco UCS Manager release 3.1 or later.

  • Any of the following supported Cisco UCS C-Series servers and corresponding CIMC release versions listed in the following table:

    Table 1. Minimum Required Version of Cisco IMC, BIOS and Cisco UCS Manager for Direct Connect Mode Integration

    Server

    Cisco IMC

    BIOS

    UCS Manager

    Cisco UCS C220 M4 Server (requires 2.2.3 or above)

    2.0(3d)

    2.0.3

    2.2(3a)

    Cisco UCS C240 M4 Server (requires 2.2.3 or above)

    2.0(3d)

    2.0.3

    2.2(3a)

    Cisco UCS C460 M4 Server

    1.5(7a)

    1.5.7

    2.2(2a)

    Cisco UCS C220 M5 Server

    3.1.1d

    3.1.1i.0

    3.2(1d)

    Cisco UCS C240 M5 Server

    3.1.1d

    3.1.1h.0

    3.2(1d)

    Cisco UCS C480 M5 Server

    3.1(2b)

    3.1.2a.0

    3.2(2b)

    Cisco UCS C125 M5 Server

    4.0(1a)

    4.0(1a)

    4.0(1a)

    Cisco UCS C480 M5 ML Server

    4.0(2a)

    4.0(2a)

    4.0(2a)

    Note

    For Cisco UCS C220 M4 and Cisco UCS C240 M4Cisco UCS C220 M4, C240 M4, C220 M5, and C240 M5 servers you can use either MLOM or VIC to integrate with Cisco UCS Manager in SingleConnect or direct connect mode.

    Cisco UCS C125 M5 Servers do not support MLOM. For Cisco UCS C125 M5 Servers, you must place the VIC card in slot 2 only.

    The adapter for connectivity is selected as explained below:

    • MLOM only: When only MLOM is present in the server, the connectivity adapter is by default MLOM,

    • MLOM and VIC adapters: MLOM is the default connectivity adapter.

    • No MLOM:

      • For Cisco UCS C220 M4/C220 M5, the VIC must be installed in Riser1 - Slot 1 for direct connect integration.

      • For Cisco UCS C240 M4/C240 M5, the VIC can be installed on any one of the following slots: 1, 2, 4, or 5 for direct connect integration.


    Note

    While using a VIC adapter, ensure that you remove MLOM if you want to use a 40 Gigabyte connection for integration.


    Note

    • No MLOM and VIC: No SingleConnect or direct connect support.


    Important

    The server must have Cisco UCS VIC 1225, Cisco VIC 1227, Cisco VIC 1225T , or Cisco VIC 1227T, Cisco VIC 1385, Cisco VIC 1387 MLOM, Cisco VIC 1455, Cisco VIC 1457 MLOM , Cisco VIC 1495, Cisco VIC 1497 MLOM or UCS VIC 15428, or UCS VIC 15238 installed. The Firmware and Boot Loader versions for Cisco UCS VIC 1225 or Cisco VIC 1227 are:

    • Recommended — 2.1(0.457a)

    • Minimum required — 2.1(0.367e)

    For Cisco UCS VIC 1385 and 1387 MLOM:

    • Recommended 4.1.1a

    • Minimum 4.1.1a

    For Cisco UCS VIC 1455 and 1457 MLOM:

    • Recommended 5.0(1a) and higher

    For Cisco UCS VIC 1495 and 1497 MLOM:

    • Recommended 5.0(2a) and higher

    Cisco UCS 1225, Cisco UCS 1225T, Cisco UCS 1227T, Cisco UCS 1227, Cisco UCS 1385, Cisco UCS 1387, Cisco UCS 1455, Cisco UCS 1457, Cisco UCS 1495, and Cisco UCS 1497

    VICs also support direct-connect. If you are using Cisco UCS VIC for direct-connect, make sure the card is inserted in one of the slots as described in the following table:


    Note

    Beginning with Cisco UCS Manager release 4.2, M3 servers are not supported.
    Table 2. Cisco VIC Slot for Direct Connect Integration

    Server

    PCIe Slot

    Cisco UCS C220 M4 Server

    MLOM, 1

    Cisco UCS C240 M4 Server

    MLOM, 1, 2, 4, 5

    Note

     

    If slots 2 and 5 are occupied with GPUs, then VICs can be placed in slots 1 and 4.

    Cisco UCS C460 M4 Server

    4, 9, 5, 10

    Cisco UCS C220 M5 Server

    MLOM, 1

    Cisco UCS C240 M5 Server

    MLOM, 1, 2, 4, 5

    Note

     

    If slots 2 and 5 are occupied with GPUs, then VICs can be placed in slots 1 and 4.

    See Cisco UCS Manager Management for Mixed VIC Set-up in C240 M5 Server for slot combination.

    Cisco UCS C480 M5 Server

    1, 2

    Cisco UCS C125 M5 Server

    2

    Cisco UCS C480 M5 ML Server

    11, 12

  • Cisco UCS 6536 Fabric Interconnect, or Cisco UCS 6454 Fabric Interconnect, Cisco UCS 6200 Series or Cisco UCS 6300 Series FIs. The ports must be configured as server ports.

  • At least two 10-Gb Small Form-Factor Pluggable (SFP) cables for one card.

  • 25G cable and 40G SFP cable to connect the 25G and 40G adapters.

  • Cisco UCS M6 Rack server is supported only with Cisco UCS 6300 or 6400 series Fabric Interconnet, and is not supported with Cisco UCS 6200 series Fabric Interconnect.

Requirement for Non-cluster Setup

If you are integrating the server in a non-cluster setup, along with the Cisco UCS Manager and C-series rack mount server, you need the following components :

  • One Cisco UCS 6536 Fabric Interconnect, or Cisco UCS 6454 Fabric Interconnect, Cisco UCS 6200 Series or 6300 Series Fabric Interconnects. The ports must be configured as server ports.

  • At least one 10-Gb SFP cable for each card. You cannot connect the card to the same FI from both the ports.

For more information on server, Cisco VIC and FEX/IOM compatibility, refer Cisco UCS Fabric Interconnect Server Compatibility Matrix - Release 4.2(3) section in Release Notes for Cisco UCS Manager, Release 4.2.

Cisco UCS Manager Management for Mixed VIC Set-up in C240 M5 Server

Cisco UCS Manager does not provide full support in a setup with Cisco UCS C240 M5 Server and mixed VIC cards.

Table 3. Cisco UCS Manager Management

Slots

Cisco UCS Manager Management

Both slot 1 and slot 2 are occupied.

Slot 1 - Cisco VIC 1455

Slot 2 - Any Cisco VIC card or third-party adapter

Cisco UCS Manager detects only slot 2.

Both slot 1 and slot 2 are occupied.

Slot 1 - Any Cisco VIC card or third-party adapter except Cisco VIC 1455

Slot 2 - Cisco VIC 1455

Cisco UCS Manager detects both the slots.

Both slot 1 and slot 2 are occupied.

Slot 1 - Any Cisco VIC card or third-party adapter except Cisco VIC 1455

Slot 2 - Any Cisco VIC card or third-party adapter

Cisco UCS Manager detects both the slots.

Only one slot is occupied.

Slot 1 - Any Cisco VIC card or third-party adapter

or

Slot 2 - Any Cisco VIC card or third-party adapter

Cisco UCS Manager detects whichever slot is occupied.

Both slot 4 and slot 5 are occupied.

Slot 4 - Cisco VIC 1455

Slot 5 - Any Cisco VIC card or third-party adapter

Cisco UCS Manager detects only slot 5.

Both slot 4 and slot 5 are occupied.

Slot 4 - Any Cisco VIC card or third-party adapter except Cisco VIC 1455

Slot 5 - Cisco VIC 1455

Cisco UCS Manager detects both the slots.

Both slot 4 and slot 5 are occupied.

Slot 4 - Any Cisco VIC card or third-party adapter except Cisco VIC 1455

Slot 5 - Any Cisco VIC card or third-party adapter

Cisco UCS Manager detects both the slots.

Only one slot is occupied.

Slot 4 - Any Cisco VIC card or third-party adapter

or

Slot 5 - Any Cisco VIC card or third-party adapter

Cisco UCS Manager detects whichever slot is occupied.

Management Connection Policy and Connection Mode

When establishing physical connections for the C-Series Rack-Mount Server in the Cisco UCS Domain, you must specify the connection mode and management policy in Cisco UCS Manager GUI or Cisco UCS Manager CLI. The management policy is global to all connected C-Series Rack-Mount Servers. The connection mode determines the rack server discovery in Cisco UCS Manager.

You can specify either one of the following connection policies for the rack servers:

  • Auto Acknowledged: This is the recommended and default connection mode for the C-Series Rack-Mount Server. If the connection mode is auto-acknowledged, immediately after establishing the physical connection, Cisco UCS Manager discovers the rack server and starts managing the server based on the specified management policy.

  • User Acknowledged: If the connection mode is user-acknowledged, after establishing the physical connection, you must manually acknowledge the connection and specify the connection mode in the Cisco UCS Manager GUI or Cisco UCS Manager CLI to begin the discovery. Cisco UCS Manager does not begin the C-Series Rack-Mount Server discovery until you specify the connection mode. See Manually Acknowledging Connection Mode - GUI or Manually Acknowledging Connection Mode - CLI for detailed procedure.


Note

When rack servers are discovered and are managed by Cisco UCS Manager, if you want to change the existing connection mode for one C-Series Rack-Mount Server, you have to decommission the server, re-commission it and specify the new connection mode. Only then Cisco UCS Manager can discover the server in the new connection mode.

Connecting the C-Series Server with Cisco UCS Domain in Cluster Setup

Before you begin


Important

Make sure the server Cisco IMC is set to factory default settings to integrate with Cisco UCS Manager.

  • Make sure the FI ports are configured as server ports. If they are not, reconfigure the ports as server ports using the Cisco UCS Manager GUI.

  • Before you connect the server, make sure the Cisco UCS VIC is installed in the correct slot for integration with Cisco UCS Manager. If the card is not installed in the correct slot, you cannot enable the direct connect management for the server.

Procedure

Step 1

Install the server in the rack. See the Install and Upgrade Guide for the server that you are using. The install guides are available at the following url: Install and Upgrade Guides.

Step 2

Configure the server ports on the FI.

Step 3

Connect the paths from Cisco UCS VIC adapter to the server ports on the FIs.

  1. Connect one supported SFP cable between the server and one port on FI A. You can use any port on FI A, but the port must be enabled for server traffic.

    Make sure to connect one cable from the VIC to the FI for one card. You cannot connect both ports in the same FI.

  2. Connect one supported SFP cable between the server and one port on FI B. You can use any port on FI B, but the port must be enabled for server traffic.

    Note

     

    • Do not mix SFP types on an uplink. If you do, you will get Discovery Failed errors.

    • For information on the C-Series integration network topology and cable connections, see the Network Topology and Supported Cables section.

Step 4

Attach a power cord to each power supply in your server, and then attach the power cord to a grounded AC power outlet.

Server discovery begins when power is applied.

Physical Connectivity Illustrations for Direct Connect Mode Cluster Setup

The following images shows a sample of direct connect mode physical connectivity for C-Series Rack-Mount Server with Cisco UCS Domain, Cisco UCS Manager, release 3.1 or later. This image shows the cabling configuration for Cisco UCS Manager integration with a C-Series Rack-Mount Server. The paths shown in gold carry both management traffic and data traffic.

The illustrations given are only for example purpose. The equipment setup may vary depending on the server, FI, and FEX.

Figure 1. Direct Connect Cabling Configuration


Cisco IMC on the C-Series Rack-Mount Server uses the first adapter for management and data connectivity and a second adapter only for data connectivity. Every virtual interface on the rack server needs to be connected to both Fabric Interconnect adapters. Different configurations require specific setup. For detailed instructions, use the information provided in the section Physical Connectivity Illustrations for Direct Connect Mode Cluster Setup in the appropriate C-Series Integration Guide.

This image shows the cabling configuration for Cisco UCS Manager integration on C-Series Rack-Mount Server with dual Cisco VIC adapters. The first adapter can be configured as management only and the second adapter can be configured as data traffic only. The gold color path carries management and the red color path carries data traffic.

Different servers require different slot locations for the management. Consult the appropriate C-Series Rack Server spec sheet to locate the slot that has Network Communications Services Interface protocol support (NCSI) and use as the management.

Figure 2. Direct Connect Cabling Configuration for Dual Adapter
Figure 3. Direct Connect Cabling Configuration with Cisco VIC 1455 (4-Port Linking)
Figure 4. Direct Connect Cabling Configuration with Cisco VIC 1455 (2-Port Linking)

Note

Ports 1 and 3 are used because the connections between ports 1 and 2 (also 3 and 4) form an internal port-channel.

Caution

Do not connect port 1 to Fabric Interconnect A and port 2 to Fabric Interconnect B. Use ports 1 and 3 only. Using ports 1 and 2 results in discovery and configuration failures.

XGb represents a 40 Gigabit Ethernet connection or a 10 Gigabit Ethernet connection. For the 10 Gigabit Ethernet, the following cables are used:

  • 4x10 Breakout Small Form-Factor Pluggable (SFP) cables

  • 4x10 Active Optical (OAC) cables

  • 10G Small Form-Factor Pluggable (SFP) cable that uses the Qualified Security Assessor (QSA) module

1

Cisco UCS 6536 Fabric Interconnect or Cisco UCS 6454 Fabric Interconnect or Cisco UCS 6200 Series or 6300 FI (Fabric A)

3

C-Series Rack-Mount Server

2

Cisco UCS 6536 Fabric Interconnect or Cisco UCS 6454 Fabric Interconnect or Cisco UCS 6200 Series or 6300 FI (Fabric B)

4

Cisco UCS VIC in supported PCIe slot

XGb represents a 40 Gigabit Ethernet connection or a 10 Gigabit Ethernet connection. For the 10 Gigabit Ethernet, the following cables are used:

  • 4x10 Breakout Small Form-Factor Pluggable (SFP) cables

  • 4x10 Active Optical (OAC) cables

  • 10G Small Form-Factor Pluggable (SFP) cable that uses the Qualified Security Assessor (QSA) module

1

or Cisco UCS 6200 Series or 6300 FI (Fabric A)

3

C-Series Rack-Mount Server

2

or Cisco UCS 6200 Series or 6300 FI (Fabric B)

4

Cisco UCS VIC in supported PCIe slot

Connecting the C-Series Server with UCS Domain in Non-Cluster Setup

To connect the C-Series server with a UCS domain in a non-cluster setup, complete the same steps described in Connecting the C-Series Server with Cisco UCS Domain in Cluster Setup. Instead of using two FIs to connect two cables as in cluster setup, you need to connect only one FI and one cable to the adapter.

Figure 5. Physical Connectivity Illustration Example for Direct Connect Mode Non-Cluster Setup

The following image show the cabling configuration for Cisco UCS Manager integration with the server. The path shown in gold carry both management and data traffic.




Note

XGb represents a 40 Gigabit Ethernet connection or a 10 Gigabit Ethernet connection or a 25 Gigabit Ethernet connection. For the 10 Gigabit Ethernet, the following cables are used:

  • 4x10 Breakout Small Form-Factor Pluggable (SFP) cables

  • 4x10 Active Optical (OAC) cables

  • 10G Small Form-Factor Pluggable (SFP) cable that uses the Qualified Security Assessor (QSA) module

1

Cisco UCS 6454 Fabric Interconnect or Cisco UCS 6200 Series or 6300 FI (Fabric A)

2

C-Series Rack-Mount Server

3

Cisco VIC in supported PCIe slot.

Physical Connectivity Illustrations for Direct Connect Mode Non-Cluster Setup

The following images shows a sample of direct connect mode physical connectivity for C-Series Rack-Mount Server with Cisco UCS Domain, Cisco UCS Manager, release 3.1 or later. This image shows the cabling configuration for Cisco UCS Manager integration with a C-Series Rack-Mount Server. The paths shown in gold carry both management traffic and data traffic.

The images given are only for example purpose. The equipment setup may vary depending on the server, FI, and FEX.

The images shown below are with Cisco UCS VIC 1455. The port connections remain same for Cisco UCS VIC 1457, VIC 1467, and VIC 15428.

Figure 6. Direct Connect Cabling Configuration with Cisco VIC 1455/1457 (Single Cable Configuration)
Figure 7. Direct Connect Cabling Configuration with Cisco VIC 1455/1457 (Dual Cable Configuration)

XGb represents a 40 Gigabit Ethernet connection or a 10 Gigabit Ethernet connection. For the 10 Gigabit Ethernet, the following cables are used:

  • 4x10 Breakout Small Form-Factor Pluggable (SFP) cables

  • 4x10 Active Optical (OAC) cables

  • 10G Small Form-Factor Pluggable (SFP) cable that uses the Qualified Security Assessor (QSA) module

Migrating C-Series Servers from SingleConnect to Direct Connect Mode

Before performing the migration procedure, on the Cisco UCS Manager GUI, Global Policies page, make sure the Rack Discovery Policy and Rack Management Connection Policies are set to the defaults such as, Immediate and Auto Acknowledge. This will make sure after connecting the server directly to the FI, the server is discovered immediately and Cisco UCS Manager starts managing the server without waiting for your interaction.


Caution

Migrating from SingleConnect FEX connectivity to direct connect is a disruptive process. Make sure you do this during a scheduled outage window. You might need 15 to 20 minutes to complete this procedure.

Procedure

Step 1

Configure the server ports on the FI.

Step 2

Disconnect the cables from the FEX and connect them to the configured server ports on Fabric Interconnect.

Step 3

On the Cisco UCS Manager GUI, right click the server, from the pop-up menu, select Recover Server or Server Maintenance.

Step 4

Click Re-acknowledge and select OK.

This will cause server reboot.
After the server reboots, you can resume normal operations.

Migrating C-Series Servers from Dual-wire Management to Direct Connect Mode

Before performing the migration procedure, on the Cisco UCS Manager GUI, Global Policies page, make sure the Rack Discovery Policy and Rack Management Connection Policies are set to the defaults such as, Immediate and Auto Acknowledge. This will make sure after connecting the server directly to the FI, the server is discovered immediately and Cisco UCS Manager starts managing the server without waiting for your interaction.


Caution

  • Migrating from Dual-wire FEX connectivity to direct connect is a disruptive process. Make sure you do this during a scheduled outage window. You might need 15 to 20 minutes to complete this procedure.

  • You must decommission the server before performing the migration.

To enable direct connect mode to a server connected in dual-wire management mode, you will need to change the adapter. Changing the adapter may impact the PCIe numbering on a bare metal Windows system. If you had Adapters listed in the OS, after you change them they will be Adapter 3 and 4. If you are using static addressing, you will need to remove the previous adapters from the server. For more information, see http://support.microsoft.com/kb/269155

Procedure

Step 1

Configure the server ports on the FI.

Step 2

Decommission the server.

Step 3

If you are not using Cisco UCS VIC 1225, Cisco UCS VIC 1227, Cisco UCS VIC 1385, or Cisco UCS VIC 1455, change the Cisco UCS VIC physically in the server:

  1. Unplug power cable, management LOMs and SFP twinax connections from the back of the server.

  2. Slide the server out of the rack and remove the top cover.

  3. Remove the PCIe riser card from the slot and remove the existing Adapter and then install Cisco UCS VIC 1225.

  4. Reinstate the PCIe riser card into the slot, place the top cover back on the server and push the server back into the rack.

  5. (Optional)If you would no longer require the FEX to connect any server in dual wire mode, unplug the FEX from Cisco UCS Manager, and power and remove the FEX from the rack server.

Step 4

Connect the SFP twinax cables from the Cisco VIC port to the FI ports.

Step 5

Reconnect the power cable to the server.

CIMC has standby, so you need not power the server.

What to do next

Recommission the server from Cisco UCS Manager GUI. After the recommission is complete, the server is ready for operation and you can apply service profiles to the server.

Managing the Rack-Mount Server in Cisco UCS Manager after Integration

You can manage and monitor all rack-mount servers that have been integrated with a Cisco UCS domain through Cisco UCS Manager. After the integration, you can perform all rack-mount server management tasks only through the service profiles from Cisco UCS Manager GUI or Cisco UCS Manager CLI. The C-Series Rack-Mount Server Cisco IMC is not accessible when you start managing the server with Cisco UCS Manager.

Cisco UCS Manager provides information, errors, and faults for each rack-mount server that was discovered.

For more information on managing C-Series Rack-Mount Servers from Cisco UCS Manager, see the chapter on Managing Rack-Mount Servers in your release specific Cisco UCS Manager Configuration Guide.

Reverting a Server From Cisco UCS Domain Mode to Standalone Mode

When you manage a Cisco UCS C-Series server with Cisco UCS Manager software, a UCS Manager service profile is associated with the server. If you want to revert a C-Series server to standalone mode (so that it can be managed by Cisco IMC software), you must do the following actions in UCS Manager:

Procedure

Step 1

Disassociate the UCS Manager service profile from the server.

Step 2

Remove the connections from the adapter.

Step 3

Manually power on the server.

Step 4

Connect the front panel KVM connection.

Step 5

Press F8 during the boot up.

Step 6

Reset CIMC to factory defaults or stand-alone mode.

Caution

 

If you do not disassociate the service profile from the server, MAC and WWN numbers assigned by UCS Manager might be kept by the server. This can cause numbering duplication and conflicts with other servers that are managed by UCS Manager. Also, if you revert a server to standalone mode without disassociating its service profile, any LSI RAID controller will not show as a bootable device in the standalone server, preventing local booting.

Special Considerations for Integrated Servers

When you connect the C-Series servers in Cisco UCS system, make sure to review the following information:

Single Link Port Channel between VIC 1455, VIC 1457, or VIC 1467 Adapters and Cisco UCS 6300 Fabric Interconnects

Cisco UCS Manager Release 4.0(1a) and later support only single link port channel between VIC 1455, 1457, or 1467 adapters on direct-connect rack-servers to Cisco UCS 6300 Series Fabric Interconnects.

Do not add a second port channel link between VIC 1455, 1457, or 1467 adapters and Cisco UCS 6300 Series Fabric Interconnects.