Connecting the Fabric Interconnect

This chapter contains the following topics:

Overview of Network Connections

After you install the UCS X-Series Direct in a rack and power it up, you are ready to make the following network connections:

  • Console connection—This is a direct local management connection that you use to initially configure the fabric interconnect. You must make this physical connection using RS-232 serial console cable with RJ-45 connector first, for initial configuration of the fabric interconnects.

  • Management connection—After you complete the initial configuration using the console, you can use this connection to manage UCS X-Series Direct either through UCS Manager or Cisco Intersight.

  • Uplink interface connections—These connections are for upstream network connectivity.

Each of these connection types is explained in one of the sections that follow.


Note

When running cables in overhead or subfloor cable trays, we strongly recommend that you locate power cables and other potential noise sources as far away as practical from network cabling that terminates on Cisco equipment. In situations where long parallel cable runs cannot be separated by at least 3.3 feet (1 meter), we recommend that you shield any potential noise sources by housing them in a grounded metallic conduit.

Fabric Interconnect Port Configuration

Port Types

The fabric interconnect has the following ports which shall be configured through supported Cisco management platforms:

  • Uplink port: Also called as border-port. An uplink port is an Ethernet port connecting to a northbound LAN aggregation switch.

  • FC Uplink port: A FC port that is connected to an uplink FC fabric. For example, an FC interface of the MDS switch.

  • FCoE Uplink port: An FCoE port that is connected to an uplink FCoE fabric. For example, an FCoE interface on a Cisco MDS or Cisco Nexus switch.

  • Appliance port: An Ethernet server-port that is directly connected to a storage appliance. This configuration requires FI to be in Ethernet-End-Host-Mode.

  • Fibre Channel Storage Port: A port that can be used for directly connecting to a fibre-channel storage array. For example, a NetApp storage© or Pure Storage© array connected directly over an FC port.

Port Configuration

The following table shows the port types that can be configured. For port type definitions, see the preceding section.

Chassis Port

Port Speed

Supported Port Type

1 and 2 (unified ports)

10G/25G/40G/100 G bps

or

8G/16G/32 Gbps

Ethernet

or

Fibre Channel

  • Ethernet Uplink Port

  • FCoE uplink port

  • Appliance port

  • FC uplink port

  • FC storage port

3 through 6

10/25/40/100 Gbps Ethernet

Native Ethernet

  • Ethernet Uplink Ports

  • FCoE uplink ports

  • Appliance ports

7 and 8

1/10/25/40/100 Gbps

Native Ethernet

  • Ethernet Uplink Ports

  • FCoE uplink ports

  • Appliance ports

Port Breakout

Port breakout is supported with the following configurations.

Breakout Level

Ports 1 and 2 (QSFP28)

Ports 3 - through 8 (QSFP28)

4x8G FC

Yes on ports 1 and 2 only (DS-SFP-4x32G-SW transceiver for breakout)

No

4x16G FC

Yes on ports 1 and 2 only (DS-SFP-4x32G-SW transceiver for breakout)

No

4x32G FC

Yes on ports 1 and 2 only (DS-SFP-4x32G-SW transceiver for breakout)

No

1x100G

Yes

Yes

4x10G

Yes

Yes

4x25G

Yes

Yes

1x40G

Yes

Yes

1x25G (Cisco QSA28 with SFP28)

Yes (QSA28)

Yes (QSA28)
1x10G QSA or QSA28 with SFP+ Yes (QSA28)

Yes (QSA28)

1G (CVR-QSFP-SFP10G+GLC-TE)

 No

Yes, ports seven and eight only.

Example Ethernet Topologies

The following sections show supported end-to-end Ethernet topologies.

With VPC (Virtual Port Channel) or MCT (Multi Channel Trunking)

Refer to the following recommended topology to guide you while connecting the fabric interconnect for end-to-end Ethernet. This topology is recommended with either Cisco Nexus 9000 Series switches in VPC or any standard top of rack (ToR) Ethernet switch in MCT (multi-chassis trunking) mode.

In this topology:

  • The Ethernet uplink ports on the fabric interconnect can be used for Ethernet uplinks to the redundant ToR (top of rack) Nexus switches.

  • Port channels connect from the fabric interconnect to the ToR Nexus switches in VPC mode, or to a different ToR Ethernet switch in a multi-channel trunk.

  • ToR switch ports should be STP (spanning-tree) edge ports.

  • This topology:

    • Provides redundancy for the fabric interconnect, the ToR switches, and uplinks.

    • Provides bandwidth aggregation to the fabric interconnect through port channels

    • Avoids ToR-to-ToR L2 switching.

  • The ToR fabric could be 3-tier, EVPN-VxLAN, an IP Fabric, or a CLOS leaf and spine topology.

No VPC (Virtual Port Channel) or MCT (Multi Channel Trunking)

Refer the following recommended topology to guide you while connecting the fabric interconnect for end-to-end Ethernet connectivity.

  • This topology is recommended if your deployment does not use VPC or MCT.

  • This topology is supported, but not recommended, if your deployment uses VPC or MCT. Instead, the previous topology is recommended for deployments that use VPC or MCT.

In this topology:

  • The Ethernet uplink ports on the fabric interconnect can be used for Ethernet uplinks to the redundant ToR (top of rack) Nexus switches.

  • Port channels connect from the fabric interconnect to the ToR Nexus switches in VPC mode, or to different ToR Ethernet switches in a multi-channel trunk.

  • In this topology:

    • vNICs are pinned to one uplink, and distributed across multiple uplinks.

    • ToR switches and uplinks are redundant, but vNICs will experience repinning time after a failover

No Top of Rack Switch Redundancy

Refer to the following supported topology to guide you while connecting the fabric interconnect for end-to-end Ethernet connectivity.

This topology conserves the number of overall links and ports, which can be used for additional server connectivity.

This topology is supported, but not recommended, because of the lack of redundancy for the ToR Ethernet switches.

Example Fibre Channel SAN Topologies

The following sections show supported end-to-end Fibre Channel SAN (FC SAN) topologies.

End Host Mode 1, With Port Channel Configured

Refer to the following recommended topology to guide you while connecting the fabric interconnect for end-to-end connectivity to FC storage. This topology is recommended with either Cisco MDS switches or any standard top of rack (ToR) FC switch.

This topology is recommended for a larger FC SAN domain. In this topology:

  • The fabric interconnect is in FC end-host mode. In this mode, the fabric interconnect uplink ports operate as a node ports (N-Port) while ports on the Cisco MDS switch operate as fabric ports (F-Port).

  • Port Channel configuration is between an FI and MDS pair per side.

  • Port-Channel configuration provides high availability (HA) and uplink bandwidth aggregation (BW).

  • Virtual SANs (vSANs) are carried into the MDS fabric with vSAN trunking.

  • Four virtual HBAs (vHBAs) are configured per server for high availability.

Switch Mode 1, With Port Channel Configuration

Refer to the following recommended topology to guide you while connecting the fabric interconnect for end-to-end connectivity to FC storage. This topology is supported with either Cisco MDS or any standard FC switch.

This topology is supported for more moderate FC SAN domains limited to a maximum of 255 FC devices. In this topology:

  • Ports on both the FC Switch and the fabric interconnects operate in E-port mode.

  • Port Channel configuration is between an FI and MDS pair per side.

  • Port Channel configuration provides high availability (HA) and uplink bandwidth aggregation (BW).

  • Virtual SANs (vSANs) are carried into the MDS fabric with vSAN trunking.

  • Four virtual HBAs (vHBAs) are configured per server for high availability.

  • This topology supports storage array connections directly to the fabric interconnects with upstream SAN fabric connectivity.

End-Host Mode 2, No Port Channels

Refer to the following supported topology to guide you while connecting the fabric interconnect for end-to-end connectivity to FC storage. This topology is not recommended if your deployment does not use Cisco MDS switches.

This topology is supported for any standard deployment that uses third-party TOR FC switches. In this topology:

  • The fabric interconnects are configured in FC end-host mode.

  • No port-channel configuration with non-MDS FC switches.

  • A virtual SAN (vs An) is not supported due to the absence of a Cisco MDS switch.

  • Four virtual HBAs (vHBAs) are configured per server for high availability.

Switch Mode, FC Storage Array Direct Connected to Fabric Interconnects

Refer to the following recommended topology to guide you while connecting the fabric interconnect for end-to-end connectivity to FC storage. This topology is supported for directly connecting an FC storage array to the fabric interconnects.

In this topology:

  • The fabric interconnect is configured in FC switch mode.

  • Four virtual HBAs (vHBAs) are configured per server for high availability.

  • FC SAN zones and zone management is supported through Cisco Intersight Managed Mode (IMM) or Cisco UCS Managed Mode (UMM).

Example IP SAN Topologies

The following sections show supported end-to-end IP SAN topologies.

With VPC (Virtual Port Channel) or MCT (Multi Chassis Trunking)

Refer to the following supported topology to guide you while connecting the fabric interconnect for end-to-end Ethernet-based IP SAN connectivity. This topology is supported for deployments that use Cisco Nexus TOR switches.

In this topology:

  • Virtual port channeling (VPC) or multi-chassis trunking (MCT) is recommended.

  • MTU setting should be set to 9216 on the fabric interconnect system QoS.

  • TCP-based storage access can be set to best-effort QOS class. For high performance storage access, no-drop class QoS settings can also be used across fabric interconnects and the TOR switch.

  • For ROCEv2, the no-drop QoS class is required, and PFC must be enabled on the ToR Nexus switch.

  • Multiple vNICs must be configured on each server for redundancy.


    Caution

    Connectivity best practices from the Cisco Nexus switch to vendor IP storage are different for each storage vendor.

Direct Connection to IP Storage Array

Refer to the following recommended topology to guide you for connecting Ethernet-based IP-SAN storage array directly to the fabric interconnects.

This topology is supported for smaller IP SANs. In this topology:

  • MTU setting should be set to 9216 on the fabric interconnect system QoS.

  • Ethernet based IP-SAN storage access is supported without the need of TOR switches.

  • Port-channel configuration between the fabric interconnects and IP-SAN storage array is supported.

  • No support exists for port-channel configuration with VPC.

Connecting a Console to the Fabric Interconnect

Before you create a network management connection for the fabric interconnect or connect the fabric interconnect to the network, you must create a local management connection through a console terminal. And then configure an IP address for the fabric interconnect. You can use the console to perform the following functions, each of which can be performed through the management interface after you make that connection:

  • Configure the fabric interconnect using the command-line interface (CLI).

  • Monitor network statistics and errors.

  • Configure Simple Network Management Protocol (SNMP) agent parameters.

  • Download software updates.

You make this local management connection between the asynchronous serial port on a fabric interconnect module and a console device capable of asynchronous transmission. Typically, you can use a computer terminal as the console device. On the supervisor modules, you use the console serial port.


Note

Before you can connect the console port to a computer terminal, make sure that the computer terminal supports VT100 terminal emulation. The terminal emulation software makes communication between the fabric interconnect and computer possible during setup and configuration.

Before you begin

  • The Cisco UCS X-Series Direct 9108 100G Fabric Interconnect must be fully installed in the Cisco UCS X9508 chassis, which is connected to a power source, and grounded.

  • The necessary cabling for the console, management, and network connections must be available.

    • An RJ-45 rollover cable provided in the fabric interconnect accessory kit.

    • Network cabling is routed to the location of the installed fabric interconnect.

Procedure

Step 1

Configure the console device to match the following default port characteristics:

  • 115200 baud

  • 8 data bits

  • 1 stop bit

  • No parity

Step 2

Connect an RJ-45 rollover cable to the console port on the fabric interconnect.

You can find this cable in the accessory kit.

Step 3

Route the RJ-45 rollover cable to the console or modem.

Step 4

Connect the other end of the RJ-45 rollover cable to the console or to a modem.