Configuring Fibre Channel Interfaces

This chapter describes interface configuration for Fibre Channel interfaces and virtual Fibre Channel interfaces. This chapter includes the following sections:

•Information About Fibre Channel Interfaces

•Configuring Fibre Channel Interfaces

•Verifying Fibre Channel Interfaces

•Default Settings

Information About Fibre Channel Interfaces

This section describes Fibre Channel interfaces and virtual Fibre Channel interfaces. This section includes the following topics:

•Licensing Requirements

•Physical Fibre Channel Interfaces

•Virtual Fibre Channel Interfaces

•Interface Modes

•Interface States

•Buffer-to-Buffer Credits

Licensing Requirements

On Cisco Nexus 5000 Series switches, Fibre Channel capability is included in the Storage Protocol Services License.

Ensure that you have the correct license installed (N5010SS or N5020SS) before using Fibre Channel interfaces and capabilities.

Physical Fibre Channel Interfaces

Cisco Nexus 5000 Series switches provide up to eight physical Fibre Channel uplinks. The Fibre Channel interfaces are supported on optional expansion modules. The Fibre Channel plus Ethernet expansion module contains four Fibre Channel interfaces.

Each Fibre Channel port can be used as a downlink (connected to a server) or as an uplink (connected to the data center SAN network). The Fibre Channel interfaces support the following modes: F, NP, E, TE, and SD.

Virtual Fibre Channel Interfaces

Fibre Channel over Ethernet (FCoE) encapsulation allows a physical Ethernet cable to simultaneously carry Fibre Channel and classic Ethernet (CE) traffic. In the Cisco Nexus 5000 Series switches, an FCoE-capable physical Ethernet interface can carry traffic for one logical CE interface and one logical Fibre Channel interface. The logical interfaces are configured in the Cisco Nexus 5000 Series switch as virtual interfaces. A virtual Fibre Channel interface represents the logical Fibre Channel interface.

A virtual Fibre Channel is configured as a subinterface of a virtual interface group (VIG).

Virtual Fibre Channel interfaces support only F mode, and offer a subset of the features that are supported on physical Fibre Channel interfaces.

The following capabilities are not supported for virtual Fibre Channel interfaces:

•SAN port channels.

•VSAN trunking. The virtual Fibre Channel is associated with one VSAN.

•The SPAN destination cannot be a virtual Fibre Channel interface.

•Buffer-to-buffer credits.

•Exchange link parameters (ELP), or Fabric Shortest Path First (FSPF) protocol.

•Configuration of physical attributes (speed, rate, mode, transmitter information, MTU size).

•Port tracking.

Interface Modes

Each physical Fibre Channel interface in a switch may operate in one of several port modes: E mode, TE mode, F mode, and SD mode (see Figure 10-1). A physical Fibre Channel interface can be configured as an E port, an F port, or an SD port. Interfaces may also be configured in Auto mode; the port type is determined during interface initialization.

In NPV mode, Fibre Channel interfaces may operate in NP mode, F mode or SD mode. For additional information about NPV mode, see Chapter 12, "Configuring N-Port Virtualization."

Virtual Fibre Channel interfaces can only be configured in F mode.

Figure 10-1 Switch Port Modes

Note Interfaces are automatically assigned VSAN 1 by default. See Chapter 15, "Configuring and Managing VSANs."

Each interface has an associated administrative configuration and an operational status:

•The administrative configuration does not change unless you modify it. This configuration has various attributes that you can configure in administrative mode.

•The operational status represents the current status of a specified attribute such as the interface speed. This status cannot be changed and is read-only. Some values may not be valid when the interface is down (for example, the operational speed).

The following sections provide a brief description of each interface mode:

•E Port

•F Port

•NP Port

•TE Port

•SD Port

•Auto Mode

E Port

In expansion port (E port) mode, an interface functions as a fabric expansion port. This port may be connected to another E port to create an Inter-Switch Link (ISL) between two switches. E ports carry frames between switches for configuration and fabric management. They serve as a conduit between switches for frames destined to remote N ports. E ports support class 3 and class F service.

An E port connected to another switch may also be configured to form a SAN port channel (see Chapter 14, "Configuring SAN Port Channels").

F Port

In fabric port (F port) mode, an interface functions as a fabric port. This port may be connected to a peripheral device (host or disk) operating as an N port. An F port can be attached to only one N port. F ports support class 3 service.

NP Port

An NP port is a port on a device that is in NPV mode and connected to the core NPV switch through an F port. NP ports operate like N ports that function as proxies for multiple physical N ports.

For more details about NP ports and NPV, see Chapter 12, "Configuring N-Port Virtualization."

TE Port

In trunking E port (TE port) mode, an interface functions as a trunking expansion port. It may be connected to another TE port to create an extended ISL (EISL) between two switches. TE ports connect to another Cisco Nexus 5000 Series switch or a Cisco MDS 9000 Family switch. They expand the functionality of E ports to support the following:

•VSAN trunking

•Fibre Channel trace (fctrace) feature

In TE port mode, all frames are transmitted in EISL frame format, which contains VSAN information. Interconnected switches use the VSAN ID to multiplex traffic from one or more VSANs across the same physical link. This feature is referred to as VSAN trunking in the Cisco Nexus 5000 Series (see Chapter 13, "Configuring VSAN Trunking"). TE ports support class 3 and class F service.

SD Port

In SPAN destination port (SD port) mode, an interface functions as a switched port analyzer (SPAN). The SPAN feature monitors network traffic that passes though a Fibre Channel interface. This monitoring is done using a standard Fibre Channel analyzer (or a similar switch probe) that is attached to an SD port. SD ports do not receive frames, instead they transmit a copy of the source traffic. The SPAN feature is nonintrusive and does not affect switching of network traffic for any SPAN source ports.

Auto Mode

Interfaces configured in auto mode can operate in one of the following modes: F port, E port, or TE port. The port mode is determined during interface initialization. For example, if the interface is connected to a node (host or disk), it operates in F port mode. If the interface is attached to a third-party switch, it operates in E port mode. If the interface is attached to another switch in the Cisco Nexus 5000 Series or Cisco MDS 9000 Family, it may become operational in TE port mode (see Chapter 13, "Configuring VSAN Trunking").

SD ports are not determined during initialization and are administratively configured.

Interface States

The interface state depends on the administrative configuration of the interface and the dynamic state of the physical link. The following sections describe the states and configuration that influence the state:

•Administrative States

•Operational States

•Reason Codes

Administrative States

The administrative state refers to the administrative configuration of the interface. Table 10-1 describes the administrative states.

Table 10-1 Administrative States 

Administrative State
Description

Up

Interface is enabled.

Down

Interface is disabled. If you administratively disable an interface by shutting down that interface, the physical link layer state change is ignored.


Operational States

The operational state indicates the current operational state of the interface. Table 10-2 describes the operational states.

Table 10-2 Operational States 

Operational State
Description

Up

Interface is transmitting or receiving traffic as desired. To be in this state, an interface must be administratively up, the interface link layer state must be up, and the interface initialization must be completed.

Down

Interface cannot transmit or receive (data) traffic.

Trunking

Interface is operational in TE mode.


Reason Codes

Reason codes are dependent on the operational state of the interface. Table 10-3 describes the reason codes for operational states.

Table 10-3 Reason Codes for Interface States 

Administrative Configuration
Operational Status
Reason Code

Up

Up

None.

Down

Down

Administratively down. If you administratively configure an interface as down, you disable the interface. No traffic is received or transmitted.

Up

Down

See Table 10-4.


Note Only some of the reason codes are listed in Table 10-4.

If the administrative state is up and the operational state is down, the reason code differs based on the nonoperational reason code. Table 10-4 describes the reason codes for nonoperational states.

Table 10-4 Reason Codes for Nonoperational States 

Reason Code (long version)
Description
Applicable Modes

Link failure or not connected

The physical layer link is not operational.

All

SFP not present

The small form-factor pluggable (SFP) hardware is not plugged in.

 

Initializing

The physical layer link is operational and the protocol initialization is in progress.

All

Reconfigure fabric in progress

The fabric is currently being reconfigured.

Offline

The switch software waits for the specified R_A_TOV time before retrying initialization.

Inactive

The interface VSAN is deleted or is in a suspended state.

To make the interface operational, assign that port to a configured and active VSAN.

Hardware failure

A hardware failure is detected.

Error disabled

Error conditions require administrative attention. Interfaces may be error-disabled for various reasons. For example:

•Configuration failure.

•Incompatible buffer-to-buffer credit configuration.

To make the interface operational, you must first fix the error conditions causing this state and then administratively shut down or enable the interface.

Isolation because limit of active port channels is exceeded.

The interface is isolated because the switch is already configured with the maximum number of active SAN port channels.

Isolation due to ELP failure

The port negotiation failed.

Only E ports and TE ports

Isolation due to ESC failure

The port negotiation failed.

Isolation due to domain overlap

The Fibre Channel domains (fcdomain) overlap.

Isolation due to domain ID assignment failure

The assigned domain ID is not valid.

Isolation due to the other side of the link E port isolated

The E port at the other end of the link is isolated.

Isolation due to invalid fabric reconfiguration

The port is isolated due to fabric reconfiguration.

Isolation due to domain manager disabled

The fcdomain feature is disabled.

Isolation due to zone merge failure

The zone merge operation failed.

Isolation due to VSAN mismatch

The VSANs at both ends of an ISL are different.

port channel administratively down

The interfaces belonging to the SAN port channel are down.

Only SAN port channel interfaces

Suspended due to incompatible speed

The interfaces belonging to the SAN port channel have incompatible speeds.

Suspended due to incompatible mode

The interfaces belonging to the SAN port channel have incompatible modes.

Suspended due to incompatible remote switch WWN

An improper connection is detected. All interfaces in a SAN port channel must be connected to the same pair of switches.


Buffer-to-Buffer Credits

Buffer-to-buffer credits (BB_credits) are a flow-control mechanism to ensure that Fibre Channel interfaces do not drop frames. BB_credits are negotiated on a per-hop basis.

In Cisco Nexus 5000 Series switches, the BB_credit mechanism is used on Fibre Channel interfaces but not on virtual Fibre Channel interfaces. Virtual Fibre Channel interfaces provide flow control based on capabilities of the underlying physical Ethernet interface.

The receive BB_credit value (fcrxbbcredit) may be configured for each Fibre Channel interface. In most cases, you do not need to modify the default configuration.

Note The receive BB_credit values depend on the port mode, as follows:

•For physical Fibre Channel interfaces, the default value is 16 for F mode and E mode interfaces. This value can be changed as required. The maximum value is 64.

•For virtual Fibre Channel interfaces, BB_credits are not used.

Configuring Fibre Channel Interfaces

This section describes how to configure Fibre Channel interfaces, and includes the following topics:

•Configuring a Fibre Channel Interface

•Setting the Interface Administrative State

•Configuring Interface Modes

•Configuring the Interface Description

•Configuring Administrative Speeds

•Configuring SD Port Frame Encapsulation

•Configuring Receive Data Field Size

•Understanding Bit Error Thresholds

•Configuring Buffer-to-Buffer Credits

Configuring a Fibre Channel Interface

You can configure native Fibre Channel interfaces using Fabric Manager by expanding Switches > Interfaces > FC Physical from the Physical Attributes pane.

Figure 10-2 shows an example of the Information pane for Fibre Channel Interfaces.

Figure 10-2 Native Fibre Channel Interface Configuration

You can configure virtual Fibre Channel interfaces using Fabric Manager by expanding Switches > Interfaces > Ethernet > FCoE > Virtual FC Interfaces from the Physical Attributes pane.

Figure 10-3 shows an example of the Information pane for virtual Fibre Channel Interfaces.

Figure 10-3 Virtual Fibre Channel Interface Configuration

Setting the Interface Administrative State

To disable or enable an interface using Fabric Manager, perform this task:

Step 1 In the Physical Attributes pane, expand Switches > Interfaces, and then choose FC Physical. For a virtual Fibre Channel Interface, expand Switches > Interfaces > Ethernet > FCoE > Virtual FC Interfaces.

You see the interface configuration in the Information pane.

Step 2 Click the General tab.

Step 3 Click Status Admin.

You see a drop-down box with a choice of up or down.

Step 4 Set the status to down (disable) or up (enable).

Step 5 Click Apply Changes.

Configuring Interface Modes

To configure the interface mode using Fabric Manager, perform this task:

Step 1 In the Physical Attributes pane, expand Switches > Interfaces, and then choose FC Physical. For a virtual Fibre Channel Interface, expand Switches > Interfaces > Ethernet > FCoE > Virtual FC Interfaces.

You see the interface configuration in the Information pane.

Step 2 Click the General tab.

Step 3 Click Mode Admin. Choose the desired mode from the pull-down list.

Step 4 Click Apply Changes icon.

Configuring the Interface Description

Interface descriptions should help you identify the traffic or use for that interface. The interface description can be any alphanumeric string.

To configure the interface using Fabric Manager, perform this task:

Step 1 In the Physical Attributes pane, expand Switches > Interfaces, and then choose FC Physical. For a virtual Fibre Channel Interface, expand Switches > Interfaces > Ethernet > FCoE > Virtual FC Interfaces.

You see the interface configuration in the Information pane.

Step 2 Click the General tab.

Step 3 Click Description. Enter the desired text.

Step 4 (Optional) Set additional configuration parameters using the other tabs.

Step 5 Click Apply Changes icon.

Configuring Administrative Speeds

Administrative speed can be configured on a physical Fibre Channel interface (but not on a virtual Fibre Channel interface). By default, the administrative speed for an interface is automatically calculated by the switch.

Caution Changing the administrative speed is a disruptive operation.

To configure administrative speed of the interface using Fabric Manager, perform this task:

Step 1 In the Physical Attributes pane, expand Switches > Interfaces, and then choose FC Physical.

You see the interface configuration in the Information pane.

Step 2 Click the General tab.

Step 3 Click Speed Admin. Set the desired speed from the drop-down list.

The number indicates the speed in megabits per second (Mbps). You can set the speed to 1-Gbps, 2-Gbps, 4-Gbps, or auto (default).

Step 4 Click Apply Changes.

Autosensing

Autosensing speed is enabled on all 4-Gbps interfaces by default. This configuration enables the interfaces to operate at speeds of 1 Gbps, 2 Gbps, or 4 Gbps on the 4-Gbps ports. When autosensing is enabled for an interface operating in dedicated rate mode, 4-Gbps of bandwidth is reserved, even if the port negotiates at an operating speed of 1-Gbps or 2-Gbps.

Configuring SD Port Frame Encapsulation

You can set the frame format to EISL for all frames transmitted by the interface in SD port mode. If you set the frame encapsulation to EISL, all outgoing frames are transmitted in the EISL frame format for all SPAN sources.

See the Cisco Cisco Nexus 5000 Series Family CLI Configuration Guide to configure frame encapsulation on an interface.

Configuring Receive Data Field Size

You can configure the receive data field size for native Fibre Channel interfaces (but not for virtual Fibre Channel interfaces). If the default data field size is 2112 bytes, the frame length will be 2148 bytes.

To configure the receive data field size using Fabric Manager, perform this task:

Step 1 In the Physical Attributes pane, expand Switches > Interfaces, and then choose FC Physical.

You see the interface configuration in the Information pane.

Step 2 Click the Other tab and set the RxDataFieldSize field (see Figure 10-4).

Figure 10-4 Changing Rx Data Size

Step 3 Click Apply Changes.

Understanding Bit Error Thresholds

The bit error rate threshold is used by the switch to detect an increased error rate before performance degradation seriously affects traffic.

The bit errors can occur for the following reasons:

•Faulty or bad cable.

•Faulty or bad GBIC or SFP.

•GBIC or SFP is specified to operate at 1 Gbps but is used at 2 Gbps.

•GBIC or SFP is specified to operate at 2 Gbps but is used at 4 Gbps.

•Short haul cable is used for long haul or long haul cable is used for short haul.

•Momentary synchronization loss.

•Loose cable connection at one or both ends.

•Improper GBIC or SFP connection at one or both ends.

A bit error rate threshold is detected when 15 error bursts occur in a 5-minute period. By default, the switch disables the interface when the threshold is reached. You can reenable the interface.

You can configure the switch to not disable an interface when the threshold is crossed.

See the Cisco Cisco Nexus 5000 Series CLI Configuration Guide to disable the bit error threshold for an interface.

Note The switch generates a syslog message when bit error threshold events are detected, even if the interface is configured not to be disabled by bit-error threshold events.

Configuring Buffer-to-Buffer Credits

The BB_credit scheme is not used for virtual Fibre Channel interfaces. To configure BB_credits for a native Fibre Channel interface using Fabric Manager, perform this task:

Step 1 In the Physical Attributes pane, expand Switches > Interfaces, and then choose FC Physical.

You see the interface configuration in the Information pane.

Step 2 Choose the Bb Credit tab.

You see the buffer credits.

Step 3 Set any of the buffer-to-buffer credits for an interface.

Step 4 Click Apply Changes.

Verifying Fibre Channel Interfaces

The following topics describe the commands for displaying Fibre Channel interfaces:

•Verifying SFP Transmitter Types

•Obtaining Interface Statistics

Verifying SFP Transmitter Types

The SPF transmitter type can be displayed for a physical Fibre Channel interface (but not for a virtual Fibre Channel).

The small form-factor pluggable (SFP) hardware transmitters are identified by their acronyms when displayed.

To display the SFP types for an interface using Fabric Manager, perform this task:

Step 1 In the Physical Attributes pane, expand Switches > Interfaces, and then choose FC Physical.

You see the interface configuration in the Information pane.

Step 2 Click the Physical tab to see the transmitter type for the selected interface.

Obtaining Interface Statistics

You can use Fabric Manager or Device Manager to collect interface statistics on any switch. These statistics are collected at intervals that you can set.

Note In Fabric Manager, you can collect interface statistics by expanding Switches > ISLs and selecting Statistics from the Physical Attributes pane.

To obtain and display interface counters using Device Manager, perform this task:

Step 1 Right-click an interface and choose Monitor in the Interface menu and choose Ethernet Enabled or FC Enabled.

You see the Interface Monitor dialog box.

Step 2 Set both the number of seconds at which you want to poll the interface statistics and how you want the data represented in the Interval drop-down menus. For example, click 10s and LastValue/sec.

Step 3 Click any tab shown in Figure 10-5 to view those related statistics.

Figure 10-5 Device Manager FC Interface Monitor Dialog Box

Step 4 (Optional) Click the Pencil icon to reset the cumulative counters.

Step 5 (Optional) Click the Save icon to save the gathered statistics to a file or click the Print icon to print the statistics.

Step 6 Click Close when you are finished gathering and displaying statistics.

Default Settings

Table 10-5 lists the default settings for native Fibre Channel interface parameters.

Table 10-5 Default Fibre Channel Interface Parameters 

Parameters
Default

Interface mode

Auto

Interface speed

Auto

Administrative state

Shutdown (unless changed during initial setup)

Trunk mode

On (unless changed during initial setup)

Trunk-allowed VSANs

1 to 4093

Interface VSAN

Default VSAN (1)

Beacon mode

Off (disabled)

EISL encapsulation

Disabled

Data field size

2112 bytes


Table 10-5 lists the default settings for virtual Fibre Channel interface parameters.

Table 10-6 Default Virtual Fibre Channel Interface Parameters 

Parameters
Default

Interface mode

Auto

Interface speed

n/a

Administrative state

Shutdown (unless changed during initial setup)

Trunk mode

n/a

Trunk-allowed VSANs

n/a

Interface VSAN

Default VSAN (1)

EISL encapsulation

n/a

Data field size

n/a