Cisco Nexus 5000 Series NX-OS Software Configuration Guide

Understanding Port Channels and VSAN Trunking

Switches in the Cisco Nexus 5000 Series implement VSAN trunking and port channels as follows:

• A SAN port channel enables several physical links to be combined into one aggregated logical link.
• An industry standard E port can link to other vendor switches and is referred to as inter-switch link (ISL), as shown on the left side of Figure 1-1.
• VSAN trunking enables a link transmitting frames in the EISL format to carry traffic for multiple VSAN. When trunking is operational on an E port, that E port becomes a TE port. EISLs connects only between Cisco switches, as shown on the right side of Figure 1-1.

See Chapter 1, “Configuring VSAN Trunking” for information on trunk interfaces.

Figure 1-1 VSAN Trunking Only

You can create a SAN port channel with members that are E ports, as shown on the left side of Figure 1-2. In this configuration, the port channel implements a logical ISL (carrying traffic for one VSAN).

You can create a SAN port channel with members that are TE-ports, as shown on the right side of Figure 1-2. In this configuration, the port channel implements a logical EISL (carrying traffic for multiple VSANs).

Figure 1-2 Port Channels and VSAN Trunking

Understanding Load Balancing

Load-balancing functionality can be provided using the following methods:

• Flow based—All frames between source and destination follow the same links for a given flow. That is, whichever link is selected for the first exchange of the flow is used for all subsequent exchanges.
• Exchange based—The first frame in an exchange is assigned to a link, and then subsequent frames in the exchange follow the same link. However, subsequent exchanges can use a different link. This method provides finer granularity for load balancing while preserving the order of frames for each exchange.

Figure 1-3 illustrates how flow-based load balancing works. When the first frame in a flow is received on an interface for forwarding, link 1 is selected. Each subsequent frame in that flow is sent over the same link. No frame in SID1 and DID1 utilizes link 2.

Figure 1-3 SID1, DID1, and Flow-Based Load Balancing

Figure 1-4 illustrates how exchange-based load balancing works. When the first frame in an exchange is received for forwarding on an interface, link 1 is chosen by a hash algorithm. All remaining frames in that particular exchange are sent on the same link. For exchange 1, no frame uses link 2. For the next exchange, link 2 is chosen by the hash algorithm. Now all frames in exchange 2 use link 2.

Figure 1-4 SID1, DID1, and Exchange-Based Load Balancing

SAN Port Channel Configuration Guidelines

Before configuring a SAN port channel, consider the following guidelines:

• Configure the SAN port channel using Fibre Channel ports from both expansion modules to provide increased availability (if one of the expansion modules failed).
• Ensure that one SAN port channel is not connected to different sets of switches. SAN port channels require point-to-point connections between the same set of switches.

If you misconfigure SAN port channels, you may receive a misconfiguration message. If you receive this message, the port channel’s physical links are disabled because an error has been detected.

If the following requirements are not met, a SAN port channel error is detected:

• Each switch on either side of a SAN port channel must be connected to the same number of interfaces.
• Each interface must be connected to a corresponding interface on the other side (see Figure 1-6 for an example of an invalid configuration).
• Links in a SAN port channel cannot be changed after the port channel is configured. If you change the links after the port channel is configured, be sure to reconnect the links to interfaces within the port channel and reenable the links.

If all three conditions are not met, the faulty link is disabled.

Enter the show interface command for that interface to verify that the SAN port channel is functioning as required.

Creating a SAN Port Channel

To create a SAN port channel, perform this task:

The following example shows SAN port channel creation:

About SAN Port Channel Modes

You can configure each SAN port channel with a channel group mode parameter to determine the port channel protocol behavior for all member ports in this channel group. The possible values for a channel group mode are as follows:

• On (default)—The member ports only operate as part of a SAN port channel or remain inactive. In this mode, the port channel protocol is not initiated. However, if a port channel protocol frame is received from a peer port, the software indicates its nonnegotiable status. Port channels configured in the On mode require you to explicitly enable and disable the port channel member ports at either end if you add or remove ports from the port channel configuration. You must physically verify that the local and remote ports are connected to each other.
• Active—The member ports initiate port channel protocol negotiation with the peer port(s) regardless of the channel group mode of the peer port. If the peer port, while configured in a channel group, does not support the port channel protocol, or responds with a nonnegotiable status, it will default to the On mode behavior. The Active port channel mode allows automatic recovery without explicitly enabling and disabling the port channel member ports at either end.

Table 1-1 compares On and Active modes.

To configure active mode, perform this task:

The following example shows how to configure active mode:

About SAN Port Channel Deletion

When you delete the SAN port channel, the corresponding channel membership is also deleted. All interfaces in the deleted SAN port channel convert to individual physical links. After the SAN port channel is removed, regardless of the mode (active and on) used, the ports at either end are gracefully brought down, indicating that no frames are lost when the interface is going down (see the “Setting the Interface Administrative State” section).

If you delete the SAN port channel for one port, then the individual ports within the deleted SAN port channel retain the compatibility parameter settings (speed, mode, port VSAN, allowed VSAN, and port security). You can explicitly change those settings as required.

• If you use the default On mode to avoid inconsistent states across switches and to maintain consistency across switches, then the ports shut down. You must explicitly enable those ports again.
• If you use the Active mode, then the port channel ports automatically recover from the deletion.

Deleting SAN Port Channels

To delete a SAN port channel, perform this task:

The following example shows SAN port channel deletion:

About Interface Addition to a SAN Port Channel

You can add a physical interface (or a range of interfaces) to an existing SAN port channel. The compatible parameters on the configuration are mapped to the SAN port channel. Adding an interface to a SAN port channel increases the channel size and bandwidth of the SAN port channel.

After the members are added, regardless of the mode (Active and On) used, the ports at either end are gracefully brought down, indicating that no frames are lost when the interface is going down.

Compatibility Check

A compatibility check ensures that the same parameter settings are used in all physical ports in the channel. Otherwise, they cannot become part of a SAN port channel. The compatibility check is performed before a port is added to the SAN port channel.

The check ensures that the following parameters and settings match at both ends of a SAN port channel:

• Capability parameters (type of interface, Fibre Channel at both ends).
• Administrative compatibility parameters (speed, mode, port VSAN, allowed VSAN, and port security).
• Operational parameters (speed and remote switch’s WWN).

A port addition procedure fails if the capability and administrative parameters in the remote switch are incompatible with the capability and administrative parameters in the local switch. If the compatibility check is successful, the interfaces are operational and the corresponding compatibility parameter settings apply to these interfaces.

Suspended and Isolated States

If the operational parameters are incompatible, the compatibility check fails and the interface is placed in a suspended or isolated state based on the configured mode:

• An interface enters the suspended state if the interface is configured in the On mode.
• An interface enters the isolated state if the interface is configured in the Active mode.

See the “Reason Codes” section.

Adding an Interface to a SAN Port Channel

To add an interface to a SAN port channel, perform this task:

The following example adds an interface to a SAN port channel:

Forcing an Interface Addition

You can force the port configuration to be overwritten by the SAN port channel. In this case, the interface is added to a SAN port channel.

• If you use the default On mode to avoid inconsistent states across switches and to maintain consistency across switches, then the ports shut down. You must explicitly enable those ports again.
• If you use the Active mode, then the port channel ports automatically recover from the addition.

Note When SAN port channels are created from within an interface, the force option cannot be used.

After the members are forcefully added, regardless of the mode (Active and On) used, the ports at either end are gracefully brought down, indicating that no frames are lost when the interface is going down (see the “Setting the Interface Administrative State” section).

To force the addition of a port to a SAN port channel, perform this task:

The following example adds an interface to a SAN port channel:

About Interface Deletion from a SAN Port Channel

When a physical interface is deleted from the SAN port channel, the channel membership is automatically updated. If the deleted interface is the last operational interface, then the port channel status is changed to a down state. Deleting an interface from a SAN port channel decreases the channel size and bandwidth of the SAN port channel.

• If you use the default On mode to avoid inconsistent states across switches and to maintain consistency across switches, then the ports shut down. You must explicitly enable those ports again.
• If you use the Active mode, then the port channel ports automatically recover from the deletion.

After the members are deleted, regardless of the mode (Active and On) used, the ports at either end are gracefully brought down, indicating that no frames are lost when the interface is going down.

Deleting an Interface from a SAN Port Channel

To delete a physical interface (or a range of physical interfaces) from a SAN port channel, perform this task:

The following example deletes an interface from a SAN port channel:

About Channel Group Creation

If channel group autocreation is enabled, ISLs can be configured automatically into channel groups without manual intervention. Figure 1-7 shows an example of channel group autocreation.

The first ISL comes up as an individual link. In the example shown in Figure 1-7, this is link A1-B1. When the next link comes up (A2-B2 in the example), the port channel protocol determines if this link is compatible with link A1-B1 and automatically creates channel groups 10 and 20 in the respective switches. Link A3-B3 can join the channel groups (and the port channels) if the respective ports have compatible configurations. Link A4-B4 operates as an individual link, because it is not compatible with the existing member ports in the channel group.

Figure 1-7 Autocreating Channel Groups

The channel group numbers are assigned dynamically (when the channel group is formed).

The channel group number may change across reboots for the same set of port channels depending on the initialization order of the ports.

Table 1-2 identifies the differences between user-configured and auto-configured channel groups.

Autocreation Guidelines

When using the autocreation protocol, follow these guidelines:

• A port is not allowed to be configured as part of a SAN port channel when the autocreation feature is enabled. These two configurations are mutually exclusive.
• Autocreation must be enabled in both the local and peer ports to negotiate a SAN port channel.
• Aggregation occurs in one of two ways:

– A port is aggregated into a compatible autocreated SAN port channel.

– A port is aggregated with another compatible port to form a new SAN port channel.

• Newly created SAN port channels are allocated from the maximum possible port channel number in a decreasing order based on availability. If all port channel numbers are used up, aggregation is not allowed.
• You cannot change the membership or delete an autocreated SAN port channel.
• When you disable autocreation, all member ports are removed from the autocreated SAN port channel.
• Once the last member is removed from an autocreated SAN port channel, the channel is automatically deleted and the number is released for reuse.
• An autocreated SAN port channel is not persistent through a reboot. An autocreated SAN port channel can be manually configured to appear the same as a persistent SAN port channel. Once the SAN port channel is made persistent, the autocreation feature is disabled in all member ports.
• You can enable or disable the autocreation feature on a per-port basis or for all ports in the switch. When this configuration is enabled, the channel group mode is assumed to be active. The default for this task is disabled.
• If autocreation of channel groups is enabled for an interface, you must first disable autocreation before downgrading to earlier software versions or before configuring the interface in a manually configured channel group.

Tip When enabling autocreation in any switch in the Cisco Nexus 5000 Series, we recommend that you retain at least one interconnected port between the switches without any autocreation configuration. If all ports between two switches are configured with the autocreation feature at the same time, a possible traffic disruption may occur between these two switches as ports are automatically disabled and reenabled when they are added to an autocreated SAN port channel.

Enabling and Configuring Autocreation

To configure automatic channel groups, perform this task:

The following example configures an automatic channel group:

About Manually Configured Channel Groups

A user-configured channel group cannot be converted to an autocreated channel group. However, you can convert an autocreated channel group to a manual channel group. This task is irreversible. The channel group number does not change, but the member ports operate according to the properties of the manually configured channel group, and channel group autocreation is implicitly disabled for all the member ports.

Tip If you enable persistence, be sure to enable it at both ends of the SAN port channel.

Converting to Manually Configured Channel Groups

You can convert autocreated channel group to a user-configured channel group using the san-port-channel channel-group-number persistent EXEC command. If the SAN port channel does not exist, this command is not executed.