Configuring Port Channels

This chapter provides information about port channels and how to configure the port channels.

Feature History for Port Channels

Feature Name

Release

Feature Information

Port channels

8.4(1)

The default port channel mode is changed from On to Active mode.

Information About Port Channels

The following sections provide information about Port Channels.

Port Channels Overview

Port channels refer to the aggregation of multiple physical interfaces into one logical interface to provide higher aggregated bandwidth, load balancing, and link redundancy (as shown in the following image). Port channels connect to interfaces across switching modules, so failure of a switching module cannot bring down the port channel link.

Figure 1. Port Channel Flexibility


Port channels on Cisco MDS 9000 Series Multilayer Switches allow flexibility in configuration. This illustrates three possible port channel configurations:

  • Port channel A aggregates two links on two interfaces on the same switching module at each end of a connection.

  • Port channel B aggregates two links, but each link is connected to a different switching module. If the switching module goes down, traffic is not affected.

  • Port channel C aggregates three links. Two links are on the same switching module at each end, while one is connected to a different switching module on switch 2.

E Port Channels

An E port channel is the aggregation of multiple E ports into one logical interface to provide higher aggregated bandwidth, load balancing, and link redundancy. Port channels connect interfaces across switching modules, so a failure of a switching module cannot bring down the port channel link.

Features and restrictions of a port channel:

  • Provides a point-to-point connection over ISL (E ports) or EISL (TE ports). You can combine multiple links into a port channel.

  • Increases the aggregate bandwidth on an ISL by distributing traffic among all functional links in the channel.

  • Load balances across multiple links and maintains optimum bandwidth utilization. There are two types of load balancing

    • Flow-based load balancing which is based on the source ID and destination ID.

    • Exchange-based load balancing which is based on source ID, destination ID, and exchange ID (OX ID).

  • Provides high availability on an ISL. If one link fails, traffic that is previously carried on this link changes over to the remaining links. If a link goes down in a port channel, the upper protocol is not aware of it. To the upper protocol, the link is still there, although the bandwidth is diminished. Link failure of a port channel member does not affect the routing tables and fspf cost of the port channel. Port channels may contain up to 16 physical links and may span multiple modules for added high availability.


Note
See Cisco MDS 9000 Series NX-OS Fabric Configuration Guide for information about Failover scenarios for port channels and FSPF links.

F, TF, NP, and TNP Port Channels


Note

It is not recommended that you use interface, fWWN, or domain-ID based zoning for devices that are connected to the edge Cisco N-Port Virtualization (NPV) switches.

F port channels provide fault tolerance and performance benefits on connections to N-Port Virtualization (NPV) switches, including Cisco UCS Fabric Interconnects (FIs). F port channels present unique challenges to ACL TCAM programming. When F ports are aggregated into a port channel, ACL TCAM programming is repeated on each member interface. Consequently, these types of port channels multiply the amount of TCAM entries needed. Because of this, it is imperative that the member interfaces are allocated as optimally as possible, and that zoning best practices are also followed. Given that F port channels can contain 100+ host logins, TCAM can easily be exceeded, especially for fabric switches if best practices are not followed.

The following is a sample topology:

This example assumes that the port channel (PC) contains 8 interfaces, fc1/1-fc1/8.

In addition, the following two zones are active: 


zone1
member host (host 0x010001)
member target1 (target1 0x010002)
zone2
member host (host 0x010001)
member target2 (target2 0x010003)

In such a scenario, the following ACL programming will be present on each member of the PC: 


fc1/1(through fc1/8) (port-channel)
Entry#    Source ID     Mask         Destination ID         Mask              Action
1         010001       ffffff        010002(target1)       ffffff             Permit
2         010001       ffffff        010003(target2)       ffffff             Permit
3         000000       000000        000000                000000             Drop

The above example shows the ACL TCAM programming that will be duplicated on each member of the F port-channel.

The following are the best practices for efficient use of TCAM with respect to F ports and F port-channels to optimize TCAM usage on a forwarding engine:

  • Distribute port-channel member interfaces into different forwarding engines, especially on fabric switches.

  • If TCAM usage is still too high in the case of port-channel with a large number of interfaces, then split the port-channel into two separate port-channels each with half the interfaces. This provides redundancy but reduces the number of FLOGIs per individual port-channel and thus reduces TCAM usage.

  • Distribute member interfaces into separate linecards on director-class switches.

  • Distribute member interfaces into forwarding engines with lower TCAM zoning region usage.

  • Use single-initiator zones, single-target zones, or Smart Zoning.

Port Channels and Trunking

Trunking is a commonly used storage industry term. However, the Cisco NX-OS software and switches in the Cisco MDS 9000 Series Multilayer Switches implement trunking and port channels as follows:

  • Port channels enable several physical links to be combined into one aggregated logical link.

  • Trunking enables a link transmitting frames in the EISL format to carry (trunk) multiple VSAN traffic. For example, when trunking is operational on an E port, that E port becomes a TE port. A TE port is specific to switches in the Cisco MDS 9000 Series Multilayer Switches. An industry standard E port can link to other vendor switches and is referred to as a non-trunking interface (see Trunking Only and Port Channeling and Trunking). See Configuring Trunking for information on trunked interfaces.

Figure 2. Trunking Only
Figure 3. Port Channeling and Trunking

Port channels and trunking are used separately across an ISL.

  • Port channels—Interfaces can be channeled between the following sets of ports:

    • E ports and TE ports

    • F ports and NP ports

    • TF ports and TNP ports

  • Trunking—Trunking permits carrying traffic on multiple VSANs between switches.

    See Cisco MDS 9000 Series NX-OS Fabric Configuration Guide.

  • Both port channeling and trunking can be used between TE ports over EISLs.

Port Channel Modes


Note

After changing the port-channel mode, each member interface must be brought down and brought back up via the shutdown and no shutdown commands for the port-channel mode to be changed. This can be done on an individual member-by-member basis such that the port-channel remains up and fully functional.

You can configure each port channel with a channel group mode parameter. Such configuration determines the port channel protocol behavior for all member ports in this channel group. The possible values for a channel group mode are as follows:

  • On—When the port channel created in ON mode, user has to run no shutdown command for each port channel member. . Port channels that are configured in the On mode require you to explicitly enable and disable the port channel member ports at either end if you add or remove ports from the port channel configuration. Physically verify that the local and remote ports are connected to each other.

    Beginning with Cisco MDS Release 8.4(1), the default mode is changed from On to Active.

  • Active—When the port channel created in ACTIVE mode, members will be operational automatically provided the ISL(s) is operationally up. The Active port channel mode allows automatic recovery without explicitly enabling and disabling the port channel member ports at either end.


Note

From Cisco MDS NX-OS Release 8.4(1), the CLI and the Device Manager create the port channel in Active mode on the NPIV core switches.

The following table provides a comparison between On and Active modes.

Table 1. Channel Group Configuration Differences

On Mode

Active Mode

No protocol is exchanged.

A port channel protocol negotiation is performed with the peer ports.

Moves interfaces to the suspended state if its operational values are incompatible with the port channel.

Moves interfaces to the isolated state if its operational values are incompatible with the port channel.

Port initialization is not synchronized.

There is synchronized startup of all ports in a channel across peer switches.

All misconfigurations are not detected as no protocol is exchanged.

Consistently detect misconfigurations using a port channel protocol.

When you use channel-groupport-channel-idforce command, the misconfigured ports are transitioned to suspended state.

When you use channel-groupport-channel-idforce command, the misconfigured ports are transitioned to isolated state.

Deleting Port Channels

When you delete port channels, their corresponding channel membership is also deleted. All interfaces in the deleted port channel convert to individual physical links. The ports going down gracefully indicates that no frames were lost when the interface went down Graceful Shutdown).

The individual ports within the deleted port channel retain compatibility parameter settings (speed, mode, port VSAN, allowed VSAN, port security and so on). You can modify these settings as required. When you delete port channel in On mode, run no shutdown command on each member to make it operational.

Interfaces in a Port Channel

You can add or remove a physical interface (or range of interfaces) to an existing port channel. For information about port channel support on Generation 2 switching modules, see Port Channel Limitations.

Adding Interfaces to a Port Channel

You can add a physical interface (or range of interfaces) to an existing port channel. The compatible parameters on the configuration are mapped to the port channel. Adding an interface to a port channel increases the channel buffer size and bandwidth of the port channel.

You can add a port as a member of the port channel if the following configurations are same in the port and the port channel:

  • Speed

  • Mode

  • Rate mode

  • Port VSAN

  • Trunking mode

  • Allowed VSAN list or VF-ID list and so on.

After the members are added, regardless of the port channel mode (Active or On), the ports at either end are gracefully down. The ports shut down gracefully indicating that no frames were lost when the interface shut down.

Compatibility Check

A compatibility check ensures that the same parameter settings are used in all physical ports in the channel. Otherwise, they cannot become part of a port channel. The compatibility check is performed before a port is added to the port channel.

The check ensures that the following parameters and settings match at both ends of a port channel:

  • Capability parameters (type of interface, Gigabit Ethernet at both ends, or Fibre Channel at both ends).

  • Administrative compatibility parameters (speed, mode, rate mode, port VSAN, allowed VSAN list, port security and so on).

  • Operational parameters (remote switch WWN and trunking mode).

Adding a port fails if the capability and administrative parameters on the remote switch are incompatible with the capability and administrative parameters on the local switch. If the compatibility check is successful, the interfaces are operational and the corresponding compatibility parameter settings apply to these interfaces.

Suspended and Isolated States

If the operational parameters are incompatible, the compatibility check fails and the interface is transitioned to a suspended or isolated state based on the configured mode:

  • An interface is moved to the suspended state if the interface is configured in the On mode.

  • An interface is moved to the isolated state if the interface is configured in the Active mode.

Force Add an Interface

You can force the port configuration to be overwritten by the port channel configuration. In this case, the interface is added to a port channel.

  • When you use channel-groupport-channel-idforce command, the misconfigured ports in On mode port channel are transitioned to suspended state.

  • When you use channel-groupport-channel-idforce command, the misconfigured ports in Active mode port channel are transitioned to isolated state.

After the members are forcefully added, regardless of the mode (Active and On) used, the ports at either end are gracefully brought down. The ports going down gracefully indicates that no frames were lost when the interface went down (see the Graceful Shutdown) section.

Deleting an Interface from a Port Channel

When a physical interface is deleted from the port channel, the channel membership is automatically updated. If the deleted interface is the last operational interface, then the port channel status is changed to a down state. Deleting an interface from a port channel decreases the channel buffer size and bandwidth of the port channel.

Port Channel Protocols

Cisco NX-OS software provides robust error detection and synchronization capabilities. You can manually configure channel groups or they can be automatically created. In both cases, the channel groups have the same capability and configuration parameters. Any change in configuration applied to the associated port channel interface is propagated to all members of the channel group.

A protocol to exchange port channel configurations is available in all Cisco MDS switches. This simplifies port channel management with incompatible ISLs.

The port channel protocol is enabled by default.

It uses exchange peer parameters (EPP) services to communicate across peer ports in an ISL. Each port uses the information that is received from the peer ports along with its local configuration and operational values to decide if it has to be part of a port channel. The protocol ensures that a set of ports is eligible to be part of the same port channel. They are only eligible to be part of the same port channel if all the ports have a compatible partner.

The port channel protocol uses Bring-up protocol which helps in Automatically detecting the misconfigurations. This protocol synchronizes the port channel at both ends so that all frames for a given flow are carried over the physical link in both directions. This allows applications such as write acceleration, work for port channels over FCIP links.

Creating Port Channels

Assuming the link A1-B1 is operational in the port channel first, that link is considered as First Operational Port (FoP) of the port channel. When the next link is operational, for example, A2-B2, the port channel protocol identifies if this link is compatible with link A1-B1. and adds it to the port channel 10 and 20 in the respective switches. If link A3-B3 can join the the port channels, the respective ports have compatible configurations. If link A4-B4 operates as an individual link, it is because of the incompatible configuration of the two end ports with the other member ports in this channel group.

Figure 4. Autocreating Channel Groups

Port channels are created manually. You can form the port channel with a subset of the ports in the channel group. Incompatible ports remain in a suspended or isolated state depending on the On or Active mode configuration. Any administrative configuration made to the port channel is applied to all ports in the port channel.

Prerequisites for Configuring Port Channels

Before configuring a port channel, consider the following guidelines:

  • Configure the port channel across switching modules to implement redundancy on switching module reboots or upgrades.

  • Ensure that you do not connect a port channel to different sets of switches. Port channels require point-to-point connections between the same set of switches.

If you misconfigure port channels, you get a misconfiguration message. If you receive this message, the port channel’s physical links are disabled because error was detected.

Port channel error is detected if the following requirements are not met:

  • Connect each switch on either side of a port channel to the same number of interfaces.

  • Connect each interface to a corresponding interface on the other side (see Misconfigured Configurations for an example of an invalid configuration).

  • You cannot change the physical links in a port channel after you configure the port channel. If you change the links after you configure the port channel, ensure to reconnect the links to interfaces within the port channel and reenable the links.

If all three conditions are not met, the faulty link is disabled.

Default Settings

The following table describes the default settings for port channels.

Table 2. Default Port Channel Parameters

Parameters

Default

Port channels

FSPF is enabled by default.

Create port channel

Administratively up.

Default port channel mode

Cisco MDS NX-OS Release 8.3(1) and earlier: On mode on non-NPV and NPIV core switches.

Cisco MDS NX-OS Release 8.4(1) and later: Active mode on non-NPV and NPIV core switches.

Active mode on NPV switches.

Guidelines and Limitations

The following sections provide information about guidelines and limitations for Port Channels.

General Guidelines and Limitations

Cisco MDS 9000 Series Multilayer switches support the following number of port channels per switch:

  • A port channel number refers to the unique identifier for each channel group. This number ranges from of 1–256.


    Note

    You cannot change the port channel number, but the member ports operate according to the properties of the port channel.

The following table describes the results of adding a member to a port channel for various configurations.

Generation 1 Port Channel Limitations

This section includes the restrictions on creation and addition of port channel members to a port channel on Generation 1 hardware:

  • The 32-port 2 or 1 Gbps switching module.

  • The MDS 9140 and 9120 switches.

When configuring the host-optimized ports on Generation 1 hardware, the following port channel guidelines apply:

  • If you execute the write erase command on a 32-port switching module, and then copy a saved configuration to the switch from a text file that contains the no system default switchport shutdown command, you need to copy the text file to the switch again for the E ports to come up without manual configuration.

  • Any (or all) full line rate ports in the Cisco MDS 9100 Series can be included in a port channel.

  • The host-optimized ports in the Cisco MDS 9100 Series are subject to the same port channel rules as 32-port switching modules. Only the first port of each group of 4 ports is included in a port channel.

    • You can configure only the first port in each 4-port group as an E port (for example, the first port in ports 1–4, the fifth port in ports 5–8, and so on). If you configure the first port in the group as a port channel, the other three ports in each group (ports 2–4, 6–8, and so on) are not usable and remain in the shutdown state.

    • If you configure any of the other three ports in a no shutdown state, you cannot configure the first port to be a port channel. The other three ports continue to remain in a no shutdown state.

Valid and Invalid Port Channel Examples

Port channels are created with default values. You can change the default configuration just like any other physical interface.

Valid Port Channel Configurations provides examples of valid port channel configurations.

Figure 5. Valid Port Channel Configurations


Misconfigured Configurations provides examples of invalid configurations. Assuming that the links are brought up in the 1, 2, 3, 4 sequence, links 3 and 4 will be operationally down because the fabric is misconfigured.

Figure 6. Misconfigured Configurations


E Port Channel Limitations

The port channel interface must be in Active mode when you configure multiple FCIP interfaces with Write-Acceleration.

F, TF, and NP Port Channel Limitations

The following guidelines and restrictions are applicable for F, TF, and NP port channels:

  • On the switch with feature npiv configured the ports must be in F mode.

  • On the switch with feature npv configured the ports must be in NP mode.

  • On mode is not supported. Only Active-Active mode is supported. By default, the mode is Active on the NPV switches.

  • Devices that are logged in through an F port channel on an MDS switch are not supported in IVR non-NAT configuration. The devices are supported only in IVR NAT configuration.

  • Port security rules are enforced only on physical pWWNs at the single link level.

  • FC-SP authenticates only the first physical FLOGI of every port channel member.

  • Since the FLOGI payload carries only the VF bits to trigger the use of a protocol after the FLOGI exchange, those bits are overridden. If the Cisco NPV switches, the core has a Cisco WWN and tries to initiate the PCP protocol.

  • The name server registration of the N ports logging in through an F port channel uses the fWWN of the port channel interface.

  • DPVM configuration is not supported.

  • You cannot configure the port channel port VSAN using DPVM.

  • The Dynamic Port VSAN Management (DPVM) database is queried only for the first physical FLOGI of each member, so that the port VSAN can be configured automatically.

  • DPVM does not bind FC_IDs to VSANs, but pWWNs to VSANs. It is queried only for the physical FLOGI.

Best Practises for Forwarding Engines

Cisco MDS switches use Ternary Content Addressable Memory (TCAM) on its Fibre Channel modules. TCAM provides an Access Control List (ACL) type of function for Cisco MDS. The process that controls this functionality is called ACLTCAM. The E or TE ports (ISLs) and F (Fabric) ports have their own programming that is unique to their respective port types.

TCAM is allocated to individual forwarding engines and forwarding engines are assigned a group of ports. Director-class Fibre Channel modules have more TCAM space than fabric switches. The number of forwarding engines, the ports assigned to each forwarding engine, and the amount of TCAM allocated to each forwarding engine is hardware dependent.

The following example shows an output from Cisco MDS 9148S:


switch# show system internal acl tcam–soc tcam–usage
TCAM Entries:
=============
                  Region1   Region2    Region3     Region4   Region5   Region6
Mod Fwd   Dir     TOP SYS  SECURITY    ZONING      BOTTOM    FCC DIS   FCC ENA
    Eng          Use/Total Use/Total  Use/Total   Use/Total Use/Total Use/Total
––– –––  –––––– –––––––––– ––––––––– –––––––––––– ––––––––– ––––––––– –––––––––
1   1    INPUT     19/407     1/407      1/2852 *    4/407     0/0       0/0
1   1    OUTPUT     0/25      0/25       0/140       0/25      0/12      1/25
1   2    INPUT     19/407     1/407      0/2852 *    4/407     0/0       0/0
1   2    OUTPUT     0/25      0/25       0/140       0/25      0/12      1/25
1   3    INPUT     19/407     1/407      0/2852 *    4/407     0/0       0/0
1   3    OUTPUT     0/25      0/25       0/140       0/25      0/12      1/25
–––––––––––––––––––––––––––––––––––––––––––––––––––
* 1024 entries are reserved for LUN Zoning purpose.

The above example indicates the following:

  • There are three forwarding engines, 1 through 3.

  • Since there are 48 ports on Cisco MDS 9148 switches, each forwarding engine handles 16 ports.

  • Each forwarding engine has 2852 entries in region 3 (the zoning region) for input. This is the main region used, and consequently, has the largest amount of available entries.

  • Forwarding engine 3 has only one entry that is currently in use out of the total 2852 in the zoning region.

The following example shows the output from Cisco MDS 9710 switch with a 2/4/8/10/16 Gbps Advanced Fibre Channel Module (DS–X9448–768K9):


F241–15–09–9710–2# show system internal acl tcam–usage
TCAM Entries:
=============
                  Region1   Region2    Region3     Region4   Region5   Region6
Mod Fwd   Dir     TOP SYS  SECURITY    ZONING      BOTTOM    FCC DIS   FCC ENA
    Eng          Use/Total Use/Total  Use/Total   Use/Total Use/Total Use/Total
––– –––  –––––– –––––––––– ––––––––– –––––––––––– ––––––––– ––––––––– –––––––––
1   0    INPUT     55/19664    0/9840     0/49136*   17/19664    0/0       0/0
1   0    OUTPUT    13/4075    0/1643     0/11467     0/4075    6/1649   21/1664
1   1    INPUT     52/19664    0/9840     2/49136*   14/19664    0/0       0/0
1   1    OUTPUT     7/4078    0/1646     0/11470     0/4078    6/1652    5/1651
1   2    INPUT     34/19664    0/9840     0/49136*   10/19664    0/0       0/0
1   2    OUTPUT     5/4078    0/1646     0/11470     0/4078    6/1652    1/1647
1   3    INPUT     34/19664    0/9840     0/49136*   10/19664    0/0       0/0
1   3    OUTPUT     5/4078    0/1646     0/11470     0/4078    6/1652    1/1647
1   4    INPUT     34/19664    0/9840     0/49136*   10/19664    0/0       0/0
1   4    OUTPUT     5/4078    0/1646     0/11470     0/4078    6/1652    1/1647
1   5    INPUT     34/19664    0/9840     0/49136*   10/19664    0/0       0/0
1   5    OUTPUT     5/4078    0/1646     0/11470     0/4078    6/1652    1/1647
...

The above example indicates the following:

  • There are six forwarding engines, 0 through 5.

  • Since there are 48 ports on a Cisco MDS DS–X9448–768K9 module, each forwarding engine handles eight ports.

  • Each forwarding engine has 49136 entries in region 3 (the zoning region) for input. This is the main region that is used, and consequently, has the largest amount of available entries.

  • Forwarding engine 2 has only two entries that are currently in use out of the total 49136 in the zoning region.

The following example shows the output from Cisco MDS 9396V switch with a 2/4/8/10/16/32/64 Gbps Advanced Fibre Channel Module (DS–X9448–768K9): 

switch9396v# show system internal acl tcam–usage
Input TCAM Entries:
=====================
                  Region1   Region2       Region3         Region4
Mod Fwd   Dir     TOP SYS  SECURITY       ZONING          BOTTOM
    Eng          Use/Total Use/Total   Use/Total(Anl)  Use/Total(Anl)
--- ---  ------ ---------- ---------- ---------------- --------------
1   0    INPUT    126/26208    0/13120     0/65536(0)    28/26208(0)
1   1    INPUT    122/26208    0/13120     2/65536(0)    27/26208(0)
1   2    INPUT    150/26208    0/13120     0/65536(0)    32/26208(0)
1   3    INPUT    126/26208    0/13120     0/65536(0)    28/26208(0)

Output TCAM Entries:
======================
    Fwd           Region1   Region2       Region3         Region4      Region5   Region6
Mod Eng/  Dir     TOP SYS  SECURITY       ZONING          BOTTOM       FCC DIS   FCC ENA
    Port         Use/Total Use/Total   Use/Total(Anl)  Use/Total(Anl) Use/Total Use/Total
    Num
--- ---  ------ ---------- ---------- ---------------- -------------- --------- ---------
1   0    OUTPUT     4/51      0/51       0/281  (0)     0/51  (0)      4/25      3/51
1   1    OUTPUT     4/51      0/51       0/281  (0)     0/51  (0)      4/25      1/51
1   2    OUTPUT     4/51      0/51       0/281  (0)     0/51  (0)      4/25      1/51
1   3    OUTPUT     4/51      0/51       0/281  (0)     0/51  (0)      4/25      1/51
1   4    OUTPUT     4/51      0/51       0/281  (0)     0/51  (0)      4/25      1/51
.
..
...
1   94    OUTPUT     4/51      0/51       0/281  (0)     0/51  (0)      4/25      1/51
1   95    OUTPUT     4/51      0/51       0/281  (0)     0/51  (0)      4/25      1/51
Note: Analytics Entry Count(Anl) included in Use count.

The above example indicates the following:

  • There are four forwarding engines, 0 through 3.

  • Since there are 96 ports on a Cisco MDS DS-C9396V-K9-SUP module, each forwarding engine handles twenty-four ports.

  • Each forwarding engine has 65536 entries in region 3 (the zoning region) for input. This is the main region that is used, and consequently, has the largest amount of available entries.

  • Forwarding engine 2 has only two entries that are currently in use out of the total 65536 in the zoning region.


Note
The commands that are used to view TCAM usage on fabric switches are different from the ones used for director–class switches. For MDS 9148, MDS 9148S, and MDS 9250i fabric switches, use the show system internal acltcam-soc tcam-usage command. For director class switches, MDS 9396V, MDS 9396S, and 32 Gbps fabric switches, use the show system internal acl tcam-usage command.

The following table provides information about ports to forwarding engines mapping:

Table 3. Ports to Forwarding Engines Mapping

Switch or Module

Forwarding Engines

Port Ranges

Forwarding Engine Number

Zoning Region Entries

Bottom Region Entries

MDS 9132T

2

1–16

0

49136

19664

17–32

1

49136

19664

MDS 9148

3

fc1/25–36 and fc1/45–48

1

2852

407

fc1/5–12 and fc1/37–44

2

2852

407

fc1–4 and fc1/13–24

3

2852

407

MDS 9148S

3

fc1/1–16

1

2852

407

fc1/17–32

2

2852

407

fc1/33–48

3

2852

407

MDS 9148T

3

1–16

0

49136

19664

17–32

1

49136

19664

33–48

2

49136

19664

MDS 9250i

4

fc1/5–12 and eth1/1–8

1

2852

407

fc1/1–4, fc1/13–20, and fc1/37–40

2

2852

407

fc1/21–36

3

2852

407

ips1/1–2

4

2852

407

MDS 9396S

12

fc1/1–8

0

49136

19664

fc1/9–16

1

49136

19664

fc1/17–24

2

49136

19664

fc1/25–32

3

49136

19664

fc1/33–40

4

49136

19664

fc1/41–48

5

49136

19664

fc1/49–56

6

49136

19664

fc1/57–64

7

49136

19664

fc1/65–72

8

49136

19664

fc1/73–80

9

49136

19664

fc1/81–88

10

49136

19664

fc1/89–96

11

49136

19664

MDS 9396T

6

1–16

0

49136

19664

17–32

1

49136

19664

33–48

2

49136

19664

49–64

3

49136

19664

65-80

4

49136

19664

81-96

5

49136

19664

DS–X9248–48K9

1

1–48

0

27168

2680

DS–X9248–96K9

2

1–24

0

27168

2680

25–48

1

27168

2680

DS–X9224–96K9

2

1–12

0

27168

2680

13–24

1

27168

2680

DS–X9232–256K9

4

1–8

0

49136

19664

9–16

1

49136

19664

17–24

2

49136

19664

25–32

3

49136

19664

DS–X9248–256K9

4

1–12

0

49136

19664

13–24

1

49136

19664

25–36

2

49136

19664

37–48

3

49136

19664

DS–X9448–768K9

6

1–8

0

49136

19664

9–16

1

49136

19664

17–24

2

49136

19664

25–32

3

49136

19664

33–40

4

49136

19664

41–48

5

49136

19664

DS–X9334–K9

3

1–8

0

49136

19664

9–16

1

49136

19664

17–24

2

49136

19664

DS–X9648–1536K9

3

1–16

0

49136

19664

17–32

1

49136

19664

33–48

2

49136

19664

DS-C9124V-K9

1

1-24

0

65536

26208

DS-C9148V-24EK9

2

1-24

0

65536

26208

25-48

1

65536

26208

DS-C9220I-K9

1

1-12

0

49136

19664

DS-X9748-3072-K9

2

1-24

0

65536

26208

25-48

1

65536

26208

DS-C9396V-K9

4

1-24

0

65536

26208

25-48

1

65536

26208

49-72

2

65536

26208

73-96

3

65536

26208

Best Practises for E and TE Port Channels and IVR

Port channels provide Inter Switch Links (ISLs) between switches. Typically, there is minimal TCAM programming on these types of interfaces. When the Inter VSAN Routing(IVR) feature is being deployed, extensive TCAM programming can exist on ISLs because the IVR topology transitions from one VSAN to another. Most of the considerations that apply on F/TF port channels will be applicable here too.

The following is an example of a topology:

In this topology:

  • Both Cisco MDS 9148S-1 and MDS 9148S-2 are in the IVR VSAN topology:

    
MDS9148S-1 vsan 1 and vsan 2
MDS9148S-2 vsan 2 and vsan 3

  • IVR NAT is configured.

  • VSAN 2 is the transit VSAN.

    
FCIDs per VSAN:
            VSAN 1  VSAN 2  VSAN 3
Host     			010001  210001  550002
Target1 			440002  360002  030001

    Note
    Domains 0x44 in VSAN 1, 0x21 and 0x36 in VSAN 2, and 0x55 in VSAN 3 are virtual domains created by IVR NAT.

  • The following is the IVR zoning topology:

    
ivr zone zone1
member host vsan 1
member target1 vsan3

  • The following is the ACL TCAM programming for the IVR zoning topology:

    
MDS9148S-1  fc1/1(Host) - VSAN 1
Entry# 			Source ID    				Mask      Destination ID  								Mask    Action
1      			010001(host) 				ffffff    440002(target1) 								ffffff  Permit
           - Forward to fc1/2
       - Rewrite the following information:
         VSAN to 2
         Source ID to 210001
         Destination ID to 360002
2      			000000     				000000    000000          								000000  Drop
MDS9148S-1  fc1/2(ISL) - VSAN 2
Entry# 			Source ID       					Mask      Destination ID  								Mask    Action
1      			360002(Target1) 					ffffff    210001(host)    								ffffff  Permit
       - Forward to fc1/2
       - Rewrite the following information:
         VSAN to 1
         Source ID to 440002
         Destination ID to 010001
MDS9148S-2 fc1/2(ISL) - VSAN 2
Entry# 			Source ID    				Mask      Destination ID  								Mask    Action
1      			210001(host) 				ffffff    360002(target1) 								ffffff  Permit
       - Forward to fc1/2
       - Rewrite the following information:
         VSAN to 3
         Source ID to 550002
         Destination ID to 030001
MDS9148S-2  fc1/1(Target1) - VSAN 3
Entry# 			Source ID       					Mask      Destination ID  								Mask    Action
1      			030001(Target1) 					ffffff    550002(host)    								ffffff  Permit
       - Forward to fc1/2
       - Rewrite the following information:
         VSAN to 2
         Source ID to 360002
         Destination ID to 210001
2      			000000     					000000    000000          									000000  Drop

    Note

    Besides the entries in this example, there are other entries that IVR adds to capture important frames such as PLOGIs, PRILIs, and ABTS.

The programming on the host and target1 ports is similar to the way it is without IVR, except that the FCIDs and VSANs are explicitly forwarded to an egress port and are rewritten to values that are appropriate for the transit VSAN (VSAN 2). These forwarding and rewrite entries are separate and are not included in the TCAM-usage values.

However,now, on the ISLs in both the switches, programming that did not exist earlier is present. When frames from Host to Target1 are received by Cisco MDS 9148S-2 fc1/2, they are rewritten to the values in VSAN3 where the target resides. In the reverse direction, when frames from Target1 to the Host are received by Cisco MDS 9148S-1 fc1/2, they are rewritten to the values in VSAN 1 where the Host resides. Therefore, for each VSAN transition on an ISL (that typically occurs across a transit VSAN) there is TCAM programming for each device in the IVR zone set.

Consequently,most of the best practices followed for the F and TF port channels should be followed to ensure that TCAM is utilized as efficiently as possible for the following purposes:


Note
Unlike F and TF port-channels, the ACLTCAM programming on ISLs will be the same quantity regardless if the ISLs are part of a port-channel or not. If there are "n" ISLs between two MDS switches, then it doesn't matter if they are in one port-channel, two port-channels or just individual links. The ACLTCAM programming will be the same.

  • Distribute port-channel member interfaces into different forwarding engines, especially on fabric switches.

  • Distribute member interfaces into different linecards on director-class switches.

  • Distribute member interfaces into forwarding engines with lower TCAM zoning region usage.

  • Use single-initiator zones, single-target zones, or Smart Zoning.

Configuring Port Channels

To create a port channel, perform these steps:

Procedure

Step 1

switch# configure terminal

Enters configuration mode.

Step 2

switch(config)# interface port-channel 1

Configures the specified port channel (1) using the default ON mode.

Configuring Port Channel Mode

From Cisco MDS NX-OS Release 8.4(1), the CLI and the Device Manager create the port channel in Active mode on the NPIV core switches.


Note
An F port channel is supported only on Active mode.

To configure Active mode, perform these steps:

Procedure

Step 1

switch# configure terminal

Enters configuration mode.

Step 2

switch(config)# interface port-channel 1

Configures the specified port channel (1) using the default Active mode.

Step 3

switch(config-if)# no channel mode active

Configures the specified port channel in default On mode.

Deleting Port Channels

To delete a port channel, perform these steps:

Procedure

Step 1

switch# configure terminal

Enters configuration mode.

Step 2

switch(config)# no interface port-channel 1

Deletes the specified port channel (1), its associated interface mappings, and the hardware associations for this port channel.

Adding an Interface to a Port Channel

To add an interface to a port channel, perform these steps:

Procedure

Step 1

switch# configure terminal

Enters configuration mode.

Step 2

switch(config)# interface fc1/15

Configures the specified port interface (fc1/15).

Step 3

switch(config-if)# channel-group 15

Adds physical Fibre Channel port 1/15 to channel group 15. If the channel-group 15 does not exist, it is created. The port is shut down if the port channel in the On mode.

Adding a Range of Ports to a Port Channel

To add range of ports to a port channel, perform these steps:

Procedure

Step 1

switch# configure terminal

Enters configuration mode.

Step 2

switch(config)# interface fc1/1 - 5

Configures the specified range of interfaces. In this example, interfaces from 1/1 to 1/5 are configured.

Step 3

switch(config-if)# channel-group 2

Adds physical interfaces 1/1, 1/2, 1/3, 1/4, and 1/5 to channel group 2. If the channel-group 2 does not exist, it is created.

If the compatibility check is successful, the interfaces are operational, and the corresponding states apply to these interfaces.

What to do next


Note
By default, the CLI adds an interface to a port channel, while NDFC SAN Controller adds the interface by force, unless specified explicitly.

Adding an Interface using force command

To force the addition of a port to a port channel, perform these steps:

Procedure

Step 1

switch# configure terminal

Enters configuration mode.

Step 2

switch(config)# interface fc1/1

Specifies the interface fc1/1.

Step 3

switch(config-if)# channel-group 1 force

Forces the addition of the physical port for interface fc1/1 to channel group 1. The port is shut down in port channel ON mode For port channel in Active mode, the interfaces recovers automatically (for consistency with earlier instances)

Step 4

switch(config-if)#no shutdown

Execute this command on the port channel in On mode to make the ports operational.

Removing an Interface from a Port Channel

To remove a physical interface (or range of physical interfaces) from a port channel, perform these steps:

Procedure

Step 1

switch# configure terminal

Enters configuration mode.

Step 2

switch(config)# interface fc1/1

Enters the selected physical interface level.

Step 3

switch(config)# interface fc1/1 - 5

Enters the selected range of physical interfaces.

Step 4

switch(config-if)# no channel-group 2

Removes the physical Fibre Channel interfaces from channel group 2.

Verifying Port Channel Configuration

To display the port channel configuration information, perform one of the following tasks:

Command

Purpose

show port-channel summary

Displays a summary of port channels within the switch. A one-line summary of each port channel provides the administrative state, the operational state, the number of attached and active interfaces (up), and the first operational port (FOP). The FOP is the primary operational interface that is selected in the port channel to carry control-plane traffic (no load-balancing). The FOP is the first port that comes up in a port channel and can change if the port goes down. The FOP is also identified by an asterisk (*).

show port-channel database

Displays the port channel that is configured in the On mode (default) and Active mode.

show port-channel consistency

Displays the consistency status without details.

show port-channel consistency detail

Displays the consistency status with details.

show port-channel usage

Displays the port channel usage.

show port-channel compatibility-parameters

Displays the port channel compatibility.

show interface fc slot/port

Displays interfaces information.

show port-channel database interface port-channel number

Displays the specified port channel interface.

how running-config interface port-channelnumber

Displays the current running configuration for port-channel interface,

For detailed information about the fields in the output from these commands, refer to Cisco MDS 9000 Series NX-OS Command Reference Guide.

You can view specific information about existing port channels at any time from EXEC mode. The following show commands provide further details on existing port channels. You can force all screen output to go to a printer or save it to a file. See Examples Displays the Port Channel Summary to Displays the Port Channel Summary.

Examples


switch# show port-channel summary
 ------------------------------------------------------------------------------
Interface                 Total Ports        Oper Ports        First Oper Port
------------------------------------------------------------------------------
port-channel 77                2                 0                  --
port-channel 78                2                 0                  --
port-channel 79                2                 2                  fcip200

Examples


switch# show port-channel database
port-channel1
     Administrative channel mode is on
     Last membership update succeeded
     First operational port is fc1/19
     4 ports in total, 2 ports up
     Ports:   fc1/19   [up] *
              fc1/20   [up]
              fc1/21   [down]
              fc1/22   [down

port-channel2
     Administrative channel mode is on
     Last membership update succeeded
     First operational port is fcip3
     2 ports in total, 2 ports up 
     Ports:	fcip1	[up]
                 fcip3	[up] *

Examples


switch# show port-channel database
 
port-channel1
     Administrative channel mode is active
     Last membership update succeeded
     First operational port is fc1/19
     4 ports in total, 2 ports up
     Ports:   fc1/19   [up] *
              fc1/20   [up]
              fc1/21   [down]
              fc1/22   [down

port-channel2
     Administrative channel mode is active
     Last membership update succeeded
     First operational port is fcip3
     2 ports in total, 2 ports up
     Ports:	fcip1	[up]
                 fcip3	[up] *

Examples

The show port-channel consistency command has two options: without details and with details. 

switch# show port-channel consistency
Database is consistent

Examples

switch# show port-channel consistency detail
Authoritative port-channel database:
================================================
 total 1 port-channels
 port-channel 100:
     4 ports, first operational port is fc1/19
     fc1/22   [down]
     fc1/21   [down]
     fc1/19   [up]
     fc1/20   [up] 
================================================
 database 1: from module 1 
================================================
 total 1 port-channels
 port-channel 100:
     4 ports, first operational port is fc1/19
     fc1/19   [up]
     fc1/20   [up]
     fc1/21   [down]
     fc1/22   [down] 
================================================ 
switch#

Examples

The show port-channel usage command displays details of the used and unused port channel numbers. 
switch# show port-channel usage 
Totally 4 port-channel numbers used
===================================
 Used :	–77 -79, 100
 Unused:	1 - 76, 80 – 99, 101 - 256

Examples

switch# show port-channel compatibility-parameters
physical port layer                 fibre channel or ethernet
    port mode                           E/AUTO only
    trunk mode
    speed
    port VSAN
    port allowed VSAN list

Examples


switch# show port-channel database
 interface port-channel 100
 port-channel 100
     Administrative channel mode is active
     Last membership update succeeded
     First operational port is fc1/19
     4 ports in total, 2 ports up
     Ports:   fc1/19   [up] *
              fc1/20   [up]
              fc1/21   [down]
              fc1/22   [down]

Examples


switch# show port-channel summary
------------------------------------------------------------------------------
Interface                 Total Ports        Oper Ports        First Oper Port
------------------------------------------------------------------------------
port-channel 1                 1                 0                  --
port-channel 2                 1                 1                  fc8/13
port-channel 3                 0                 0                  --
port-channel 4                 0                 0                  --
port-channel 5                 1                 1                  fc8/3
port-channel 6                 0                 0                  --

Configuration Examples for F and TF Port Channels

This example shows how to configure an F port channel in shared mode and bring up the link (not supported on the MDS 91x4 switches) between F ports on the Cisco NPIV core switches and NP ports on the Cisco NPV switches:

Procedure

Step 1

Enable the F port trunking and channeling protocol on the MDS core switch.

Example:

switch(config)# feature fport-channel-trunk

Step 2

Enable NPIV on the MDS core switch:

Example:

switch(config)# feature npiv

Step 3

Create the port channel on the MDS core switch:

Example:

switch(config)# interface port-channel 1
switch(config-if)# switchport mode F
switch(config-if)# channel mode active
switch(config-if)# switchport trunk mode off
switch(config-if)# exit

Step 4

Configure the port channel member interfaces on the core switch:

Example:

switch(config)# interface fc2/1-3
switch(config-if)# shut
switch(config-if)# switchport mode F
switch(config-if)# switchport trunk mode off
switch(config-if)# switchport speed 8000
switch(config-if)# channel-group 1
switch(config-if)# no shut
switch(config-if)# exit

Step 5

Create the port channel on the NPV switch:

Example:

switch(config)# interface port-channel 1
switch(config-if)# switchport mode NP
switch(config-if)# exit

Step 6

Configure the port channel member interfaces on the NPV switch:

Example:

switch(config)# interface fc2/1-3
switch(config-if)# shut
switch(config-if)# switchport mode NP
switch(config-if)# switchport speed 8000
switch(config-if)# switchport trunk mode off
switch(config-if)# channel-group 1
switch(config-if)# no shut
switch(config-if)# exit

Step 7

Set the administrative state of all the port-channel member interfaces in both NPIV core switch and the NPV switch to ON:

Example:

switch(config)# interface fc1/1-3
switch(config-if)# shut
switch(config-if)# no shut
switch(config)# interface fc2/1-3
switch(config-if)# shut
switch(config-if)# no shut