Configuring IEEE 802.3ad Link Bundling and Load Balancing

This document describes how the IEEE 802.3ad link bundling and load balancing leverages the EtherChannel infrastructure within Cisco software to manage the bundling of various links. Also described are network traffic load-balancing features to help minimize network disruption that results when a port is added or deleted from a link bundle.

Finding Feature Information

Your software release may not support all the features documented in this module. For the latest caveats and feature information, see Bug Search Tool and the release notes for your platform and software release. To find information about the features documented in this module, and to see a list of the releases in which each feature is supported, see the feature information table.

Use Cisco Feature Navigator to find information about platform support and Cisco software image support. To access Cisco Feature Navigator, go to www.cisco.com/​go/​cfn. An account on Cisco.com is not required.

Prerequisites for Configuring IEEE 802.3ad Link Bundling and Load Balancing

  • Knowledge of how EtherChannels and Link Aggregation Control Protocol (LACP) function in a network

  • Knowledge of load balancing to mitigate network traffic disruptions

  • Verification that both ends of the LACP link have the same baseline software version

Restrictions for Configuring IEEE 802.3ad Link Bundling and Load Balancing

  • The number of links supported per bundle is bound by the platform.

  • All links must operate at the same link speed and in full-duplex mode (LACP does not support half-duplex mode).

  • All links must be configured either as EtherChannel links or as LACP links.

  • Only physical interfaces can form aggregations. Aggregations of VLAN interfaces are not possible nor is an aggregation of aggregations.

  • If a router is connected to a switch, the bundle terminates on the switch.

  • An EtherChannel will not form if one of the LAN ports is a Switched Port Analyzer (SPAN) destination port.

  • All ports in an EtherChannel must use the same EtherChannel protocol.

  • The LACP Single Fault Direct Load Balance Swapping feature is limited to a single bundled port failure.

  • The LACP Single Fault Direct Load Balance Swapping feature cannot be used with the Port Aggregation Protocol (PagP).

  • LACP port priority cannot be configured with LACP single fault direct load balance swapping.

  • The adaptive algorithm does not apply to service control engines (SCEs) when EtherChannel load distribution is used.

Information About Configuring IEEE 802.3ad Link Bundling and Load Balancing

Gigabit EtherChannel

Gigabit EtherChannel is high-performance Ethernet technology that provides Gbps transmission rates. A Gigabit EtherChannel bundles individual Gigabit Ethernet links into a single logical link that provides the aggregate bandwidth of up to eight physical links. All LAN ports in each EtherChannel must be the same speed and all must be configured either as Layer 2 or as Layer 3 LAN ports. Inbound broadcast and multicast packets on one link in an EtherChannel are blocked from returning on any other link in the EtherChannel.

When a link within an EtherChannel fails, traffic previously carried over the failed link switches to the remaining links within that EtherChannel. Also when a failure occurs, a trap is sent that identifies the device, the EtherChannel, and the failed link.

Port Channel and LACP-Enabled Interfaces

Each EtherChannel has a numbered port channel interface that, if not already created, is created automatically when the first physical interface is added to the channel group. The configuration of a port channel interface affects all LAN ports assigned to that port channel interface.

To change the parameters of all ports in an EtherChannel, change the configuration of the port channel interface: for example, if you want to configure Spanning Tree Protocol or configure a Layer 2 EtherChannel as a trunk. Any configuration or attribute changes you make to the port channel interface are propagated to all interfaces within the same channel group as the port channel; that is, configuration changes are propagated to the physical interfaces that are not part of the port channel but are part of the channel group.

The configuration of a LAN port affects only that LAN port.

IEEE 802.3ad Link Bundling

The IEEE 802.3ad Link Bundling feature provides a method for aggregating multiple Ethernet links into a single logical channel based on the IEEE 802.3ad standard. This feature helps improve the cost effectiveness of a device by increasing cumulative bandwidth without necessarily requiring hardware upgrades. In addition, IEEE 802.3ad link bundling provides a capability to dynamically provision, manage, and monitor various aggregated links and enables interoperability between various Cisco devices and devices of third-party vendors.

LACP uses the following parameters:

  • LACP port priority—You must configure an LACP port priority on each port configured to use LACP. The port priority can be configured automatically or through the CLI. LACP uses the port priority to decide which ports should be put in standby mode when there is a hardware limitation that prevents all compatible ports from aggregating. LACP also uses the port priority with the port number to form the port identifier.

Benefits of IEEE 802.3ad Link Bundling

  • Increased network capacity without changing physical connections or upgrading hardware

  • Cost savings from the use of existing hardware and software for additional functions

  • A standard solution that enables interoperability of network devices

  • Port redundancy without user intervention when an operational port fails

EtherChannel Load Balancing

EtherChannel load balancing can use MAC addresses; IP addresses; Layer 4 port numbers; either source addresses, destination addresses, or both; or ports. The selected mode applies to all EtherChannels configured on the device.

Traffic load across the links in an EtherChannel is balanced by reducing part of the binary pattern, formed from the addresses in the frame, to a numerical value that selects one of the links in the channel. When a port is added to an EtherChannel or an active port fails, the load balance bits are reset and reassigned for all ports within that EtherChannel and reprogrammed into the ASIC for each port. This reset causes packet loss during the time the reassignment and reprogramming is taking place. The greater the port bandwidth, the greater the packet loss.

Load Distribution in an EtherChannel

In earlier Cisco software releases, only a fixed load distribution algorithm was supported. With this fixed algorithm, the load share bits are assigned sequentially to each port in the bundle. Consequently, the load share bits for existing ports change when a member link joins or leaves the bundle. When these values are programmed in the ASIC, substantial traffic disruption and, in some cases, duplication of traffic can occur.

The EtherChannel Load Distribution feature enhances the load distribution mechanism with the adaptive load distribution algorithm. This algorithm uses a port reassignment scheme that enhances EtherChannel availability by limiting the load distribution reassignment to the port that is added or deleted. The new load on existing bundled ports does not conflict with the load programmed on those ports when a port is added or deleted.

You can enable this feature in either global configuration mode or interface configuration mode. The algorithm is applied at the next hash-distribution instance, which usually occurs when a link fails, is activated, added, or removed, or when shutdown or no shutdown is configured.

Because the selected algorithm is not applied until the next hash-distribution instance, the current and configured algorithms could be different. If the algorithms are different, a message is displayed alerting you to take appropriate action. For example: 

Device(config-if)# port-channel port hash-distribution fixed 
This command will take effect upon a member link UP/DOWN/ADDITION/DELETION event.
Please do a shut/no shut to take immediate effect

Also, the output of the show etherchannel command is enhanced to show the applied algorithm when the channel group number is specified. This output enhancement is not available, though, when the protocol is also specified because only protocol-specific information is included. Following is an example of output showing the applied algorithm: 

Device# show etherchannel 10 summary
Flags:  D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      N - not in use, no aggregation
        f - failed to allocate aggregator
<snip>
Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
10     Po10(RU)        LACP      Gi3/7(P)       Gi3/9(P)       
! The following line of output is added with support 
of the EtherChannel Load Distribution feature. !
Last applied Hash Distribution Algorithm: Fixed

How to Configure IEEE 802.3ad Link Bundling and Load Balancing

Enabling LACP

SUMMARY STEPS

    1.    enable

    2.    configure terminal

    3.    interface port-channel channel-number

    4.    exit

    5.    interface type number

    6.    channel-group channel-group-number mode {active | passive}

    7.    end


DETAILED STEPS
     Command or ActionPurpose
    Step 1 enable


    Example: 
    Device> enable
     

    Enables privileged EXEC mode.

    • Enter your password if prompted.

     
    Step 2 configure terminal


    Example: 
    Device# configure terminal
     

    Enters global configuration mode.

     
    Step 3 interface port-channel channel-number


    Example: 
    Device(config)# interface port-channel 10
     

    Identifies the interface port channel and enters interface configuration mode.

     
    Step 4 exit


    Example: 
    Device(config-if)# exit
     

    Returns to global config mode.

     
    Step 5 interface type number


    Example: 
    Device(config)# interface Gigabitethernet 1/0/5
     

    Configures an interface and enters interface configuration mode.

     
    Step 6 channel-group channel-group-number mode {active | passive}


    Example: 
    Device(config-if)# channel-group 10 mode active
     

    Configures the interface in a channel group and sets it as active.

    • In active mode, the port initiates negotiations with other ports by sending Link Aggregate Control Protocol (LACP) packets.

     
    Step 7 end


    Example: 
    Device(config-if)# end
     

    Returns to privileged EXEC mode.

     

    Configuring a Port Channel

    You must manually create a port channel logical interface. Perform this task to configure a port channel.

    SUMMARY STEPS

      1.    enable

      2.    configure terminal

      3.    interface port-channel channel-number

      4.    no switchport

      5.    ip address ip-address mask

      6.    end

      7.    show running-config interface port-channel group-number


    DETAILED STEPS
       Command or ActionPurpose
      Step 1 enable


      Example: 
      Device> enable
       

      Enables privileged EXEC mode.

      • Enter your password if prompted.

       
      Step 2 configure terminal


      Example: 
      Device# configure terminal
       

      Enters global configuration mode.

       
      Step 3 interface port-channel channel-number


      Example: 
      Device(config)# interface port-channel 10
       

      Identifies the interface port channel and enters interface configuration mode.

       
      Step 4 no switchport


      Example: 
      Device(config-if)# no switchport
       

      Puts an interface into Layer 3 mode.

       
      Step 5 ip address ip-address mask


      Example: 
      Device(config-if)# ip address 172.31.52.10 255.255.255.0
       

      Assigns an IP address and subnet mask to the EtherChannel.

       
      Step 6 end


      Example: 
      Device(config-if)# end
       

      Returns to privileged EXEC mode.

       
      Step 7 show running-config interface port-channel group-number


      Example: 
      Device# show running-config interface port-channel 10
       

      Displays the port channel configuration.

       

      Example

      This example shows how to verify the configuration: 

      Device# show running-config interface port-channel10
 
Building configuration...
Current configuration:
!
no switchport
interface Port-channel10
 ip address 172.31.52.10 255.255.255.0
 no ip directed-broadcast
end

      Setting LACP System Priority

      Perform this task to set the Link Aggregation Control Protocol (LACP) system priority. The system ID is the combination of the LACP system priority and the MAC address of a device.

      SUMMARY STEPS

        1.    enable

        2.    configure terminal

        3.    lacp system-priority priority

        4.    end

        5.    show lacp sys-id


      DETAILED STEPS
         Command or ActionPurpose
        Step 1 enable


        Example:
        Device> enable
         

        Enables privileged EXEC mode.

        • Enter your password if prompted.

         
        Step 2 configure terminal


        Example:
        Device# configure terminal
         

        Enters global configuration mode.

         
        Step 3 lacp system-priority priority


        Example:
        Device(config)# lacp system-priority 200
         

        Sets the system priority.

         
        Step 4 end


        Example:
        Device(config)# end
         

        Returns to privileged EXEC mode.

         
        Step 5 show lacp sys-id


        Example:
        Device# show lacp sys-id
         

        Displays the system ID, which is a combination of the system priority and the MAC address of the device.

         

        Example

        This example shows how to verify the LACP configuration: 

        Device# show lacp sys-id
20369,01b2.05ab.ccd0

        Adding and Removing Interfaces from a Bundle

        SUMMARY STEPS

          1.    enable

          2.    configure terminal

          3.    interface type number

          4.    channel-group channel-group-number mode {active | passive}

          5.    no channel-group

          6.    end


        DETAILED STEPS
           Command or ActionPurpose
          Step 1 enable


          Example:
          Device> enable
           

          Enables privileged EXEC mode.

          • Enter your password if prompted.

           
          Step 2 configure terminal


          Example:
          Device# configure terminal
           

          Enters global configuration mode.

           
          Step 3 interface type number


          Example:
          Device(config)# interface gigabitethernet 5/0/0
           

          Configures an interface and enters interface configuration mode.

           
          Step 4 channel-group channel-group-number mode {active | passive}


          Example:
          Device(config-if)# channel-group 5 mode active
           

          Adds an interface to a channel group.

           
          Step 5 no channel-group


          Example:
          Device(config-if)# no channel-group
           

          Removes the interface from the channel group.

           
          Step 6 end


          Example:
          Device(config-if)# end
           

          Returns to privileged EXEC mode.

           

          Monitoring LACP Status

          SUMMARY STEPS

            1.    enable

            2.    show lacp {number | counters | internal | neighbor | sys-id}


          DETAILED STEPS
             Command or ActionPurpose
            Step 1 enable


            Example:
            Device> enable
             

            Enables privileged EXEC mode.

            • Enter your password if prompted.

             
            Step 2 show lacp {number | counters | internal | neighbor | sys-id}


            Example:
            Device# show lacp internal
             

            Displays internal device information.

             

            Troubleshooting Tips

            To verify and isolate a fault, start at the highest level maintenance domain and do the following:

            • Check the device error status.

            • When an error exists, perform a loopback test to confirm the error.

            • Run a traceroute to the destination to isolate the fault.

            • If the fault is identified, correct the fault.

            • If the fault is not identified, go to the next lower maintenance domain and repeat these four steps at that maintenance domain level.

            • Repeat the first four steps, as needed, to identify and correct the fault.

            Configuration Examples for IEEE 802.3ad Link Bundling and Load Balancing

            Example: Adding and Removing Interfaces from a Bundle

            The following example shows how to add an interface to a bundle: 

            Device# show lacp internal
Flags:  S - Device is requesting Slow LACPDUs 
        F - Device is requesting Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode     
Channel group 5
                            LACP port     Admin     Oper    Port        Port
Port      Flags   State     Priority      Key       Key     Number      State
Gi7/0/0   SA      bndl      32768         0x5       0x5     0x43        0x3D  
Device# configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
Device(config)# interface gigabitethernet 5/0/0
Device(config-if)# channel-group 5 mode active
Device(config-if)#
*Aug 20 17:10:19.057: %LINK-3-UPDOWN: Interface GigabitEthernet5/0/0, changed state to down
*Aug 20 17:10:19.469: %C10K_ALARM-6-INFO: ASSERT CRITICAL GigE 5/0/0 Physical Port Link Down 
*Aug 20 17:10:19.473: %C10K_ALARM-6-INFO: CLEAR CRITICAL GigE 5/0/0 Physical Port Link Down 
*Aug 20 17:10:21.473: %LINK-3-UPDOWN: Interface GigabitEthernet5/0/0, changed state to up
*Aug 20 17:10:21.473: GigabitEthernet7/0/0 taken out of port-channel5
*Aug 20 17:10:23.413: GigabitEthernet5/0/0 added as member-1 to port-channel5
 
*Aug 20 17:10:23.473: %LINK-3-UPDOWN: Interface Port-channel5, changed state to up
Device(config-if)# end
Device#
*Aug 20 17:10:27.653: %SYS-5-CONFIG_I: Configured from console by console
*Aug 20 17:11:40.717: GigabitEthernet7/0/0 added as member-2 to port-channel5
 
Device# show lacp internal
Flags:  S - Device is requesting Slow LACPDUs 
        F - Device is requesting Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode     
Channel group 5
                            LACP port     Admin     Oper    Port        Port
Port      Flags   State     Priority      Key       Key     Number      State
Gi7/0/0   SA      bndl      32768         0x5       0x5     0x43        0x3D  
Gi5/0/0   SA      bndl      32768         0x5       0x5     0x42        0x3D  
Device# show interface port-channel5
Port-channel5 is up, line protocol is up 
  Hardware is GEChannel, address is 0014.a93d.4aa8 (bia 0000.0000.0000)
  MTU 1500 bytes, BW 2000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  ARP type: ARPA, ARP Timeout 04:00:00
    No. of active members in this channel: 2 
        Member 0 : GigabitEthernet5/0/0 , Full-duplex, 1000Mb/s  <---- added to port channel bundle
        Member 1 : GigabitEthernet7/0/0 , Full-duplex, 1000Mb/s  
  Last input 00:00:00, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/150/0/0 (size/max/drops/flushes); Total output drops: 0
  Interface Port-channel5 queueing strategy: PXF First-In-First-Out
  Output queue 0/8192, 0 drops; input queue 0/150, 0 drops
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     104 packets output, 8544 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

            The following example shows how to remove an interface from a bundle: 

            Device# configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
Device(config)# interface gigabitethernet 7/0/0
Device(config-if)# no channel-group
Device(config-if)#
*Aug 20 17:15:49.433: GigabitEthernet7/0/0 taken out of port-channel5
*Aug 20 17:15:49.557: %C10K_ALARM-6-INFO: ASSERT CRITICAL GigE 5/0/0 Physical Port Link Down 
*Aug 20 17:15:50.161: %C10K_ALARM-6-INFO: CLEAR CRITICAL GigE 5/0/0 Physical Port Link Down 
*Aug 20 17:15:51.433: %LINK-3-UPDOWN: Interface GigabitEthernet7/0/0, changed state to down
*Aug 20 17:15:52.433: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet7/0/0, changed state to down
Device(config-if)# end
Device#
*Aug 20 17:15:58.209: %SYS-5-CONFIG_I: Configured from console by console
Device#
*Aug 20 17:15:59.257: %C10K_ALARM-6-INFO: ASSERT CRITICAL GigE 7/0/0 Physical Port Link Down 
*Aug 20 17:15:59.257: %C10K_ALARM-6-INFO: CLEAR CRITICAL GigE 7/0/0 Physical Port Link Down   
Device#
*Aug 20 17:16:01.257: %LINK-3-UPDOWN: Interface GigabitEthernet7/0/0, changed state to up
*Aug 20 17:16:02.257: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet7/0/0, changed state to up
Device# show lacp internal
Flags:  S - Device is requesting Slow LACPDUs 
        F - Device is requesting Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode     
Channel group 5
                            LACP port     Admin     Oper    Port        Port
Port      Flags   State     Priority      Key       Key     Number      State
Gi5/0/0   SA      bndl      32768         0x5       0x5     0x42        0x3D

            Example: Monitoring LACP Status

            The following example shows Link Aggregation Protocol (LACP) activity that you can monitor by using the show lacp command. 

            Device# show lacp internal
Flags:  S - Device is requesting Slow LACPDUs 
        F - Device is requesting Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode     
Channel group 5
                            LACP port     Admin     Oper    Port        Port
Port      Flags   State     Priority      Key       Key     Number      State
Gi5/0/0   SA      bndl      32768         0x5       0x5     0x42        0x3D  
Device# show lacp 5 counters
             LACPDUs         Marker      Marker Response    LACPDUs
Port       Sent   Recv     Sent   Recv     Sent   Recv      Pkts Err
---------------------------------------------------------------------
Channel group: 5
Gi5/0/0     21     18       0      0        0      0         0     
Device# show lacp 5 internal
Flags:  S - Device is requesting Slow LACPDUs 
        F - Device is requesting Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode     
Channel group 5
                            LACP port     Admin     Oper    Port        Port
Port      Flags   State     Priority      Key       Key     Number      State
Gi5/0/0   SA      bndl      32768         0x5       0x5     0x42        0x3D  
Device# show lacp 5 neighbor
Flags:  S - Device is requesting Slow LACPDUs 
        F - Device is requesting Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode     
Channel group 5 neighbors
Partner's information:
          Partner Partner   LACP Partner  Partner   Partner  Partner     Partner
Port      Flags   State     Port Priority Admin Key Oper Key Port Number Port State
Gi5/0/0   SP      32768     0011.2026.7300  11s    0x1     0x14     0x3C  
Device# show lacp counters
             LACPDUs         Marker      Marker Response    LACPDUs
Port       Sent   Recv     Sent   Recv     Sent   Recv      Pkts Err
---------------------------------------------------------------------
Channel group: 5
Gi5/0/0     23     20       0      0        0      0         0     
Device# show lacp sys-id
32768,0014.a93d.4a00

            Additional References for IEEE 802.3ad Link Bundling and Load Balancing

            Related Documents

            Related Topic

            Document Title

            Configuring EtherChannels

            “Configuring Layer 3 and Layer 2 EtherChannel” chapter of the Catalyst 6500 Release 12.2SXF Software Configuration Guide

            Configuring the Cisco Catalyst 3850 Series Switch

            Catalyst 3850 Series Switch Configuration Guide

            Configuring Carrier Ethernet

            Carrier Ethernet Configuration Guide

            Link Aggregation Control Protocol (LACP) commands: complete command syntax, command mode, command history, defaults, usage guidelines, and examples

            Cisco IOS Carrier Ethernet Command Reference

            Cisco IOS commands: master list of commands with complete command syntax, command mode, command history, defaults, usage guidelines, and examples

            Cisco IOS Master Command List, All Releases

            Standards

            Standard

            Title

            IEEE 802.3ad-2000

            IEEE 802.3ad-2000 Link Aggregation

            MIBs

            MIB

            MIBs Link

            802.3ad MIB

            To locate and download MIBs for selected platforms, Cisco software releases, and feature sets, use Cisco MIB Locator found at the following URL:

            http:/​/​www.cisco.com/​go/​mibs

            Technical Assistance

            Description

            Link

            The Cisco Support and Documentation website provides online resources to download documentation, software, and tools. Use these resources to install and configure the software and to troubleshoot and resolve technical issues with Cisco products and technologies. Access to most tools on the Cisco Support and Documentation website requires a Cisco.com user ID and password.

            http:/​/​www.cisco.com/​cisco/​web/​support/​index.html

            Feature Information for Configuring IEEE 802.3ad Link Bundling and Load Balancing

            The following table provides release information about the feature or features described in this module. This table lists only the software release that introduced support for a given feature in a given software release train. Unless noted otherwise, subsequent releases of that software release train also support that feature.

            Use Cisco Feature Navigator to find information about platform support and Cisco software image support. To access Cisco Feature Navigator, go to . An account on Cisco.com is not required.
            Table 1 Feature Information for Configuring IEEE 802.3ad Link Bundling and Load Balancing

            Feature Name

            Releases

            Feature Information

            IEEE 802.3ad Link Bundling and Load Balancing

            Cisco IOS XE 3.2SE

            IEEE 802.3ad link bundling and load balancing leverages the EtherChannel infrastructure within Cisco software to manage the bundling of various links. The network traffic load-balancing features help minimize network disruption that results when a port is added or deleted from a link bundle.

            In Cisco IOS XE Release 3.2SE, this feature is supported on the following platforms:

            • Cisco 5700 Series Wireless LAN Controllers

            • Cisco Catalyst 3850 Series Switches

            In Cisco IOS XE Release 3.3SE, this feature is supported on the following platforms:

            • Cisco Catalyst 3650 Series Switches