This list describes the properties of ethernet link bundles:

• The router supports mixed speed bundles. Mixed speed bundles allow member links of different bandwidth to be configured as active members in a single bundle. The ratio of the bandwidth for bundle members must not exceed 10. Also, the total weight of the bundle must not exceed 64.

• The weight of each bundle member is the ratio of its bandwidth to the lowest bandwidth member. Total weight of the bundle is the sum of weights or relative bandwidth of each bundle member. Since the weight for a bundle member is greater than or equal to 1 and less than or equal to 10, the total member of links in a bundle is less than 64 in mixed bundle case.

• Any type of Ethernet interfaces can be bundled, with or without the use of LACP (Link Aggregation Control Protocol).

• A single router can support maximum number of bundle interfaces. Link bundles of only physical interfaces are supported. Following are maximum numbers of bundle interfaces supported on NCS 540 variants:

  Medium Density XR NCS 540 - N540-24Z8Q2C-SYS, N540-28Z4C-SYS, N540X-ACC-SYS, N540-ACC-SYS

  Medium Density XR NCS 540 - N540-28Z4C-SYS-A, N540-28Z4C-SYS-D, N540X-16Z4G8Q2C-A, N540X-16Z4G8Q2C-D, N540X-16Z8Q2C-D, N540-12Z20G-SYS-A,N540-12Z20G-SYS-D, N540X-12Z16G-SYS-A, N540X-12Z16G-SYS-D

  Small Density XR NCS 540 - N540X-6Z18G-SYS-A, N540X-6Z18G-SYS-D, N540X-8Z16G-SYS-A, N540X-8Z16G-SYS-D

  Table 1. Bundle Interfaces on NCS 540 Routers

  Supported Features	Medium Density XR NCS 540		Small Density XR NCS 540
  Bundle Interfaces	256	256	24
  Maximum bundle members	64	64	8
  Bundle sub-interfaces	1024	1024	1024
  Layer2 Bundle Interfaces	1023	1023	1023
  hw-module profile bundle-scale command	Supported	Not Supported	Not Supported

• The hw-module profile bundle-scale <256/512/1024> command is supported only on the following NCS 540 router variants:

  N540-24Z8Q2C-SYS, N540-28Z4C-SYS, N540X-ACC-SYS, N540-ACC-SYS

  The total number of supported bundle members with HQoS profile on Layer2 and Layer3 interfaces:

  • hw-module profile bundle-scale 256 —Total bundle interfaces + total bundle sub-interfaces is 256

  • hw-module profile bundle-scale 512 — Total bundle interfaces + total bundle sub-interfaces is 512

  • hw-module profile bundle-scale 1024 — Total bundle interfaces + total bundle sub-interfaces is 1024

• The following limitations apply to bundle sub-interfaces and the number of members per bundle on Layer3 interfaces on Medium Density XR NCS 540 variants:

  • Maximum of 1024 bundle sub-interfaces, each containing up to 16 member-links.

  • Maximum of 256 bundle sub-interfaces, each containing up to 64 member-links

  • Maximum of 512 bundle sub-interfaces, each containing up to 32 member-links

• Physical layer and link layer configuration are performed on individual member links of a bundle.

• Configuration of network layer protocols and higher layer applications is performed on the bundle itself.

• IPv4 and IPv6 addressing is supported on ethernet link bundles.

• A bundle can be administratively enabled or disabled.

• Each individual link within a bundle can be administratively enabled or disabled.

• Ethernet link bundles are created in the same way as Ethernet channels, where the user enters the same configuration on both end systems.

• QoS is supported and is applied proportionally on each bundle member.

• In case static MAC address is configured on a bundle-ether interface, the following limitations are applied:

  • Locally generated packets, such as ICMP, BGP, and so on, going out from the interface have the source MAC address as the statically configured MAC address.

  • Transit (forwarded) packets going out of the interface do not have the configured static MAC as source MAC address. In such a scenario, the upper 36-bits come from the system MAC address (or the original/dynamic MAC address) and the lower 12-bits come from the MAC address configured on the bundle. To check the dynamic pool of MAC addresses included, use the show ethernet mac-allocation detail command.

    For example, if the dynamic MAC address was 008A.9624.48D8 and the configured static MAC address is 0011.2222.ABCD. Then, the source MAC for transit (forwarded) traffic will be 008A.9624.4BCD.

    Note	This limitation can cause traffic blackholing for the transit traffic, in case there is L2 ACL applied for security purpose. In such case, it is necessary to add permit statement for both MAC addresses in the L2 ACL.

• Load balancing (the distribution of data between member links) is done by flow instead of by packet. Data is distributed to a link in proportion to the bandwidth of the link in relation to its bundle.

• All links within a single bundle must terminate on the same two systems.

• Bundled interfaces are point-to-point.

• A link must be in the up state before it can be in distributing state in a bundle.

• Only physical links can be bundle members.

• Multicast traffic is load balanced over the members of a bundle. For a given flow, the internal processes selects the member link, and the traffic for the flow is sent over that member.

To configure link bundling, you must understand the following concepts:

802.1Q VLAN subinterfaces can be configured on 802.3ad Ethernet link bundles. Keep the following information in mind when adding VLANs on an Ethernet link bundle:

Note	The memory requirement for bundle VLANs is slightly higher than standard physical interfaces.

To create a VLAN subinterface on a bundle, include the VLAN subinterface instance with the interface Bundle-Ether command, as follows:

interface Bundle-Ether interface-bundle-id.subinterface

After you create a VLAN on an Ethernet link bundle, all VLAN subinterface configuration is supported on that link bundle.

VLAN subinterfaces can support multiple Layer 2 frame types and services, such as Ethernet Flow Points - EFPs) and Layer 3 services.

Layer 2 EFPs are configured as follows:

interface bundle-ether instance.subinterface l2transport. encapsulation dot1q xxxxx 

Layer 3 VLAN subinterfaces are configured as follows:

interface bundle-ether instance.subinterface, encapsulation dot1q xxxxx 

Note	The difference between the Layer 2 and Layer 3 interfaces is the l2transport keyword. Both types of interfaces use dot1q encapsulation .

As packets are exchanged across member links of a bundled interface, some member links may slow down or time-out and fail. LACP packets are exchanged periodically across these links to verify the stability and reliability of the links over which they pass. The configuration of short period time intervals, in which LACP packets are sent, enables faster detection and recovery from link failures.

Short period time intervals are configured as follows:

• In milliseconds

• In increments of 100 milliseconds

• In the range 100 to 1000 milliseconds

• The default is 1000 milliseconds (1 second)

• Up to 1280 packets per second (pps)

After 6missed packets, the link is detached from the bundle.

When the short period time interval is not configured, LACP packets are transmitted over a member link every 30 seconds by default.

When the short period time interval is configured, LACP packets are transmitted over a member link once every 1000 milliseconds (1 second) by default. Optionally, both the transmit and receive intervals can be configured to less than 1000 milliseconds, independently or together, in increments of 100 milliseconds (100, 200, 300, and so on).

When you configure a custom LACP short period transmit interval at one end of a link, you must configure the same time period for the receive interval at the other end of the link.

Note

You must always configure the transmit interval at both ends of the connection before you configure the receive interval at either end of the connection. Failure to configure the transmit interval at both ends first results in route flapping (a route going up and down continuously). When you remove a custom LACP short period, you must do it in reverse order. You must remove the receive intervals first and then the transmit intervals.

In a scaled setup, a bundle programming check is difficult to perform and time consuming. Moreover, an issue is reported only when the user detects it, and not automatically. During multiple test executions, it isn’t possible to detect the initial failure, which causes other subsequent failures. Bundle Consistency Checker (BCC) implements bundle programming and consistency check by using the following steps:

1. BCC uses the running configuration to detect discrepancies.

2. BCC forms a Bundle Consistency Checker Data Base (BCCDB) with the bundle, sub-bundle, or member information fetched from the running configuration.

3. BCC dumps the required data from all available nodes. It then uses BCCDB as a source to verify bundle programming and consistency in all other layer dumps.

4. BCC reports inconsistencies, programming errors, stale entries, and deletes any pending objects.

Supporting Interfaces

The following interfaces support BCC:

• Bundle

• Bundle sub-interface

The following table lists BCC behaviour during inconsistencies in bundle configuration or programming errors.

Router# show bundle consistency 
Building configuration...
Dumping Data............................ Done
Parsing Data............................ Not Done

BCC Stopped: Found 3 info/exceptions/errors

Logs Preview:

2020-07-13 10:34:22,774: INFO: Bundlemgr PD dont have any bundle data

2020-07-13 10:34:22,832: INFO: BMPI dont have any bundle data

2020-07-13 10:34:23,728: INFO: No Bundle is configured/No member is added to Bundle

Logs: 
/var/log/bcc_exception.log
/var/log/bcc_debug.log

Router# show bundle consistency
Building configuration...
Dumping Data............................ Done
Parsing Data............................ Not Done

BCC Stopped: Found 4 info/exceptions/errors

Logs Preview:

2020-07-13 10:36:32,513: INFO: Bundlemgr PD dont have any bundle data

2020-07-13 10:36:32,566: INFO: No member is added to bundle BE1(0x3c00400c)

2020-07-13 10:36:32,566: INFO: BMPI dont have any bundle data

2020-07-13 10:36:33,453: INFO: No Bundle is configured/No member is added to Bundle

Logs: 
/var/log/bcc_exception.log
/var/log/bcc_debug.log

Router# show bundle consistency
Building configuration...
Dumping Data............................ Done
Parsing Data............................ Done
Bundle Consistency Check................ Done
Bundle Programming Check................ Done
Stale Entry Check....................... Done
Bundle Health Check..................... Done

Overall Results:

Inconsistencies             : 0        Stale Entries                : 0
BCM Programming Error       : 0        Delete Pending DPA Objects   : 0
Info/Error/Python Exception : 0        Overall Bundle Health Status : WARNING

Execute 'show bundle status' to see detailed reason for 'WARNING' in bundle health check

Router# show bundle consistency
Building configuration...
Dumping Data............................ Done
Parsing Data............................ Done
Bundle Consistency Check................ Done
Bundle Programming Check................ Done
Stale Entry Check....................... Done
Bundle Health Check..................... Done

Overall Results:

Inconsistencies             : 0        Stale Entries                : 0
BCM Programming Error       : 0        Delete Pending DPA Objects   : 0
Info/Error/Python Exception : 3        Overall Bundle Health Status : WARNING

Execute 'show bundle status' to see detailed reason for 'WARNING' in bundle health check

Logs Preview:


2020-07-12 17:38:26,568: INFO: No member is added to bundle BE2(0x80042bc) ==> l2 bundle main

2020-07-12 17:38:32,573: interface Bundle-Ether1.1: Dont have any encapsulation config. ==> l3 sub

2020-07-12 17:38:32,574: interface Bundle-Ether1.130: Dont have any encapsulation config.  ==> l2sub

Logs:

/var/log/bcc_inconsistencies.log
/var/log/bcc_programming_error.log
/var/log/bcc_stale_entries.log
/var/log/bcc_delay_delete.log
/var/log/bcc_bundle_health.log
/var/log/bcc_exception.log
/var/log/bcc_debug.log

Router# show bundle consistency

Building configuration...
Dumping Data............................ Done
Parsing Data............................ Done
Bundle Consistency Check................ Done
Bundle Programming Check................ Done
Stale Entry Check....................... Done
Bundle Health Check..................... Done

Overall Results:

Inconsistencies             : 0        Stale Entries                : 0
BCM Programming Error       : 1        Delete Pending DPA Objects   : 0
Info/Error/Python Exception : 0        Overall Bundle Health Status : WARNING

Execute 'show bundle status' to see detailed reason for 'WARNING' in bundle health check

Logs Preview:

2020-07-12 18:48:22,658: Programming Error 1: BE1(0x80041ec) NPU 0,0/RP0/CPU0 Vlan Domain 0x33 != GigabitEthernet0_0_0_2 Vlan Domain 0xa

Logs:

/var/log/bcc_inconsistencies.log
/var/log/bcc_programming_error.log
/var/log/bcc_stale_entries.log
/var/log/bcc_delay_delete.log
/var/log/bcc_bundle_health.log
/var/log/bcc_exception.log
/var/log/bcc_debug.log