Cisco Nexus 5000 Series NX-OS FabricPath Configuration Guide, Release 5.2(1)N1(1)

FabricPath Encapsulation

This section includes the following topics:

• FabricPath Headers
• Forwarding Tags (FTags)

FabricPath Headers

When a frame enters the FabricPath network, the software encapsulates the Layer 2 frame with a new FabricPath header. The switch IDs that the software assigns to each FabricPath switch as it enters the FabricPath network is used as the outer MAC destination address (ODA) and outer MAC source address (OSA) in the FabricPath header. Figure 2-1 shows the FabricPath header encapsulation of the classical Ethernet (CE) frame.

Figure 2-1 FabricPath Frame Encapsulation

The software applies the encapsulation on the ingressing edge port of the FabricPath network and decapsulates the frame on the egressing edge port of the FabricPath network; all the ports within the FabricPath network are FabricPath ports that use only the hierarchical MAC address (see Chapter 3, “Configuring FabricPath Interfaces”, for more information on configuring FabricPath interfaces). This feature greatly reduces the size of the MAC tables in the core of the FabricPath network.

The software automatically assigns each switch in the FabricPath network with a unique switch ID. Optionally, you can configure the switch ID for the FabricPath switch.

The outer source address (OSA) is the FabricPath switch ID of the switch where the frame ingresses the FabricPath network, and the outer destination address (ODA) is the FabricPath switch ID of the switch where the frame egresses the FabricPath network. When the frame egresses the FabricPath network, the FabricPath switch strips the FabricPath header, and the original CE frame continues on the CE network. The FabricPath network uses only the OSA and ODA, with the Layer 2 IS-IS protocol transmitting the topology information. Both the FabricPath ODA and OSA are in a standard MAC format (xxxx.xxxx.xxxx).

The FabricPath hierarchical MAC address carries the reserved EtherType 0x8903.

When the frame is originally encapsulated, the software sets the time to live (TTL) to 32; on each hop through the FabricPath network, each switch decrements the TTL by 1. If the TTL reaches 0, that frame is discarded. This feature prevents the continuation of any loops that might form in the network.

Forwarding Tags (FTags)

The Forwarding Tag (FTag) in the FabricPath header specifies which path the packet traverses throughout the FabricPath network. The switch uses the FTag-specified paths for multidestination packets that enter the FabricPath network. The FTag, which is a fixed route that the switch learns from the topology, is a 10-bit field with the values from 1 to 1024 (see “Configuring FabricPath Forwarding,” for more information).

The software assigns the FTag to the FabricPath header on the edge port as the frame ingresses the FabricPath network. All subsequent FabricPath switches in that FabricPath network recognize the FTag. Each FTag is unique within the FabricPath topology.

Default IS-IS Behavior with FabricPath

The interfaces in a FabricPath network run only the FabricPath Layer 2 IS-IS protocol; you do not need to run STP in the FabricPath network because FabricPath Layer 2 IS-IS discovers topology information dynamically.

Note Cisco Nexus 5500 Series switches supports two Fabricpath topologies, the default or base topology (topology 0), and the local VLANs topology (topology 1).

FabricPath Layer 2 IS-IS is a dynamic link-state routing protocol that detects changes in the network topology and calculates loop-free paths to other nodes in the network. Each FabricPath switch maintains a link-state database (LSDB) that describes the state of the network; each switch updates the status of the links that are adjacent to the switch. The FabricPath switch sends advertisements and updates to the LSDB through all the existing adjacencies. FabricPath Layer 2 IS-IS protocol packets do not conflict with standard Layer 2 IS-IS packets because the FabricPath packets go to a different Layer 2 destination MAC address than that used by standard IS-IS for IPv4/IPv6 address families.

The software sends hello packets on the FabricPath core ports to form adjacencies. After the software forms IS-IS adjacencies, the FabricPath unicast traffic uses the equal cost multipath (ECMP) feature of Layer 2 IS-IS to forward traffic. ECMP provides up to 16 paths for unicast traffic.

Within the FabricPath network, you use a single control plane protocol, Layer 2 IS-IS, for all unicast, multicast, and broadcast traffic. To use the basic FabricPath functionality, you do not need to configure Layer 2 IS-IS. The control plane Layer 2 IS-IS comes up and runs automatically when you enable FabricPath on the switch.

The loop-free Layer 2 IS-IS protocol builds two trees for the topology. One tree carries unknown unicast and broadcast traffic and the second tree carries multicast traffic. Multicast and unknown unicast traffic are load-balanced between the two trees (see “Configuring FabricPath Forwarding,” for more information about trees and topology).

FabricPath Layer 2 IS-IS is based on the standard IS-IS protocol with the following extensions for the FabricPath environment:

• FabricPath has a single IS-IS area with no hierarchical Layer 1/Layer 2 routing as prescribed within the IS-IS standard. All switches within the FabricPath network are in a single Layer 1 area.
• The switch uses a MAC address that is different from the MAC address used for Layer 3 IS-IS instances.
• The software adds a new sub-TLV that carries switch ID information, which is not in standard IS-IS. This feature allows Layer 2 information to be exchanged through the existing IS-IS protocol implementation.
• Each switch computes its shortest path to every other switch in the network using the shortest-path first (SPF) algorithm. This path is used for forwarding unicast FabricPath frames. FabricPath Layer 2 IS-IS uses the standard IS-IS functionality to populate up to 16 routes for a given destination switch. The software uses multiple equal-cost available parallel links that provide ECMP.
• FabricPath IS-IS introduces certain modifications to the standard IS-IS in order to support the construction of broadcast and multicast trees (identified by the FTags). Specifically, using FabricPath, the software constructs two loop-free trees for forwarding multidestination traffic.

Once the adjacency is established among the switches in the FabricPath network, the software sends update information to all neighbors.

By default, you can run Layer 2 IS-IS with FabricPath with no configuration, However, you can fine-tune some of the Layer 2 IS-IS parameters (see “Configuring Advanced FabricPath Features,” for information on configuring optional IS-IS parameters).

Additionally, FabricPath IS-IS helps to ensure that each switch ID in steady-state is unique within the FabricPath network. If FabricPath networks merge, switch IDs might collide. If the IDs are all dynamically assigned, FabricPath IS-IS ensures that this conflict is resolved without affecting any FabricPath traffic in either network,

Conversational MAC Address Learning

In traditional MAC address learning, each switch learns the MAC address of every other host in the network. When you configure a VLAN for conversational learning, the associated switches learn only those MAC addresses that are actively in conversation with the hosts connected to the edge interfaces of those switches. Not all switches have to learn all the MAC addresses, which greatly reduces the size of the MAC address tables.

You can optimize the MAC learning process. Conversational MAC learning is configured per VLAN. All FabricPath VLANs use conversational learning if a switch virtual interface (SVI) is not enabled on the switch. (See “Configuring FabricPath Forwarding,” for more information on CE and FabricPath VLANs.)

Conversational MAC address learning results in MAC address tables that are much smaller for FabricPath.

The MAC address learning modes are the traditional learning and conversational learning. The following VLAN modes have the following MAC learning modes:

• FabricPath VLANs—Conversational MAC learning is enabled unless you enable an SVI.
• CE VLANs—Only traditional learning.

With conversational MAC learning, the switch learns the source MAC address of an ingressing frame only if destination MAC address is already learnt and present in the MAC address table. It requires three-way handshake for the MAC address to be learnt. Unknown MAC address are forwarded, or flooded throughout the network.

Switching Using FabricPath

The FabricPath hierarchical MAC address scheme and conversational learning result in much smaller, conversational learning MAC tables within the FabricPath network. Within the FabricPath network, the software uses Layer 2 IS-IS to transmit topology information. The switches on the edge of the network, which use conversational MAC address learning, do not have to learn all the MAC addresses in the network (see Figure 2-2).

Figure 2-2 FabricPath Ports Use Only the FabricPath Header to Switch Frames

MAC mobility is expedited using the FabricPath hierarchical MAC addresses. That is, when you want to move a host and keep its same MAC address and VLANs, only the switches at the edge of the FabricPath network track this change. Within the FabricPath network, the FabricPath switches update their tables with only the outer MAC addresses (ODA and OSA) that have changed from the FabricPath encapsulation.

The switch on the edge of the FabricPath network encapsulates the original frame inside the FabricPath header. Once the frame reaches the last, or directly connected, FabricPath switch, the egress interface strips the FabricPath header and forwards the frame as a normal CE frame.

The ports at the edge of a FabricPath network can use conversational learning to learn only those MAC addresses that the specified edge port is having a bidirectional conversation with. Every edge switch does not have to learn the MAC address of every other edge switch; they only need to learn the MAC addresses of the speakers.

As the frame traverses the FabricPath network, all the switches work only with the FabricPath header. So, the FabricPath switches work only with the ODAs and OSAs; they do not need to learn the MAC address for any of the CE hosts or other switches attached to the network. The hierarchical MAC addressing provided by the FabricPath headers results in much smaller MAC tables in the FabricPath network. The switches in the FabricPath network only need to know how to forward frames to another FabricPath switch so they can forward traffic without requiring large MAC address lookup tables in the core of the network.

The switches in the FabricPath network decrement the TTL in the FabricPath header by 1 at each hop. When the TTL reaches 0, the packet is dropped. This process prevents the continuation of any loops that might form in the network.

Conflict Resolution and Optional FabricPath Tunings

After you enable FabricPath in all switches, the software automatically assigns a random switch ID to each FabricPath switch. The switch ID is a 12-bit value that is dynamically assigned to every switch in the FabricPath network, with each switch being a unique value in that FabricPath network. Optionally, you can configure a specific switch ID. If any of the switch IDs in the FabricPath network are not unique, the software provides automatic conflict resolution.

The software chooses a random value for the switch ID and sets this value as tentative during a period when the software waits to hear if this value is already in use. If this value is being used by another switch in the network, the software begins a conflict resolution process. The switch with the lower switch ID keeps the specified value and the other switch gets a new value for its switch ID.

If a single switch joins an existing FabricPath network, that switch changes its switch ID value rather than any of the existing switches in the network changing values. If the specified value is not in use by another switch or after the conflict is resolved, the switch ID is marked as confirmed.

Graceful migration provides that there is no traffic disruption if a conflict arises in the resources, such as two switches that temporarily have the same switch ID.

The FabricPath interfaces come up, but they are not operational until the switch checks for FabricPath conflicts and resolves those conflicts.

The FabricPath resources timers have default values, but you can also change the timer values. You can tune the switch to wait longer or shorter periods to check the conflicts.

Some of the important processes of the FabricPath network are as follows:

• Achieves a conflict-free allocation of switch IDs and FTags
• Provides graceful resource migration during network merges or partition healing
• Supports static switch IDs
• Provides fast convergence during link bringup or network merges

FabricPath uses the Layer 2 IS-IS protocol to transport the database to all switches in the network. The information is distributed among the FabricPath network switches using an IS-IS TLV. Each switch sends its version of the database that contains information about all the switches.

The software allocates the FabricPath values, guarantees their uniqueness within the FabricPath network, and deletes the value from the database once that resource is no longer needed.

When you manually configure static switch IDs for the switch, the automatic conflict resolution process does not work and the network does not come up. You will see syslog messages about the conflict and must manually change one or more switch IDs of the switches in the network.

Installing the FabricPath Feature Set

You must install the FabricPath feature set on all switches on which you want to configure FabricPath.

BEFORE YOU BEGIN

Ensure that you have installed an Enhanced Layer 2 license on all switches on which you want to install FabricPath.

SUMMARY STEPS

1. configure terminal

2. install feature-set fabricpath

3. exit

4. (Optional) show feature-set

5. (Optional) copy running-config startup-config

DETAILED STEPS

This example shows how to install the FabricPath feature set on a switch:

Uninstalling the FabricPath Feature Set

BEFORE YOU BEGIN

Ensure that you have disabled the FabricPath featureset on all switches from which you want to uninstall FabricPath. See the “Disabling the FabricPath Feature Set” section for more information.

SUMMARY STEPS

1. configure terminal

2. no install feature-set fabricpath

3. exit

4. (Optional) show feature-set

5. (Optional) copy running-config startup-config

DETAILED STEPS

This example shows how to uninstall the FabricPath feature set from a switch:

Enabling the FabricPath Feature Set

You must enable the FabricPath feature set on all switches on which you want to run FabricPath. You cannot access the commands that you use to configure the feature until the feature set is installed and enabled.

BEFORE YOU BEGIN

Ensure that you have installed the FabricPath feature on the switch.

SUMMARY STEPS

1. configure terminal

2. feature-set fabricpath

3. exit

4. (Optional) show feature-set

5. (Optional) copy running-config startup-config

DETAILED STEPS

This example shows how to enable the FabricPath feature set on a switch:

Disabling the FabricPath Feature Set

When you disable the FabricPath feature set, the switch clears all FabricPath configurations from the switch and you can no longer see any of the CLI commands that you need to configure FabricPath.

Note If your FabricPath configuration is large (multiple megabytes in size), disabling the FabricPath feature set may take some time to complete.

SUMMARY STEPS

1. configure terminal

2. no feature-set fabricpath

3. exit

4. (Optional) show feature-set

5. (Optional) copy running-config startup-config

DETAILED STEPS

This example shows how to disable the FabricPath feature:

Creating a FabricPath Topology

You must install the FabricPath feature set on all switches on which you want to configure FabricPath.

SUMMARY STEPS

1. configure terminal

2. fabricpath topology topology-number

3. (Optional) copy running-config startup-config

DETAILED STEPS

	Command	Purpose
Step 1	configure terminal   Example: switch# configure terminal switch(config)#	Enters global configuration mode.
Step 2	fabricpath topology topology-number   Example: switch(config)# fabricpath topology 1	Specifies the FabricPath topology number. Note Cisco Nexus Series only supprts 2 topologies; the default or base topology (topology 0) and the local VLAN topology (topology 2).
Step 3	show fabricpath topology   Example: switch# show fabricpath	(Optional) Displays information about the FabricPath IS-IS toopology.
Step 4	copy running-config startup-config   Example: switch# copy running-config startup-config	(Optional) Copies the running configuration to the startup configuration.

This example shows how to configure the FabricPath topology:

Changing a FabricPath Switch ID (Optional)

By default, FabricPath assigns a unique switch ID to a switch after you enable FabricPath on the switch. However, you can manually change the switch ID if you prefer.

Note You do not lose any traffic during switch ID changes.

SUMMARY STEPS

1. configure terminal

2. fabricpath switch-id value

3. exit

4. (Optional) show fabricpath switch-id

5. (Optional) copy running-config startup-config

DETAILED STEPS

This example shows how to manually change the FabricPath switch ID on a switch to 25:

Configuring the FabricPath Timers (Optional)

You can configure the following FabricPath timers:

• allocate-delay—Configures the delay for a new switch ID to be propagated throughout the network before that value becomes available and permanent.
• linkup-delay—Configures the link bringup delay to detect conflicts in the switch ID. If the software does find a conflict, the software takes some time to resolve the conflict and bring FabricPath to an operational state.
• transition-delay—Configures the delay for propagating a transitioned value in the network; during this period, all old and new switch ID values exist in the network. This situation occurs only while the link comes up and the software checks to see if the network has two identical switch IDs.

SUMMARY STEPS

1. configure terminal

2. fabricpath timers { allocate-delay seconds | linkup-delay seconds | transition-delay seconds}

3. exit

4. (Optional) show fabricpath timers

5. (Optional) copy running-config startup-config

DETAILED STEPS

This example shows how to configure the allocation-delay for a FabricPath timer to 600 seconds:

Disabling FabricPath Graceful Merges (Optional)

By default, graceful merges are enabled. If desired, you can disable this aspect of the FabricPath feature. However, if you disable graceful merges, you must do so on all FabricPath switches.

Note You may experience traffic drops if you disable this feature.

SUMMARY STEPS

1. configure terminal

2. [no] fabricpath graceful-merge disable

3. exit

4. (Optional) show running-config

5. (Optional) copy running-config startup-config

DETAILED STEPS

This example shows how to disable graceful merges on a FabricPath switch:

Forcing the Links to Come Up (Optional)

Note We do NOT recommend that you use the fabricpath force link-bringup command to force the network links to connect.

As a one-time event, you can force the FabricPath network links to connect if they are not coming up because of switch ID conflicts or other problems in the network.

You must repeat this configuration on every switch in the FabricPath network.

SUMMARY STEPS

Step 1 fabricpath force link-bringup

DETAILED STEPS

This example shows how to force the FabricPath network links to come up as a one-time event: