Feedback
Table Of Contents
Configuring FabricPath Switching
Information About FabricPath Switching
Default IS-IS Behavior with FabricPath
Conversational MAC Address Learning
Conflict Resolution and Optional FabricPath Tunings
Interoperation between the M Series and the F Series Modules
Licensing Requirements for FabricPath Switching
Prerequisites for FabricPath Switching
Guidelines and Limitations for FabricPath Switching
Configuring FabricPath Switching
Enabling the FabricPath Feature Set on the VDC on the device
Disabling the FabricPath Feature Set on the VDC
Configuring the MAC Learning Mode for CE VLANs (Optional)
Configuring the Switch ID (Optional)
Configuring the FabricPath Timers (Optional)
Disabling FabricPath Graceful Merges (Optional)
Forcing the Links to Come Up (Optional)
Verifying FabricPath Switching
Displaying and Clearing FabricPath Switching Statistics
FabricPath Switching Example Configuration
Feature History for Configuring FabricPath Switching Using the CLI
Configuring FabricPath Switching
Note
You must have an F Series module installed in your Cisco Nexus 7000 Series chassis in order to run FabricPath and conversational learning.
This chapter describes how to configure FabricPath switching on the Cisco NX-OS devices.
This chapter includes the following sections:
•
•
•
•
•
•
•
•
•
Information About FabricPath Switching
FabricPath switching allows multipath networking at the Layer 2 level. The FabricPath network still delivers packets on a best-effort basis (which is similar to the Classical Ethernet [CE] network), but the FabricPath network can use multiple paths for Layer 2 traffic. In a FabricPath network, you do not need to run the Spanning Tree Protocol (STP) with its blocking ports. Instead, you can use FabricPath across data centers, some of which have only Layer 2 connectivity, with no need for Layer 3 connectivity and IP configurations.
The FabricPath encapsulation facilitates MAC mobility and server virtualization, which means that you can physically move the Layer 2 node but retain the same MAC address and VLAN association for the virtual machine. FabricPath also allows LAN extensions across data centers at Layer 2, which is useful in disaster recovery operations, as well as clustering applications such as databases. Finally, FabricPath is very useful in high-performance, low-latency computing.
With FabricPath, you use the Layer 2 intermediate System-to-Intermediate System (IS-IS) protocol for a single control plane that functions for unicast, broadcast, and multicast packets. There is no need to run the Spanning Tree Protocol (STP); it is a purely Layer 2 domain. This FabricPath Layer 2 IS-IS is a separate process than Layer 3 IS-IS.
Beginning in the Cisco NX-OS Release 5.1 and when you use the F Series module, Cisco supports the conversation-based MAC learning schema. Conversational learning can be applied to both FabricPath (FP) and CE VLANs. Using FabricPath and conversational MAC address learning, the device has to learn far fewer MAC addresses, which results in smaller, more manageable MAC tables.
This section includes the following topics:
•
•
•
•
FabricPath Encapsulation
This section includes the following topics:
FabricPath Headers
When a frame enters the FabricPath network, the system encapsulates the Layer 2 frame with a new FabricPath header. The switch IDs that the system assigns to each FabricPath device as it enters the FabricPath network is used as the outer MAC destination address (ODA) and outer MAC source address (OSA) in the FabricPath header. Figure 2-1 shows the FabricPath header encapsulating the classical Ethernet (CE) frame.
Figure 2-1 FabricPath Frame Encapsulation
The system applies the encapsulation on the ingressing edge port of the FabricPath network and decapsulates the frame on the egressing edge port of the FabricPath network; all the ports within the FabricPath network are FabricPath ports that use only the hierarchical MAC address (see Chapter 3 "FabricPath Interfaces", for more information on configuring FabricPath interfaces). This feature greatly reduces the size of the MAC tables in the core of the FabricPath network.
The system automatically assigns each device in the FabricPath network with a unique switch ID. Optionally, you can configure the switch ID for the FabricPath device.
The outer source address (OSA) is the FabricPath switch ID of the device where the frame ingresses the FabricPath network, and the outer destination address (ODA) is the FabricPath switch ID of the device where the frame egresses the FabricPath network. When the frame egresses the FabricPath network, the FabricPath device strips the FabricPath header, and the original CE frame continues on the CE network. The FabricPath network uses only the OSA and ODA, with the Layer 2 IS-IS protocol transmitting the topology information. Both the FabricPath ODA and OSA are in a standard MAC format (xxxx.xxxx.xxxx).
The FabricPath hierarchical MAC address carries the reserved EtherType 0x8903.
When the frame is originally encapsulated, the system sets the TTL to 16; on each hop through the FabricPath network, each switch decrements the TTL by 1. If the TTL reaches 0, that frame is discarded. This feature prevents the continuation of any loops that may form in the network.
Forwarding Tags (FTags)
The Forwarding Tag (FTag) in the FabricPath header specifies which one of multiple paths that the packet traverses throughout the FabricPath network. The system uses the FTag-specified paths for multidestination packets that enter the FabricPath network. The FTag is a fixed route that the software learns from the topology. The FTag is a 10-bit field with the values from 1 to 1024 (see Chapter 4 "FabricPath Forwarding", for more information on topologies and multiple paths).
This FTag is assigned on the edge port as the frame ingresses the FabricPath network and is honored by all subsequent FabricPath switches in that FabricPath network. Each FTag is unique within one FabricPath topology.
Default IS-IS Behavior with FabricPath
The interfaces in a FabricPath network run only the FabricPath Layer 2 IS-IS protocol; you do not need to run STP in the FabricPath network because FabricPath Layer 2 IS-IS discovers topology information dynamically.
FabricPath Layer 2 IS-IS is a dynamic link-state routing protocol that detects changes in the network topology and calculates loop-free paths to other nodes in the network. Each FabricPath device maintains a link-state database (LSDB) that describes the state of the network; each device updates the status of the links that are adjacent to the device. The FabricPath device sends advertisements and updates to the LSDB through all the existing adjacencies. FabricPath Layer 2 IS-IS protocol packets do not conflict with standard Layer 2 IS-IS packets because the FabricPath packets go to a different Layer 2 destination MAC address than that used by standard IS-IS for IPv4/IPv6 address families.
The system sends hello packets on the FabricPath core ports to form adjacencies. After the system forms IS-IS adjacencies, the FabricPath unicast traffic uses the equal-cost multipathing (ECMP) feature of Layer 2 IS-IS to forward traffic, which provides up to 16 paths for unicast traffic.
Within the FabricPath network, you use a single control plane protocol, Layer 2 IS-IS, for all unicast, multicast, and broadcast traffic. To use the basic FabricPath functionality, you do not need to configure Layer 2 IS-IS because you can use the default topology. The control plane Layer 2 IS-IS comes up and runs automatically when you enable FabricPath on the device.
The loop-free Layer 2 IS-IS protocol builds two trees for the topology. One tree carries unknown unicast, broadcast, and multicast traffic, and the second tree carries load-balanced multicast traffic. The system load balances multicast traffic across both trees (see Chapter 4 "FabricPath Forwarding", for more information about trees and topology).
FabricPath Layer 2 IS-IS is based on the standard IS-IS protocol with the following extensions for the FabricPath environment:
•
•
•
•
•
•
Once the adjacency is established among the devices in the FabricPath network, the system sends update information to all neighbors.
By default, you can run Layer 2 IS-IS with FabricPath with no configuration, However, you can fine-tune some of the Layer 2 IS-IS parameters (see Chapter 5 "Advanced FabricPath Features," for information on configuring optional IS-IS parameters).
Additionally, FabricPath IS-IS helps to ensure that each switch ID in steady-state is unique within the FabricPath network. If FabricPath networks merge, switch IDs might collide. If the IDs are all dynamically assigned, FabricPath IS-IS ensures that this conflict is resolved without affecting any FabricPath traffic in either network,
Conversational MAC Address Learning
Note
In traditional MAC address learning, each host learns the MAC address of every other device on the network. When you configure a VLAN for conversational learning, the associated interfaces learn only those MAC addresses that are actively speaking to them. Not all interfaces have to learn all the MAC addresses on an F Series module, which greatly reduces the size of the MAC address tables.
Beginning with Cisco NX-OS Release 5.1 when you use the F Series module, you can optimize the MAC learning process. Conversational MAC learning is configured per VLAN. All FabricPath VLANs always use conversational learning; you can configure CE VLANs for conversational learning on this module also. (See Chapter 4 "FabricPath Forwarding" for more information on CE and FabricPath VLANs.)
The F Series modules have 16 forwarding engines (FEs), and the MAC learning takes place on only one of these FEs. Each FE performs MAC address learning independently of the other 15 FEs on the module. An interface only maintains a MAC address table for the MACs that ingress or egress through that FE; the interface does not have to maintain the MAC address tables on the other 15 FEs on the module.
Conversational MAC address learning, and the 16 forward engines (FEs) on each F Series module result in MAC address tables that are much smaller for FabricPath.
The MAC address learning modes available on the F Series modules are the traditional learning and conversational learning. The learning mode is configurable and is set by VLAN mode.
The following VLAN modes have the following MAC learning modes:
•
•
With conversational MAC learning, the interface learns only the source MAC address of an ingressing frame if that interface already has the destination MAC address present in the MAC address table. If the source MAC address interface does not already know the destination MAC address, it does not learn that MAC address. Each interface learns only those MAC addresses that are actively speaking with the interface. In this way, conversational MAC learning consists of a three-way handshake.The interface learns the MAC address only if that interface is having a bidirectional conversation with the corresponding interface. Unknown MAC address are forwarded, or flooded, throughout the network.
This combination of conversational MAC address learning and multiple FEs on each F Series module produces greatly reduced MAC address tables on each F Series module.
For CE VLANs, you can configure conversational learning per VLAN on the F Series module using the command-line interface (CLI). CE VLANs use traditional MAC address learning by default.Traditional MAC learning is not supported on FabricPath VLANs with Cisco Release NX-OS 5.1 or higher.
Figure 2-2 shows the allowed FabricPath and CE ports on the M and F Series modules and the allowed FP and CE VLANs.
Figure 2-2 FP and CE VLAN Examples
Switching Using FabricPath
The FabricPath hierarchical MAC address scheme and conversational learning result in much smaller, conversational learning MAC tables within the FabricPath network. Within the FabricPath network, the system uses Layer 2 IS-IS to transmit topology information. The interfaces on the edge of the network, which use conversational MAC address learning, do not have to learn all the MAC addresses in the network (see Figure 2-3).
Figure 2-3 FabricPath Ports Use Only the FabricPath Header to Switch Frames
MAC mobility is expedited using the FabricPath hierarchical MAC addresses. That is, when you want to move a host and keep its same MAC address and VLANs, only the interfaces at the edge of the FabricPath network track this change. Within the FabricPath network, the FabricPath interfaces update their tables with only the outer MAC addresses (ODA and OSA) that have changed from the FabricPath encapsulation.
See Chapter 3 "FabricPath Interfaces", for information on FabricPath interfaces.
The interface on the edge of the FabricPath network encapsulates the original frame inside the FabricPath header. Once the frame reaches the last, or directly connected, FabricPath switch, the egress interface strips the FabricPath header and forwards the frame as a normal CE frame.
The ports on an F Series module at the edge of a FabricPath network can use conversational learning to learn only those MAC addresses that the specified edge port is having a bidirectional conversation with. Every edge interface does not have to learn the MAC address of every other edge interface; it just learns the MAC addresses of the speakers.
As the frame traverses the FabricPath network, all the devices work only with the FabricPath header. So, the FabricPath interfaces work only with the ODAs and OSAs; they do not need to learn the MAC address for any of the CE hosts or other devices attached to the network. The hierarchical MAC addressing provided by the FabricPath headers results in much smaller MAC tables in the FabricPath network, which are proportional to the number of devices in that network. The interfaces in the FabricPath network only need to know how to forward frames to another FabricPath switch so they can forward traffic without requiring large MAC address lookup tables in the core of the network.
The switches in the FabricPath network decrement the TTL in the FabricPath header by 1 at each hop. When the TTL reaches 0, the packet is dropped. This process prevents the continuation of any loops that might form in the network.
Conflict Resolution and Optional FabricPath Tunings
After you enable FabricPath in all devices, the system automatically assigns a random switch ID to each FabricPath device. The switch ID is a 12-bit value that is dynamically assigned to every switch in the FabricPath network, with each switch being a unique value in that FabricPath network. Optionally, you can configure a specific switch ID. If any of the switch IDs in the FabricPath network are not unique, the system provides automatic conflict resolution.
The FabricPath system chooses a random value for the switch ID and sets this value as tentative during a period when the system waits to hear if this value is already in use. If this value is being used by another device in the network, the system begins a conflict resolution process. The switch with the lower system ID keeps the specified value and the other switch gets a new value for its switch ID.
In the case of a single switch joining an existing FabricPath network, the single switch changes the switch ID value rather than any switches in the existing switches in the network changing values. If the specified value is not in use by another device or after the conflict is resolved, the switch ID is marked as confirmed.
Graceful migration provides that there is no traffic disruption if a conflict arises in the resources, such as two switches that temporarily have the same switch ID.
Note
The FabricPath resources timers have default values, but you can also change the timer values. You can tune the device to wait longer or shorter periods to check the conflicts.
Some of the important processes of the FabricPath network are as follows:
•
•
•
•
FabricPath uses the Layer 2 IS-IS protocol to transport the database to all switches in the network. The information is distributed among the FabricPath network devices using an IS-IS TLV. Each switch sends its version of the database that contains information about all the switches.
The system allocates the FabricPath values, guarantees their uniqueness within the FabricPath network, and deletes the value from the database once that resource is no longer needed.
Note
Interoperation between the M Series and the F Series Modules
Beginning with Cisco Release 5.2(1) for the Nexus 7000 Series devices, the MAC learning for the F Series FabricPath-enabled modules when an M Series module is present in the chassis has changed. In this configuration, the FabricPath switches copy all locally learned MAC address entries onto the core port, which is the default learning mode in a chassis that contains both F Series and M Series modules.
When you have an M Series module and an F Series module in the same Cisco Nexus 7000 Series chassis, the FabricPath interface on the F Series modules also learns the MAC addresses that traverse that port from the M Series module. The FabricPath interface provides proxy learning for the MAC addresses on the M Series module in the mixed chassis.
Because M Series modules cannot enable FabricPath, those FabricPath-enabled interfaces that coexist in the same Cisco Nexus 7000 Series chassis do have to learn the MAC addresses of the packets that are traversing the FabricPath-enabled F Series interfaces from the M Series interfaces. The FabricPath interface provides proxy learning for the MAC addresses on the M Series module in the mixed chassis.
See the Cisco Nexus 7000 Series NX-OS Unicast Routing Configuration Guide, Release 5.x, and Cisco Nexus 7000 Series NX-OS Multicast Routing Configuration Guide, Release 5.x, for more information on interoperation between the F1 Series and M Series modules.
High Availability
The FabricPath topologies retain their configuration through ISSU.
See the Cisco Nexus 7000 Series NX-OS High Availability and Redundancy Guide, Release 5.x, for more information on high availability.
Virtual Device Contexts
All the interfaces for one FabricPath network on the same device must be in the same virtual device context (VDC).
You must install the FabricPath feature set before you enable FabricPath on the switch. See the Configuring Feature Set for FabricPath guide for information on installing the FabricPath feature set.
Because of the multiple FEs on the F Series modules, the following port pairs must be in the same VDC:
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
See the Virtual Device Context Configuration Guide, Cisco DCNM for LAN, Release 5.x, for more information on VDCs.
Licensing Requirements for FabricPath Switching
The following table shows the licensing requirements for this feature:
Prerequisites for FabricPath Switching
FabricPath switching has the following prerequisites:
•
•
•
•
•
•
Guidelines and Limitations for FabricPath Switching
FabricPath switching has the following configuration guidelines and limitations:
•
•
•
•
•
•
–
–
•
•
Configuring FabricPath Switching
After you enable FabricPath switching on each device, the encapsulation, default IS-IS, and learning occur automatically.
Note
Instead of using the default values, you can optionally configure the following FabricPath features manually:
•
–
•
–
–
–
(Enabled by default. You might experience traffic drops if the feature is disabled.)–
This section includes the following topics:
•
•
•
•
•
•
•
Enabling the FabricPath Feature Set on the VDC on the device
You must enable the FabricPath feature set before you can access the commands that you use to configure the feature.
Note
BEFORE YOU BEGIN
Ensure that you have installed the Enhanced Layer 2 license.
Ensure that you have installed an F Series module.
SUMMARY STEPS
1.
2.
3.
4.
5.
DETAILED STEPS
This example shows how to enable the FabricPath feature on the VDC:
switch# config terminalswitch(config)# feature-set fabricpathswitch(config)#Disabling the FabricPath Feature Set on the VDC
Note
When you disable the FabricPath functionality, you will not see any of the CLI commands that you need to configure FabricPath. And, the system removes all the FabricPath configurations when you disable the feature set.
Note
BEFORE YOU BEGIN
Ensure that you have installed the Enhanced Layer 2 license.
Ensure that you have installed an F Series module.
SUMMARY STEPS
1.
2.
3.
4.
5.
DETAILED STEPS
This example shows how to disable the FabricPath feature:
switch# config terminalswitch(config)# no feature-set fabricpathswitch(config)#Configuring the MAC Learning Mode for CE VLANs (Optional)
CE VLANs use traditional learning mode by default. However, you can configure CE VLANs on the F Series modules to use conversational MAC address learning.
Note
BEFORE YOU BEGIN
Ensure that you have installed the Enhanced Layer 2 license.
Ensure that you have installed an F Series module.
Ensure that you are working with CE VLANs.
SUMMARY STEPS
1.
2.
3.
4.
5.
DETAILED STEPS
This example shows how to configure conversational MAC address learning on specified CE VLANs on the F Series module:
switch# config terminalswitch(config)# mac address-table learning-mode conversational vlan 1-10switch(config)#Configuring the Switch ID (Optional)
Note
By default, FabricPath assigns each FabricPath device with a unique switch ID after you enable FabricPath on the devices. However, you can manually configure the switch ID.
Note
BEFORE YOU BEGIN
Ensure that you are working on an F Series module.
Ensure that you have enabled the FabricPath feature on all devices.
Ensure that you have installed the Enhanced Layer 2 license.
SUMMARY STEPS
1.
2.
3.
4.
5.
DETAILED STEPS
This example shows how to manually configure a device to have the FabricPath switch ID of 25:
switch# config terminalswitch(config)# fabricpath switch-id 25switch(config)#Configuring the FabricPath Timers (Optional)
Note
You can change the following FabricPath timers:
•
•
•
BEFORE YOU BEGIN
Ensure that you are working on an F Series module.
Ensure that you have enabled the FabricPath feature on all devices.
Ensure that you have installed the Enhanced Layer 2 license.
SUMMARY STEPS
1.
2.
3.
4.
5.
DETAILED STEPS
This example shows how to configure the allocation-delay FabricPath value to 600 seconds:
switch# config terminalswitch(config)# fabricpath timer allocate-delay 600switch(config)#Disabling FabricPath Graceful Merges (Optional)
Note
By default, graceful-merge is enabled; you can disable this aspect of the FabricPath feature.
Note
BEFORE YOU BEGIN
Ensure that you are working on an F Series module.
Ensure that you have enabled the FabricPath feature on all devices.
Ensure that you have installed the Enhanced Layer 2 license.
SUMMARY STEPS
1.
2.
3.
4.
5.
DETAILED STEPS
This example shows how to disable the graceful merge aspect of the FabricPath feature:
switch# config terminalswitch(config)# fabricpath graceful-merge disableswitch(config)#Forcing the Links to Come Up (Optional)
Note
As a one-time event, you can force the FabricPath network links to connect if they are not coming up because of switch ID conflicts or other problems in the network.
Note
Note
BEFORE YOU BEGIN
Ensure that you are working on an F Series module.
Ensure that you have enabled the FabricPath feature on all devices.
Ensure that you have installed the Enhanced Layer 2 license.
SUMMARY STEPS
Step 1
DETAILED STEPS
This example shows how to force the FabricPath network links to come up one time:
switch# fabricpath force link-bringupswitch#Verifying FabricPath Switching
To display FabricPath switching information, perform one of the following tasks:
See Chapter 5 "Advanced FabricPath Features," for more commands that display FabricPath switching functionality.
Displaying and Clearing FabricPath Switching Statistics
Use the following commands to display FabricPath switching statistics:
•
•
•
•
•
See the Cisco Nexus 7000 Series NX-OS Interfaces Command Reference for information on these commands.
FabricPath Switching Example Configuration
After installing the feature set (see Configuring Feature-Set for FabricPath for complete information on installing and enabling the FabricPath feature set), you must enable the FabricPath functionality on all the VDCs that you are using.
Note
To configure FabricPath switching. follow these steps:
Step 1
switch# config terminalswitch(config)# feature-set fabricpathswitch(config)#Step 2
switch(config)# mac address learning-mode conversational vlan 1-10switch(config)# show mac address-table learning-modeswitch(config)# exitStep 3
switch# config terminalswitch(config)# fabricpath switch-id 25switch(config)#Step 4
switch(config)# save running-config startup-configswitch(config)#
Default Settings
Table 2-1 lists the default settings for FabricPath switching parameters.
Feature History for Configuring FabricPath Switching Using the CLI
Table 2-2 lists the release history for this feature.
