Cable Management
In a highly meshed network such as Clos topology based network fabric, miscabling can be a pragmatic problem leading to painful troubleshooting without sufficient support. The cable management feature calls out for two mechanisms to address the miscabling issues caused due to human errors. The first mechanism is based on the tier-based checks and the second mechanism is based on a user-defined cabling plan.
Tier-Based Miscabling Detection
The figure below shows the properly cabled (in green) and miscabled (in red) links in a hierarchical network designed based on Clos design principle. The cable verification rules based on tier-level is applicable only for Clos topologies. Each box represents a switch in the network.
As shown in the figure, every switch in a stage is associated with the corresponding tier-level number assigned to the stage it is in. That means that all the leaf switches (in the lowest-level Clos stage) are provisioned with a tier-level of 1, the next higher-level stage switches (1st stage of spine switches) are provisioned with a tier-level value of 2, and the next higher-level stage switches (2nd stage of spine switches) are provisioned with a tier-level of 3, so on.
Switch(config)# fabric connectivity tier 2
Note |
You can configure this automatically using the POAP template on DCNM when the switch is designated as a spine or leaf. |
You have to enable features 'cable-management' and 'LLDP' and this command enables tier-based checks on the switch.
Every switch in the network advertises its tier-level number (only if one is configured with), as part of the link layer PDUs (of LLDP) in addition to the chassis-id, and physical port-id to their adjacent peers.
-
Performed only at leaf switches (switches assigned with a tier-level of 1) -
The received tier level from adjacent link partners must be 2. There will be an exception to this rule. If the receiving link is vPC peer link (detected automatically via LLDP), a remote tier-level of 1 will be allowed.
-
-
Performed only at spine switches (switches assigned with tier-level > 1) -
The received tier level from adjacent link partners must be either (my_tier_level + 1) or (my_tier_level - 1), where my_tier_level is the tier level assigned to the spine switch performing the check.
-
If any cabling inconsistency is detected while validating the remote link information from the adjacent switches, then the miscabling action (error disabling the port by default) is performed and a cabling mismatch error is logged. The logged error includes necessary information about the involved local chassis-id, local port-id, local tier-level, and remote switch details such as peer chassis-id, peer port-id, and peer tier-level.
The following example shows the fabric connectivity neighbors with tier details.
Switch# show fabric connectivity neighbors all
--------------------------------------------------------------------------------
Local System:
Device Tier Config: Enabled Device Tier Level: 1
Tier-mismatch Retry Config: Disabled Tier-mismatch Retry Timeout: 0
Cable Plan Check: Enabled
DeviceID: poap-integ-leaf1 ChassisID: 0003.0111.0008
--------------------------------------------------------------------------------
Codes: (Ok) Normal, (ErrT) Tier error , (ErrC) Cabling Plan error,
(Ok/VPC) VPC leaf-to-leaf connection, (S) Stale entry
(Unkn) Unknown Tier, (Enp) Plan Entry not present
Neighbor Table:
--------------------------------------------------------------------------------
Local DeviceID PortID Tier Cable-Plan Status
Intf
Eth2/1 spine0 Eth2/1 2 spine2,Eth2/1 ErrC
Eth2/6 spine5 Eth2/2 2 spine5,Eth2/2 Ok
Eth2/5 spine4 Eth2/2 2 spine0,Eth2/2 ErrC
Eth2/3 spine6 Eth2/2 2 Enp Ok
Eth2/4 spine3 Eth2/2 2 spine7,Eth2/1 ErrC
poap-integ-spine0# show fabric connectivity neighbors errors
--------------------------------------------------------------------------------
Local System:
Device Tier Config: Enabled Device Tier Level: 2
Tier-mismatch Retry Config: Disabled Tier-mismatch Retry Timeout: 0
Cable Plan Check: Disabled
DeviceID: poap-integ-spine0 ChassisID: 0003.0210.0008
--------------------------------------------------------------------------------
Codes: (Ok) Normal, (ErrT) Tier error , (ErrC) Cabling Plan error,
(Ok/VPC) VPC leaf-to-leaf connection, (S) Stale entry
(Unkn) Unknown Tier
Neighbors with Miscabling Warning/Error:
--------------------------------------------------------------------------------
Local DeviceID ChassisID PortID Tier Status
Interface
Ethernet2/1 poap-integ-leaf0 0003.0110.0001 Eth2/1 2 ErrT,S
Ethernet2/2 poap-integ-leaf1 0003.0111.0001 Eth2/1 2 ErrT,S
Total entries displayed: 2
In case switch is configured with a tier level that is connected to another switch that does not support this feature and hence does not send tier level information, then the link will not be checked for miscabling.
Overview of Cable-Plan Miscabling Detection
Cable-plans also rely on LLDP to exchange information, for example switch ID, port number, between two ends of a cable connection. This information is carried in vendor specific LLDP TLVs.
You must first import a valid cable-plan to the switch and check against the information in these received TLVs. Cable-plans are XML files that fully or partially describe the topology of the data center.
Once a plan is imported, checks will happen to make sure that the imported plan matches the received TLV information. If it matches nothing will happen, however if there is a mismatch then the miscabling action will be taken on the port in question (err-disable by default). The user can change the actions taken on a miscabling event.
Administrators will then be warned via SysLog of any such miscablings, and they will take the appropriate action to fix the problem (in most cases like above, they will simply need to fix the miscabling to the correct configuration as described in the cable-plan, and clear the err-disable).
Switch(config)# feature lldp
Switch(config)# feature cable-management
Cable-Plan XML Schema
The following schema of cable-plan XML is for a data center with two spine switches, and three leaf switches:
<?xml version="1.0" encoding="UTF-8"?>
<CISCO_NETWORK_TYPES version="1.0" xmlns="http://www.cisco.com/cableplan/Schema2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.cisco.com/cableplan/Schema2 nxos-cable-plan-schema.xsd ">
<DATA_CENTER networkLocation="san-jose" idFormat="hostname">
<CHASSIS_INFO sourceChassis="spine1" type="n7k">
<LINK_INFO sourcePort="Eth2/1" destChassis="leaf1" destPort="Eth2/1"/>
<LINK_INFO sourcePort="Eth2/2" destChassis="leaf2" destPort="Eth2/1"/>
<LINK_INFO sourcePort="Eth2/3" destChassis="leaf3" destPort="Eth2/1"/>
</CHASSIS_INFO>
<CHASSIS_INFO sourceChassis="spine2" type="n7k">
<LINK_INFO sourcePort="Eth1/1" destChassis="leaf1" destPort="Eth1/2"/>
<LINK_INFO sourcePort="Eth1/2" destChassis="leaf2" destPort="Eth1/2"/>
<LINK_INFO sourcePort="Eth1/3" destChassis="leaf3" destPort="Eth1/2"/>
</CHASSIS_INFO>
</DATA_CENTER>
</CISCO_NETWORK_TYPES>
Following is the breakdown of each line and the XML tags and attributes:
<?xml version="1.0" encoding="UTF-8"?>
<CISCO_NETWORK_TYPES version="1.0" xmlns="http://www.cisco.com/cableplan/Schema2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.cisco.com/cableplan/Schema2 nxos-cable-plan-schema.xsd ">
-
CISCO_NETWORK_TYPES - Parent tag for the entire XML cable-plan. The entire cable-plan must be within this tag.
<DATA_CENTER networkLocation="san-jose" idFormat="hostname">
-
DATA_CENTER - Houses all the information for each chassis in the plan.
-
networkLocation - States where the data center is for clerical purposes. No affect on miscabling.
-
idFormat - States the format in which IDs will be presented in subsequent entries. As of 12/2/2014, the only supported format is "hostname". All cable-plans that do not use “hostname” as the ID format will be rejected.
<CHASSIS_INFO sourceChassis="spine1" type="n7k">
-
CHASSIS_INFO - Describes a single chassis. All interfaces that belong to sourceChassis that administrators want to have checked by cable-plan must be within one, and only one of these tags.
-
sourceChassis - The chassis that all subsequent interfaces (described in LINK_INFO tags) belongs to. In this case, all interfaces within the CHASSIS_INFO tags are said to belong to the chassis called "spine1".
-
type - The type of chassis. Only the Cisco Nexus Switches are supported, so this attribute must read "n#k", such as "n7k" (non-case sensitive). All cable-plans that do not adhere to the "n#k" format will be rejected.
<LINK_INFO sourcePort="Eth2/1" destChassis="leaf1" destPort="Eth2/1"/>
<LINK_INFO sourcePort="Eth2/2" destChassis="leaf2" destPort="Eth2/1"/>
<LINK_INFO sourcePort="Eth2/3" destChassis="leaf3" destPort="Eth2/1"/>
-
LINK_INFO - Fully describes an interface connection from sourceChassis (as mentioned previously) to destChassis. In this case, we are stating that spine1's port eth2/1 is connected to leaf1's eth2/1, spine1's eth2/2 is connected to leaf2's eth2/1, and so on.
-
sourcePort - The sourcePort is the port on the sourceChassis (as mentioned previously). Source ports should be unique per chassis (that is, spine1 should not describe multiple connections coming from port eth2/1). While an import will not fail on cable-plans with non-unique ports, a warning will be printed and only the first entry will be read and checked by cable-plan.
-
destChassis - The destination chassis that the sourceChassis is connected to.
-
destPort - The port on the destination chassis that has the connection. Like sourcePort, this should be unique to destChassis.
</CHASSIS_INFO>
<CHASSIS_INFO sourceChassis="spine2" type="n7k">
<LINK_INFO sourcePort="Eth1/1" destChassis="leaf1" destPort="Eth1/2"/>
<LINK_INFO sourcePort="Eth1/2" destChassis="leaf2" destPort="Eth1/2"/>
<LINK_INFO sourcePort="Eth1/3" destChassis="leaf3" destPort="Eth1/2"/>
</CHASSIS_INFO>
Here you close the CHASSIS_INFO tag and we completely describe the connections on all interfaces that spine1 can have. You can open another tag to describe interfaces in the same way, on a different router that you want it checked. There is no limit to the number of unique CHASSIS_INFO tags you can describe in a cable-plan.
</DATA_CENTER>
</CISCO_NETWORK_TYPES>
Finish closing the tags to complete our XML file. At this point we have a valid XML cable-plan that completely describes a data center containing the following switches: spine1, spine2, leaf1, leaf2, leaf3.
Note that you need not repeat connections from each chassis' perspective. Since you describe that sourceChassis spine2 is connected to destChassis leaf1 via Eth1/1 - Eth1/2, you do not need an entry describing sourceChassis leaf1 and destChassis spine2. While the plan will not fail if you include this redundant information, it will be ignored and add unnecessary length to the plan.
Also ensure that other interfaces may be present that were not described in the cable-plan. Consider a third spine, spine3, connected to all the leaf nodes. Since this is not described in the cable-plan, LLDP TLVs with this information will not be checked and only an indicator warning administrators of its absence from the cable-plan will be noted. As long as the un-described spine is not interfering with ports already described in the cable-plan, then no actions will occur with the missing ports (that is, the new spine3 cannot be connected to leaf1's eth1/2, since you explicitly state that leaf1's eth1/2 is connected to spine2's eth1/1).
Cable-Plan Specific Commands
leaf1# fabric connectivity cable-plan import [means]:[location] {update} {verbose}
-
means - Describes the way you may want to import. Currently the only local means for importing is via bootflash and USB. If you would like to import via remote location, FTP, SCP, SFTP and TFTP protocols are supported for this option.
-
location - The location of the file. For instance "scp://calinfor@171.77.77.7/nobackup/test.xml" will scp the file located at /nobackup/test.xml on 171.77.77.7.
-
update - (Optional) To add the entries in the importing cable-plan to the cable-plan entries that have already been imported in a previous import command. If there is no cable-plan already imported then this keyword has no effect. If this keyword is not specified during the import command, then the previously imported cable-plan (if there is one) will be deleted and it will be replaced with the new cable-plan specified in the import command.
-
verbose - (Optional) To print all errors associated with the import. If this option is not specified, only a one line description of success or failure will be printed. If the option is specified, detailed information on why the import failed will be printed to the console.
leaf1# fabric connectivity cable-plan export[cable-plan location]
The export CLI will take a previously imported cable-plan and write it to a file. Since cable-plans are stored in memory after they are imported, users may have lost or altered the previous XML file containing the cable-plan. This allows these users to export the current cable-plan stored in memory and regain their XML file.
leaf1# fabric connectivity cable-plan generate [cable-plan location] [name]
Auto generate a cable-plan based on the current LLDP neighbors. This creates a valid cable-plan based on the local switch's perspective. Outputs a time-stamped file in the bootflash when done. Useful for allowing users to quickly create a cable-plan out of current topology without writing it by hand. Users can then either import this cable-plan to enforce the current topology or users can use it as a template of sorts to create a more complex cable-plan. Note, if there are no LLDP neighbors present then only a template that will need to be edited will be created.
leaf1# configure terminal
leaf1(config)# [no] fabric connectivity cable-plan enforce
leaf1(config)# exit
leaf1# show fabric connectivity cable-plan
This command prints the information in the currently stored cable-plan. If no information is stored this will be blank. This will print the local and remote Chassis and Port IDs, and the cabling status of this connection. This is just a printing of the cable-plan and the status of each link.
Switch Links Cable State
spine1# show fabric connectivity neighbors {errors | interface | tier}
-
errors - (Optional) Displays only interfaces with errors
-
interface - (Optional) Displays information pertaining to the specified interface
-
tier - (Optional) Displays only interfaces with the specified tier level
-
default (blank) - Displays all interfaces that have received an LLDP TLV since either tier or cable-plan checks have been toggled on. Unlike "show fabric connectivity cable-plan", the show neighbor command will show the actual data being received by the switch via LLDP TLVs
Local Chassis and Port IDs, remote Chassis and Port IDs, the tier levels of the remote Chassis, the expected cable-plan entry and all the status will be displayed.
-
Ok - Everything working as intended, the check succeeded
-
Unkn - Unable to determine the status, most likely because cable-check and tier-check has been disabled
-
ErrC - The port has been err-disabled due to a mismatch (that is the TLV received did not match the specific entry in the cable-plan)
-
ErrT - A tier level mismatch error
-
S - May be specified at the end of a status. This means the port is "stale" in that the switch has received a purge event from the remote peer. This usually happens when the port has gone err-disabled or when we have not seen LLDP events from this link in a long time.
Examples
switch# show fabric connectivity neighbors
-------------------------------------------------------------------------------
Local System:
Device Tier Config: Disabled Device Tier Level: Unknown
Mismatch Delay Config: Disabled Mismatch Delay Timeout: 0
Cable-Plan Enforce: Enabled
DeviceID: switch ChassisID: f866.f2d6.37c4
-------------------------------------------------------------------------------
Codes: (Ok) Normal, (ErrT) Tier error , (ErrC) Cable-Plan error,
(V) VPC Peer connection, (S) Stale entry, (Unkn) Unknown,
(Enp) Entry not present in Cable-Plan, (Tl) Tier level
Neighbor Table:
-------------------------------------------------------------------------------
Local DeviceID PortID Tl Cable-Plan Status
Intf Entry
Eth1/13 switch Eth1/14 Unkn switch2,Eth1/14 ErrC,S
Eth1/14 switch Eth1/13 Unkn switch9,Eth1/13 ErrC,S
Spine11# show interface Eth2/1
Ethernet2/1 is down (errDisabled)
……
……
switch# show interface status err-disabled
----------------------------------------------------------------------------
Port Name Status Reason
----------------------------------------------------------------------------
Eth2/1 -- Miscabled Miscabled
Miscabling Error Disable Action Control
By default any tier based or cable-plan based mismatches trigger err disabling of the ports. After the miscabling or topological errors are corrected, the miscabled ports can be brought back live either by manually removing them out of error disabled state or automatically through a configuration CLI. The error disabling action can also be turned off or delayed. The following knobs can control these optional behaviors.
-
To remove the default errdisable on error action: Switch(config)# no errdisable detect cause miscabled
-
Miscabled error disabling action can be delayed for specified time (timeout) using the following configurations. Here action will be taken after <timeout> seconds have passed: Switch(config)# fabric connectivity mismatch action delay <timeout>
-
Auto recovery of the miscabled ports out of error disabled state can be enabled using the following command. The below command will attempt to recover all miscabled ports at some set interval. Switch(config)# errdisable recovery cause miscabled
-
To change the interval at which the errdisable recovery command attempt to recovery, the following configure command can be used: Switch(config)# errdisable recovery interval <time>
-
To clear a single entry or all entries from the Clos neighbor cache immediately, the following clear command can be used: Switch# clear fabric connectivity neighbors [interface | stale]
This CLI will allow the you to clear a single entry or all error entries or all entries from the Clos neighbor cache immediately. You should manually clear an already secured port in the neighbor cache if recabling is desired to remove old stale entry immediately. The entry will be automatically removed after the hold time otherwise. If you have multiple errors that you just fixed, performing a 'clear neighbors' on the affected switches is the easiest way to bring up all interfaces again if 'error recovery' is not enabled. Also, if you have a switch that was previously in the network, but has gone inactive (may be you removed it or it was taken down), performing a clear command is the only way to completely remove it from the neighbor cache (so it stops showing up in the show neighbors command mentioned above).
-
To clear the cable-plan from this switch, the following command can be used: Switch# clear fabric connectivity cable-plan
Clears the current cable-plan. If you had previously written a cable-plan to the startup config and you want this clearing to persist, use the copy running-config startup-config command.