Introduction
This document describes how to troubleshoot suspended VLANs due to Light Distribution Box (LDB) allocation fail when using fabric extender (FEX) with M1 parent cards on a Nexus 7000 series switch.
Prerequisites
Requirements
There are no specific requirements for this document.
Components Used
The information in this document is based on these sofware and hardware versions:
- Cisco NX-OS Version 6.2(x)
- Cisco Nexus 7000 Series switch
- Cisco Nexus 2000 Series fabric extender
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Background Information
LDB is a hardware table that stores properties for frames received by the forwarding engine, including trunk to vlan mapping for FEX portchannels. Failed LDB allocation indicates that this databse has been exhausted.
When a FEX module is connected to a N7K-M132XP-12 (non XL) the maximum number of LDB entries is approximately 228k. With the XL module, the limit is approximately 512K.
Total LDB entries are the sum of VLAN ranges defined on each FEX HIF (difference between highest and lowest VLAN defined on a HIF).
For example:
switchport trunk allowed vlan 1-4 = 4 LDB entries
switchport trunk allowed vlan 1-4, 70-80, 800 = 800 LDB entries
switchport trunk allowed vlan 200-800 = 600 LDB entries
switchport trunk allowed vlan 200, 800 = 600 LDB entries
If ports 101/1/1 to 101/1/10 are configured with switchport trunk allowed vlan 200, 800 the total number of LDB entries consumed would be 6000 (10x600).
Access ports consumes 1 LDB entry (the access VLAN defined on the HIF).
Check LDB entries for each module by using this command:
N7K-A# attach mod 1
Attaching to module 1 ...
To exit type 'exit', to abort type '$.'
module-1#
module-1# show system internal eltmc info ldb summary
LDB allocation summary:
Max dynamic ldb entries: 203776
Total number of entries: 199680
Number of free entries: 197
Number of free regions: 2
Number of allocated entries: 199483
Number of allocated regions: 280
Number of fail allocations: 21
[an error occurred while processing this directive]
In this example there have been 21 failed LDB allocations. Due to the hashing mechanism used to hash particular entries to particular parts of the database, it is not necessary for the total number of entries to reach the maximum dynamic LDB entries in order to see failed allocations.
This output displays the number of LDB entries (in hexadecimal) used by each port-channel or interface:
module-1# show system internal eltmc info ldb all
LDB allocation maps :
base size allocation
0xd400 0x1000 Shared
0xe400 0x1000 Shared
0xf400 0x1000 Shared
0x10400 0x1000 Ethernet1/4
0x11400 0x1000 Ethernet1/7
0x12400 0x1 Ethernet179/1/30
0x12401 0x1 port-channel1093
0x12402 0x1 port-channel1564
0x12403 0x1 port-channel1550
0x12404 0x1 port-channel1527
0x12405 0x1 port-channel1546
0x12426 0x1 Ethernet169/1/47
0x12427 0x1 Ethernet169/1/48
0x12428 0x1 Ethernet181/1/33
0x12429 0x1 Ethernet181/1/34
0x1242a 0x1 Ethernet163/1/4
0x1242b 0x1 Ethernet163/1/5
0x1242c 0x506 Ethernet183/1/7
0x12932 0x1 port-channel1096
0x12933 0x1 port-channel1095
0x12934 0x1 port-channel1092
0x12935 0x2c8 port-channel1084
0x12bfd 0x506 Ethernet183/1/8
0x13103 0x2c8 port-channel1086
0x133cb 0x1 port-channel1589
0x133cc 0x1 port-channel1063
0x133cd 0x1 port-channel1654
0x133ce 0x1 port-channel1652
0x133d4 0x1 port-channel1520
0x133d5 0x1 port-channel1560
0x133d6 0x1 port-channel1561
0x133d7 0x506 Ethernet167/1/4
0x138dd 0x506 Ethernet167/1/2
0x13de3 0x403 Ethernet165/1/2
0x141e6 0x403 Ethernet151/1/1
<snip>
[an error occurred while processing this directive]
Note: The two above commands provide incorrect LDB values for N7K-M132XP-12 (non XL) in NX-OS 6.0.3 and 5.2.4.
NX-OS 5.2.5 and 6.1 will correct this.
Problem: VLAN goes into suspended state due to LDB allocation fail on a Nexus 7000 with connected Nexus 2000 fabric extenders.
Symptoms:
1. Error message in logs indicate LDB Allocation Failed
2015 Feb 3 00:01:27.260 N7k1 %ETHPORT-5-IF_SEQ_ERROR: Error ("LDB Allocation Failed") communicating with MTS_SAP_ELTM for opcode MTS_OPC_ETHPM_PORT_LOGICAL_BRINGUP (RID_PORT: port-channel1001)
2015 Feb 3 00:01:27.261 N7k1 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 268,1261-1262,1268 on Interface port-channel1001 are being suspended. (Reason: LDB Allocation Failed)
[an error occurred while processing this directive]
2. Connectivity lost to multiple hosts connected to FEX
3. Output of show interface status err-vlans shows VLAN's suspended due to LDB Failed Allocation
N7kA# show interface status err-vlans
--------------------------------------------------------------------------------
Port Name Err-Vlans Status
--------------------------------------------------------------------------------
Po1001 ***dcn2pclx01a** 268,1261-1262,1268 LDB Allocation
*LOG Failed[an error occurred while processing this directive]
Solution
This is a hardware limitation associated with the linecard, as such this issue is not addressed by software upgrades.
The recommendation is to prune VLAN's from HIF or reduce the VLAN ranges on FEX HIF to reduce the total number of LDB entries.
Each VLAN instance on each interface will consume LDB entries (eg if portchannel 1 has 100 VLAN's defined and four physical ports in the portchannel, the total number of LDB entries consumed will be 400, 100 instances per port).
Feedback