Troubleshoot mVPN Profile 14
Available Languages
Contents
Introduction
This document describes how to troubleshoot Multicast VPN (mVPN) profile 14 within the Cisco IOS® XR.
Prerequisites
You must have knowledge of Multicast profile14 and its behaviour.
Requirements
Cisco recommends that MVPN BGP auto-discovery must be configured.
Also, it is recommended that you know these topics:
- Profiles – 14
- Core Tree – mLDP
- Service – IPv4 VPN
- C-Overlay Signaling – BGP
- Auto-Discovery – BGP
- Model – Partition MDT P2MP with BGP AD
- Encapsulation/forwarding – MPLS
Components Used
This document is not restricted to specific software and hardware versions.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
mLDP Concepts
Core Tree - mLDP
- Multipoint LDP = LDP + extensions
- P2MP tree -Receiver driven –Root learned from routing
- MP2MP tree –Configuration driven –Root configured
- Protection by MPLS TE or Loop-Free Alternate (LFA)
- No PHP –Top label identifies the tree
- Replication of mcast on the core routers
- FEC elements hold: Type of tree + Root + Opaque value: (S, G), MDT number, LSP ID
mLDP
MP2MP LSP vs P2MP LSP
We have two ways to send traffic between the PEs.
1. MP2MP - a static default MDT is created among all PEs for that particular user, this works as if all PEs for that user were connected in the same LAN segment.
2. P2MP - when a configurable threshold value is reached, the sender PE can create a new LSP so only the PEs that are involved in that communication see the traffic, this LSP is created to avoid unnecessary waste of bandwidth sending traffic to PEs that did not join the stream. This LSP only creates for an (S, G) entry in the user network, if you send traffic that exceeds the configured threshold through an (*, G) entry, no new P2MP is created (the concept of PHP can not be used in this scenario when the leaf acts as a root, otherwise it does not know which user that traffic belongs to).
mLDP work with a concept as root, it is somewhat similar to a Rendezvous Point in the PIM-SM scenario, but remember, in mLDP, a PIM-free MPLS core is mentioned, and PIM is only enabled in the user-facing interfaces.
In the MP2MP tree, the roots are responsible to receive the traffic from a leaf and replicate it to all directly connected neighbors.
Every PE can have more than one root configured, they choose its active root based on metric, if the metric is the same the highest IP address is elected as the root for that particular PE.
In the P2MP tree, the root is always the device sending the traffic (In this document the term root is used more to reference the MP2MP tree unless specified).
P2MP and MP2MP
Recursive Forwarding
Recursive forwarding means mLDP control plane programs just the (Downstream ID, Outgoing label) to the forwarding plane. The Downstream ID is usually the LDP router-id of the downstream node. The forwarding plane recursively resolves the reachability to the downstream device and figures out the actual outgoing interface and the nexthop. Recursive forwarding enables features like ECMP/LAG load balancing, LFA, RSVP-TE protection and so on. Unfortunately, not all platforms support this.
With Non-recursive forwarding, mLDP determines the outgoing interface and NH to the downstream router and explicitly programs (Intf, NH, Outgoing label) in the forwarding plane.
Configuration
This is the configuration required for mLDP:
mpls ldp
log
hello-adjacency
neighbor
nsr
graceful-restart
!
igp sync delay on-session-up 25
mldp
address-family ipv4
make-before-break delay 60 60
forwarding recursive
recursive-fec
This concept is explained in the troubleshoot section. This configuration is required in the MID and BUD router.
Head/Root = This router has a Multicast source connected
Tail/Leaf = This router has Multicast receivers
Bud = This router is in transit and has some receivers connected
Mid = This router is in transit
Label Switch Multicast (LSM)/NG mVPN
- Leverage MPLS encapsulation > Share control and data plane with unicast
- Leverage new core tree protocols > mLDP, P2MP TE, BIER
- Fast ReRoute (FRR)
- Leverage proven functionality: Default and Data MDT
- More flexible designs per VPN
- Manageability: no need to track Multicast Groups per VPN/Default/Data MDT
BGP
AF IPv4 mVPN
AF IPv4 mVPN is added to be able to support the NG profiles. It is referred to as the MCAST-VPN NLRI.
There are two main functions of this address family:
- Auto-Discovery (AD)
- Customer Signaling in overlay
Auto-Discovery means that the PE routers learn of each other in an automatic manner. PE routers learn which other PE routers are enabled for multicast for a VPN. This avoids the need to manually configure this on every PE router.
The user signaling in overlay is the signaling of (*, G) or (S, G) information, but now in BGP instead of PIM.
Look at Figure A to see the overview of BGP AD IPv4 MVPN.
Figure A - BGP IPv4 AD MVPN
There are two important pieces of information carried out by BGP in the AF IPv4 MVPN. The first part of information sits in the NLRI itself. There is a number from 1-7 that represents the Route Type. Each Route Type has a specific function. The first 5 are used for BGP-AD. Note that Route Type 3 is actually also used for Data MDT (S-PMSI) signaling.
The last two types, 6 and 7, are used for C-multicast signaling. Next to the Route Type, there is also specific Route Type information embedded which pertains to the RD (if VPN context is used), specific C-multicast info (Source, Group) and the IP address (PE) of the originator.
The second part is the PMSI Tunnel Attribute (PTA). This PTA holds information that pertains to the core tree tunnels that are used. The tunnel type indicates a number of which core tree protocol is used to build the tunnel. The tunnel identifier is a set of fields which uniquely describes the tunnel by specific core tree protocol fields.
BGP mVPN Route Types
|
Route Type |
Name |
AD or C-mcast |
Usage |
|
1 |
Intra-AS I-PMSI A-D route |
AD |
AD Advertise member PE |
|
2 |
Inter-AS I-PMSI A-D route |
AD |
Same as type 1, but for inter-AS |
|
3 |
S-PMSI A-D route |
AD |
Advertised by Source PE to signal Data MDT |
|
4 |
Leaf A-D route |
AD |
Advertised by Receiver-PE as a response to type 3 route with leaf info required flag set |
|
5 |
Source Active A-D route |
AD |
Advertised by Source PE with an active source to facilitate SPT switchover |
|
6 |
Shared Tree Join route |
C-mcast |
Advertise (*, G) by Receiver PE |
|
7 |
Source Tree Join route |
C-mcast |
Advertise (S, G) by Receiver PE |
Route Types in AF IPv4 MVPN
This section provides an overview of the different route types and their purpose.
Intra-AS I-PMSI A-D Route (Type 1)
Purpose: To advertise the PE as to participate in MVPN for a certain VPN. This is BGP-AD.
Encoding:
+-----------------------------------+
| RD (8 octets) |
+-----------------------------------+
| Originating Router's IP Addr |
+-----------------------------------+
Inter-AS I-PMSI A-D Route (Type 2)
Purpose
Advertised by an ASBR, so that Inter-Autonomous mVPN can be signaled.
Encoding
+-----------------------------------+
| RD (8 octets) |
+-----------------------------------+
| Source AS (4 octets) |
+-----------------------------------+
S-PMSI A-D Route (Type 3)
Purpose
1. Advertised by the PE router, in order for it to signal to other PE routers that it is the Ingress PE for a certain C-(S, G) and that it switches over to a Data MDT. The core tree information needed to receive PE routers joins this stream is encoded in the PTA.
2. Advertised by a PE router, it indicates that it wants to participate in a Partitioned MDT model. In that case, the Source and Group fields are wildcard (*,*) elements.
Encoding
+-----------------------------------+
| RD (8 octets) |
+-----------------------------------+
| Multicast Source Length (1 octet) |
+-----------------------------------+
| Multicast Source (variable) |
+-----------------------------------+
| Multicast Group Length (1 octet) |
+-----------------------------------+
| Multicast Group (variable) |
+-----------------------------------+
| Originating Router's IP Addr |
+-----------------------------------+
The Source and Group fields relate to C-multicast state.
Leaf A-D Route (Type 4)
Purpose
This message is sent in response to obtain an Inter-AS I-PMSI A-D route or a S-PMSI A-D route, only if the Leaf Information (LI) Required bit is set.
Encoding
+-----------------------------------+
| Route Key (variable) |
+-----------------------------------+
| Originating Router's IP Addr |
+-----------------------------------+
The Route Key is set to the NLRI of the received route with the Leaf Information Required bit set.
If the Leaf A-D is generated that corresponds to inter-as I-PMSI route, then its NLRI looks like this:
+-----------------------------------+
| Type (1 octet): 4 |
+-----------------------------------+
| Length (1 octet) |
+-----------------------------------+
| Type (1 octet): 2 |
+-----------------------------------+
| Length (1 octet) |
+-----------------------------------+
| RD (8 octet) |
+-----------------------------------+
| Source AS (4 octet) |
+-----------------------------------+
| Originating Router's IP Addr |
+-----------------------------------+
If the Leaf A-D route is generated that corresponds to the S-PMSI route, then its NLRI looks like this:
+-----------------------------------+
| Type (1 octet): 4 |
+-----------------------------------+
| Length (1 octet) |
+-----------------------------------+
| Type (1 octet): 3 |
+-----------------------------------+
| Length (1 octet) |
+-----------------------------------+
| RD (8 octet) |
+-----------------------------------+
| Multicast Source Length (1 octet) |
+-----------------------------------+
| Multicast Source (Variable) |
+-----------------------------------+
| Multicast Group Length (1 octet) |
+-----------------------------------+
| Multicast Group (Variable) |
+-----------------------------------+
| Origin Routers IP addr for SPMSI |
+-----------------------------------+
|Origin Router's IP Addr for Leaf-ad|
+-----------------------------------+
Source Active A-D Route (Type 5)
Purpose
The usage of the Source Active Route is related to PIM Sparse Mode in overlay signaling. Refer to that section for more information.
Encoding
+-----------------------------------+
| RD (8 octets) |
+-----------------------------------+
| Multicast Source Length (1 octet) |
+-----------------------------------+
| Multicast Source (variable) |
+-----------------------------------+
| Multicast Group Length (1 octet) |
+-----------------------------------+
| Multicast Group (variable) |
+-----------------------------------+
Shared Tree Join Route (Type 6)
Purpose
The BGP Update or Withdraw message is the equivalent of a PIM (*, G) Join or Prune message.
Encoding
Type 6 & 7 Routes have the same NLRI encoding.
+-----------------------------------+
| RD (8 octets) |
+-----------------------------------+
| Source AS (4 octets) |
+-----------------------------------+
| Multicast Source Length (1 octet) |
+-----------------------------------+
| Multicast Source (variable) |
+-----------------------------------+
| Multicast Group Length (1 octet) |
+-----------------------------------+
| Multicast Group (variable) |
+-----------------------------------+
Source Tree Join Route (Type 7)
Purpose
The BGP Update or Withdraw message is the equivalent of a PIM (S,G) Join or Prune message.
Encoding
Type 6 & 7 Routes have the same NLRI encoding.
+-----------------------------------+
| RD (8 octets) |
+-----------------------------------+
| Source AS (4 octets) |
+-----------------------------------+
| Multicast Source Length (1 octet) |
+-----------------------------------+
| Multicast Source (variable) |
+-----------------------------------+
| Multicast Group Length (1 octet) |
+-----------------------------------+
| Multicast Group (variable) |
+-----------------------------------+
AF IPv4 MVPN NLRI Encoding
This section lists the encoding of the network part for IPv4 MVPN routes per Route Type. Note that the RD is not included for Route Types 1, 2, and 3. The network part is listed with the number in () and it represents the number of octets for each part. The first number is always the Route Type (1-7). The Prefix-length part is listed in the number of bits.
Intra-AS A_D Route
Network: [1] [PE address (4)]
Prefix-length: 40
Inter-as A-D route
Network: [2] [AS (4)]
Prefix-length: 40
S-PMSI A-D route
Network: [3] [srclen (1)] [src (4)] [1 grplen] [grp (4)] [PE address (4)]
Prefix-length: 120
Leaf A-D route
Network: [4] [2] [RD] (8)] [AS (4)] [ PE address (4)]
Prefix-length: 144
or
Network: [4] [3] [RD (8)] [srclen (1)] [src (4)] [grplen (1)] [grp (4)] [PE address (4)] [PE address (4)]
Prefix-length: 224
Souce-Active A-D route
Network: [5] [srclen (1)] [src (4)] [grplen (1)] [grp (4)]
Prefix-length: 88
C-Multicast Route - Shared-tree Join
Network: [6] [RD (8)] [AS (4)] [srclen (1)] [src (4)] [grplen (1)] [grp (4)]
Prefix-length: 184
C-Multicast Route Source-tree Join
Network: [7] [RD (8)] [AS (4)] [srclen (1)] [src (4)] [grplen (1)] [4 grp]
Prefix-length: 184
The network and prefix length for each route type can be found in the output of show bgp ipv4 mvpn command in Cisco IOS or Cisco IOS XR. Here is an example of the output for this command from Cisco IOS XR:
DRP/0/3/CPU1:Router#show bgp ipv4 mvpn vrf one
Route Distinguisher: 1:2 (default for vrf one)
*>i[1][10.1.100.1]/40 10.1.100.1 100 0 i
*>i [3][1:1][*][*][10.100.1.1]/14
10.100.1.1 0 100 0 ?
*>i[3][3][1:2][32][10.2.2.9][32][225.1.1.1][10.1.100.2][10.1.100.5]/224
10.1.100.5 100 0 i
*>i[4][3][1:2][32][10.2.100.9][32][225.1.1.1][10.1.100.2][10.1.100.1]/224
10.1.100.1 100 0 i
*>i[5][32][10.2.2.9][32][225.1.1.1]/88
10.1.100.2 100 0 i
*>i[6][1:5][1][32][10.2.100.12][32][225.1.1.1]/184
10.1.100.1 100 0 i
*>i[7][1:2][1][32][10.2.2.9][32][232.1.1.1]/184
10.1.100.1 100 0 i
Configuration
Refer to this document to know how to configure mVPN with Cisco IOS XR.
Topology
Partitioned MDT - mLDP P2MP - BGP-AD - BGP C-Mcast Signaling - SSM
Multicast Profile 14 topology
Platform: ASR9K
Steps to Troubleshoot
1. Check if there are any system-level alarms on the reported node (Source PE/Receiver PE/MID Router).
Show pfm location all
Show logging (Check for critical or major logs)
admin show alarm
2. Check the MPLS core connectivity between the Headend router and Tail End Router.
3. Check the local connectivity of the Multicast Source/Receiver to connected PE routers.
4. Check the End-to-End connectivity between Multicast Source and receiver.
5. Check the configuration as per the configuration guide mentioned in the previously mentioned document.
6. Basic verification from system health check:
- Check if any system alarm (
show pfm location all) - Check memory and CPU
show memory summary location all
show shmwin summary location all
show processes memory detail location all
show process cpu location <> - Check interface and QoS Stats
- Check show logging, if there is any critical notification or errors
- Check for any configuration changes due to which impact or issue observed
- Platform drops
System Drops Verify Commands
Follow these steps to troubleshoot multicast profile 14.
Example: group: 232.11.11.11 source 10.1.0.2
Tail End/Receiver PE Node
Towards Multicast Receiver:
IGMP Join Check
RP/0/RSP0/CPU0:AG2-1#show igmp vrf MLDP-14 groups | in "232.11.11.11"
Fri Mar 12 02:43:14.720 EST
232.11.11.11 Bundle-Ether5000.1 15:26:11 never 10.1.0.2
232.11.11.11 Bundle-Ether5000.2 15:26:03 not used 10.1.1.2
RP/0/RSP0/CPU0:AG2-1#sh igmp vrf MLDP-14 groups 232.11.11.11 detail
Fri Mar 12 02:44:19.393 EST
Interface: Bundle-Ether5000.1
Group: 232.11.11.11
Uptime: 15:27:15 >>>>>>>> Check the IGMP join timestamp
Router mode: INCLUDE
Host mode: INCLUDE
Last reporter: 10.1.0.2
Suppress: 0
Group source list:
Source Address Uptime Expires Fwd Flags
10.1.0.2 15:27:15 00:03:46 Yes Remote Local e
Interface: Bundle-Ether5000.2
Group: 232.11.11.11 >>> Multicast Group
Uptime: 15:27:07
Router mode: INCLUDE
Host mode: INCLUDE
Last reporter: 10.1.1.2 >>> Receiver IP
Suppress: 0
Group source list:
Source Address Uptime Expires Fwd Flags
10.1.0.2 15:27:07 00:03:59 Yes Remote 4 >>> Source is 10.1.0.2 & uptime
RP/0/RSP0/CPU0:AG2-1#
Check the interface status with bit rates and errors:
RP/0/RSP0/CPU0:AG2-1#sh int be5000
Fri Mar 12 02:45:32.033 EST
Bundle-Ether5000 is up, line protocol is up
Interface state transitions: 1
Hardware is Aggregated Ethernet interface(s), address is 0026.9813.b986
Internet address is Unknown
MTU 1514 bytes, BW 10000000 Kbit (Max: 10000000 Kbit)
reliability 255/255, txload 20/255, rxload 0/255
Encapsulation ARPA,
Full-duplex, 10000Mb/s
loopback not set,
Last link flapped 15:28:35
No. of members in this bundle: 1
TenGigE0/0/0/17 Full-duplex 10000Mb/s Active
Last input 00:00:00, output 00:00:00
Last clearing of "show interface" counters never
30 second input rate 1000 bits/sec, 2 packets/sec
30 second output rate 793646000 bits/sec, 200031 packets/sec
118212 packets input, 8341734 bytes, 0 total input drops
0 drops for unrecognized upper-level protocol
Received 6 broadcast packets, 6819 multicast packets
0 runts, 0 giants, 0 throttles, 0 parity
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
2857201595 packets output, 1416646430161 bytes, 0 total output drops
Output 242 broadcast packets, 2857201353 multicast packets
0 output errors, 0 underruns, 0 applique, 0 resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
Check if any LPTS drops:
show lpts pifib hardware static-police location <> | i "IGMP|PIM|Flow"
RP/0/RSP0/CPU0:AG2-1#show lpts pifib hardware static-police location 0/0/CPU0 | i "IGMP|PIM|Flow"
Fri Mar 12 02:45:56.259 EST
Punt Reason SID Flow Rate Burst Rate Accepted Dropped Destination
PUNT_IGMP_SNOOP NETIO_MED 4000 2000 0 0 0x0030 (0/RSP0/CPU0)
PUNT_IFIB_IGMP_OPT NETIO_LOW 2000 500 0 0 Local
PUNT_IFIB_PIM_OPT NETIO_LOW 2000 500 36 0 Local
PIM Entry Check:
RP/0/RSP0/CPU0:AG2-1#sh pim vrf MLDP-14 topology 232.11.11.11
Fri Mar 12 02:47:57.462 EST
IP PIM Multicast Topology Table
Entry state: (*/S,G)[RPT/SPT] Protocol Uptime Info
Entry flags: KAT - Keep Alive Timer, AA - Assume Alive, PA - Probe Alive
RA - Really Alive, IA - Inherit Alive, LH - Last Hop
DSS - Don't Signal Sources, RR - Register Received
SR - Sending Registers, SNR - Sending Null Registers
E - MSDP External, EX - Extranet
MFA - Mofrr Active, MFP - Mofrr Primary, MFB - Mofrr Backup
DCC - Don't Check Connected, ME - MDT Encap, MD - MDT Decap
MT - Crossed Data MDT threshold, MA - Data MDT Assigned
SAJ - BGP Source Active Joined, SAR - BGP Source Active Received,
SAS - BGP Source Active Sent, IM - Inband mLDP, X - VxLAN
Interface state: Name, Uptime, Fwd, Info
Interface flags: LI - Local Interest, LD - Local Dissinterest,
II - Internal Interest, ID - Internal Dissinterest,
LH - Last Hop, AS - Assert, AB - Admin Boundary, EX - Extranet,
BGP - BGP C-Multicast Join, BP - BGP Source Active Prune,
MVS - MVPN Safi Learned, MV6S - MVPN IPv6 Safi Learned
(10.1.0.2,232.11.11.11)SPT SSM Up: 15:30:53
JP: Join(BGP) RPF: LmdtMLDP-14,10.44.0.10 Flags:
Bundle-Ether5000.1 15:30:53 fwd LI LH >>>>> UPTIME of PIM & all egress Interfaces last hop & local interest
Bundle-Ether5000.2 15:30:45 fwd LI LH
RP/0/RSP0/CPU0:AG2-1#
RP/0/RSP0/CPU0:AG2-1#sh pim vrf MLDP-14 interface be5000.1 detail
Fri Mar 12 02:49:36.698 EST
PIM interfaces in VRF MLDP-14
IP PIM Multicast Interface State
Flag: B - Bidir enabled, NB - Bidir disabled
P - PIM Proxy enabled, NP - PIM Proxy disabled
V - Virtual Interface
BFD State - State/Interval/Multiplier
Interface PIM Nbr Hello DR
Count Intvl Prior
Bundle-Ether5000.1 on 2 30 1 >>> PIM ON, Hello Interval 30 sec
Primary Address : 10.1.0.1
Flags : NB NP V
BFD : On/500 ms/3 >>> BFD ON
DR : this system >>> This is DR
Propagation delay : 500
Override Interval : 2500
Hello Timer : 00:00:27
Neighbor Filter : -
RP/0/RSP0/CPU0:AG2-1#sh pim vrf MLDP-14 neighbor bundle-ether 5000.1 detail
Fri Mar 12 03:03:43.892 EST
PIM neighbors in VRF MLDP-14
Flag: B - Bidir capable, P - Proxy capable, DR - Designated Router,
E - ECMP Redirect capable
* indicates the neighbor created for this router
Neighbor Address Interface Uptime Expires DR pri Flags
10.1.0.1* Bundle-Ether5000.1 15:46:40 00:01:38 1 (DR) B E >>>> Verify DR
Expiry Timer: 00:00:07
10.1.0.2 Bundle-Ether5000.1 15:46:27 00:01:25 0
BFD State: enabled
RP/0/RSP0/CPU0:AG2-1#sh pim vrf MLDP-14 mdt cache | utili egrep -w "(10.1.0.2, 232.11.11.11)"
Fri Mar 12 03:04:19.783 EST
10.44.0.10 (10.1.0.2, 232.11.11.11) [global-id 4] never >>>> Verify Global ID for specific S,G
Note: This global id remains the same/unique from receiver to Source for this specific S, G.
PIM Errors/Stats:
show pim traffic
PIM Traffic Counters
Elapsed time since counters cleared: 1d01h
Received Sent
Valid PIM Packets 15759217 15214426 >>> verify the pim packets increment
Hello 9207 12336 >>> verify the pim hello Negotiations
Join-Prune 1076805 531981
Data Register 14673205 0
Null Register 73205 0
Register Stop 0 14673205
Assert 0 0
Batched Assert 0 0
Bidir DF Election 0 0
BSR Message 0 0 >>> If you have RP check this stats
Candidate-RP Adv. 0 0 >>> If you have RP check this stats
Join groups sent 0
Prune groups sent 0
Output JP bytes 0
Output hello bytes 410
Errors: >>>>> Check below paramters for errors or drops
Malformed Packets 0
Bad Checksums 0
Socket Errors 0
Subnet Errors 0
Packets dropped since send queue was full 0
Packets dropped due to invalid socket 0
Packets which couldn't be accessed 0
Packets sent on Loopback Errors 6
Packets received on PIM-disabled Interface 0
MRIB Check:
RP/0/RSP0/CPU0:AG2-1#sh mrib vrf MLDP-14 route 10.1.0.2 232.11.11.11 detail
Fri Mar 12 03:05:38.300 EST
IP Multicast Routing Information Base
Entry flags: L - Domain-Local Source, E - External Source to the Domain,
C - Directly-Connected Check, S - Signal, IA - Inherit Accept,
IF - Inherit From, D - Drop, ME - MDT Encap, EID - Encap ID,
MD - MDT Decap, MT - MDT Threshold Crossed, MH - MDT interface handle
CD - Conditional Decap, MPLS - MPLS Decap, EX - Extranet
MoFE - MoFRR Enabled, MoFS - MoFRR State, MoFP - MoFRR Primary
MoFB - MoFRR Backup, RPFID - RPF ID Set, X - VXLAN
Interface flags: F - Forward, A - Accept, IC - Internal Copy,
NS - Negate Signal, DP - Don't Preserve, SP - Signal Present,
II - Internal Interest, ID - Internal Disinterest, LI - Local Interest,
LD - Local Disinterest, DI - Decapsulation Interface
EI - Encapsulation Interface, MI - MDT Interface, LVIF - MPLS Encap,
EX - Extranet, A2 - Secondary Accept, MT - MDT Threshold Crossed,
MA - Data MDT Assigned, LMI - mLDP MDT Interface, TMI - P2MP-TE MDT Interface
IRMI - IR MDT Interface
(10.1.0.2,232.11.11.11) Ver: 0x7023 RPF nbr: 10.44.0.10 Flags: RPF RPFID,
PD: Slotmask: 0x4 >>>>Slot Mask denotes which slot it will be flooded (binary conversion : 0100) Slot 2 (slot 02 -> 0/0/CPU0)
MGID: 5489 >> Note MGID
Up: 15:48:34
RPF-ID: 639, Encap-ID: 0 >>Note RFID & ENCAP-ID. EnCAP ID 0 is Default (Partition MDT)
Incoming Interface List
LmdtMLDP-14 Flags: A LMI, Up: 00:36:35
Outgoing Interface List
Bundle-Ether5000.1 (0/0/CPU0) Flags: F NS LI, Up: 15:48:34 >>Match this slot
Bundle-Ether5000.2 (0/0/CPU0) Flags: F NS LI, Up: 15:48:26
RP/0/RSP0/CPU0:AG2-1#
Check bundle member interface active:
RP/0/RSP0/CPU0:AG2-1#sh int bundle-ether 5000 | in Active
Fri Mar 12 03:09:52.186 EST
TenGigE0/0/0/17 Full-duplex 10000Mb/s Active >>> It belong to LC0 , NP5, FIA 2
RP/0/RSP0/CPU0:AG2-1#
Check NP and FIA mapping of TenGigE0/0/0/17:
RP/0/RSP0/CPU0:AG2-1#sh controllers np ports all location 0/0/CPU0
Fri Mar 12 03:11:53.080 EST
Node: 0/0/CPU0:
----------------------------------------------------------------
NP Bridge Fia Ports
-- ------ --- ---------------------------------------------------
0 -- 0 TenGigE0/0/0/0 - TenGigE0/0/0/2
1 -- 0 TenGigE0/0/0/3 - TenGigE0/0/0/5
2 -- 1 TenGigE0/0/0/6 - TenGigE0/0/0/8
3 -- 1 TenGigE0/0/0/9 - TenGigE0/0/0/11
4 -- 2 TenGigE0/0/0/12 - TenGigE0/0/0/14
5 -- 2 TenGigE0/0/0/15 - TenGigE0/0/0/17
6 -- 3 TenGigE0/0/0/18 - TenGigE0/0/0/20
7 -- 3 TenGigE0/0/0/21 - TenGigE0/0/0/23
BGP mVPN check:
RP/0/RSP0/CPU0:AG2-1#sh bgp ipv4 mvpn vrf MLDP-14
Fri Mar 12 03:14:31.903 EST
BGP router identifier 10.44.0.5, local AS number 55836
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0 RD version: 0
BGP main routing table version 13952
BGP NSR Initial initsync version 207 (Reached)
BGP NSR/ISSU Sync-Group versions 13952/0
BGP scan interval 60 secs
Status codes: s suppressed, d damped, h history, * valid, > best
i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1000:1001 (default for vrf MLDP-14)
*> [1][10.44.0.5]/40 0.0.0.0 0 i
*>i[1][10.44.0.10]/40 10.44.0.10 100 0 i
*>i[1][10.44.0.110]/40
10.44.0.110 100 0 i
*> [3][0][0.0.0.0][0][0.0.0.0][10.44.0.5]/120
0.0.0.0 0 i
*>i[3][0][0.0.0.0][0][0.0.0.0][10.44.0.10]/120
10.44.0.10 100 0 i
*>i[3][0][0.0.0.0][0][0.0.0.0][10.44.0.110]/120
10.44.0.110 100 0 i
*>i[3][32][10.1.0.2][32][232.11.11.11][10.44.0.10]/120 >>> Type 3 is received from Source PE
10.44.0.10 100 0 i
*> [7][1000:1001][55836][32][10.1.0.2][32][232.11.11.11]/184 >>> Type 7 is advertised from receiver PE
0.0.0.0 0 i
Processed 8 prefixes, 8 paths
LC mFIB Programming Check:
RP/0/RSP0/CPU0:AG2-1#sh mfib vrf MLDP-14 platform route 10.1.0.2 232.11.11.11 location 0/0/CPU0
Fri Mar 12 03:19:41.712 EST
LC Type: A9K-24x10GE-SE
--------------------------------------------------------------------------
Legend:
Common SW:
Route Information
B: BACL check S: RPF Interface signal
DC: Directly connected PL: Punt to LC CPU
PR: Punt to RP PF: Punt if forwarded
DR: Drop all RI: RPF interface
MF: MoFRR enabled TR: Tunnel route
TE: Tunnel encap CD: Conditional decap
MI: MVET index NI: Not Installed
BD: Bidir
MVPN Information (Common SW)
MLI: MVPN table index VTID: VRF Table ID
RPFID: RPF Identifier
MoFRR Information
A: Active RPF interface RI: RPF interface (or RPF ID)
WDI: Watchdog counter index
--------------------------------------------------------------------------
Source: 10.1.0.2 Group: 232.11.11.11 Mask: 64 RPF ID : 639 >> Verify RFID
Common SW Information
---------------------
MGID: 5489 MLI: 2132 Fabric Slotmask: 0x4 FGID: 0x4 >>> Verify MGID & Slotmask
------------------------------------------------------------------------------
B S DC PL PR PF DR BD RI MF TR TE CD MI
------------------------------------------------------------------------------
F F F F F F F F 639 F F F F 0x0
---------------------------------------------------------------------------
MVPN/MLDP Decap Information:
RPF ID: 639
RPF ID MGID: Primary:7737 Backup:7738
Last refcount bitmask (RPF ID): 0x20, Sets: 21, Clears: 20
MNP Olist: [NP0:0 NP1:0 NP2:0 NP3:0 NP4:0 NP5:2 NP6:0 NP7:0 ] >>> Verify NP as mentioned above it is NP5
Turnaround flag:
Check forwarding rate in bytes and packets for specific S, G on platform/PD:
RP/0/RSP0/CPU0:AG2-1#sh mfib vrf MLDP-14 route rate detail
Fri Mar 12 03:17:06.998 EST
IP Multicast Forwarding Rates
(Source Address, Group Address)
Incoming rate: (Incoming interface)
Node: (Incoming node) : pps/bps
HW Incoming count : (in packets)
HW Drop count : (in packets)
Outgoing rate:
Node: (Outgoing node) : pps/bps
HW Forwarding count: (in packets)
HW Drop count: (in packets)
Interfaces: (Outgoing interface list)
(10.1.0.2,232.11.11.11) >>>> Forwarding rate for specific S,G
Incoming rate : MDT Lm14
Node : 0/0/CPU0 : 200565 / 766963242 >>>> Incoming rate bps/pps
HW Incoming count : 3233699818 packets >>> Incoming rate in packets
HW Drop count : 0 packets >>> Incoming drops
Outgoing rate :
Node : 0/0/CPU0 : 99013 / 378628394 >>>> Outgoing rate bps/pps
HW Forwarding count: 1616848961 packets >>>> Outgoing rate in packets
HW Drop count: 0 packets >>>> Outgoing drops
Interfaces: BE5000.1 BE5000.2
PD statistics for S, G:
RP/0/RSP0/CPU0:AG2-1#sh mfib vrf MLDP-14 platform route statistics 10.1.0.2 232.11.11.11 location 0/0/CPU0
Fri Mar 12 03:21:57.222 EST
LC Type: A9K-24x10GE-SE
--------------------------------------------------------------------------
Legend:
N: NP ID R: Received
F: Forwarded P: Punted to CPU
ID: Ingress Drop ED: Egress Drop
Source: 10.1.0.2 Group: 232.11.11.11 Mask:64
-------------------------------------------------------------------------
NP R(packets:bytes)/F(packets:bytes)/P(packets)/ID(packets)/ED(packets)
-------------------------------------------------------------------------
0 1645909137:786744567486 / 0:0 / 0 / 0 / 1645909136
1 0:0 / 0:0 / 0 / 0 / 0
2 0:0 / 0:0 / 0 / 0 / 0
3 0:0 / 0:0 / 0 / 0 / 0
4 0:0 / 0:0 / 0 / 0 / 0
5 1645819072:786701516416 / 1645914052:786746916856 / 0 / 0 / 0 >> Forwarding rate on NP5 in bytes & packets cumulative
6 0:0 / 0:0 / 0 / 0 / 0
7 0:0 / 0:0 / 0 / 0 / 0
-------------------------------------------------------------------------
Interface Statistics:
-------------------------------------------------------------------------
C Interface F/P/D (packets:bytes)
-------------------------------------------------------------------------
5 BE5000.1 1645819072:786701516416 / 0:0 / 0:0 >> Forwarding rate on interface
5 BE5000.2 1645914051:786746916378 / 0:0 / 0:0
-------------------------------------------------------------------------
Check forwarding rate in bps and pps for specific S, G on platform/PD:
RP/0/RSP0/CPU0:AG2-1#sh mfib vrf MLDP-14 route stati 10.1.0.2 232.11.11.11 location 0/0/CPU0
Fri Mar 12 03:25:44.237 EST
IP Multicast Forwarding Information Base
Entry flags: C - Directly-Connected Check, S - Signal, D - Drop,
IA - Inherit Accept, IF - Inherit From, EID - Encap ID,
ME - MDT Encap, MD - MDT Decap, MT - MDT Threshold Crossed,
MH - MDT interface handle, CD - Conditional Decap,
DT - MDT Decap True, RPFID - RPF ID Set
Interface flags: F - Forward, A - Accept, IC - Internal Copy,
NS - Negate Signal, DP - Don't Preserve, SP - Signal Present,
EG - Egress, EI - Encapsulation Interface, MI - MDT Interface
SW/HW Forwarding/Replication Counts: Packets in/Packets out/Bytes out
SW Failure Counts: RPF / TTL / Empty Olist / Encap RL / Other
HW Drop Counts: Ingress / Egress
HW Forwarding Rates: bps In/pps In/bps Out/pps Out
(10.1.0.2,232.11.11.11), Flags:
Up: 16:08:40
Last Used: never
SW Forwarding Counts: 0/0/0
SW Replication Counts: 0/0/0
SW Failure Counts: 0/0/0/0/0
HW Forwarding Counts: 3337154990/1668576693/797579659254
HW Replication Counts: 3337154990/3337166638/1595165652964
HW Drop Counts: 0/0 >>>> DROP counter if it is incrementing then it is a problem (Check np DROP counters)
HW Forwarding Rates: 766982645/200570/388346650/101555 -à It is in current bps & pps on LC (it should be incremental)
LmdtMLDP-14 Flags: A LMI, Up:00:56:41 >>> MDT uptime
Bundle-Ether5000.1 Flags: NS EG, Up:16:08:40 >>>> uptime in PD
Bundle-Ether5000.2 Flags: NS EG, Up:16:08:32
RP/0/RSP0/CPU0:AG2-1#
Use MGID Index (noted previously) to check the clients and mapping of respective FIA and XBAR:
RP/0/RSP0/CPU0:AG2-1#sh controllers mgidprgm mgidindex 5489 location 0/0/CPU0
Fri Mar 12 03:28:22.682 EST
Device MGID-Bits Client-Last-Modified
=======================================================
XBAR-0 100 MFIBV4 >> X-BAR instance 0
FIA-0 0 MFIBV4
FIA-1 0 MFIBV4
FIA-2 10 MFIBV4 >> Ten-0/0/0/17 is part of FIA 2
FIA-3 0 MFIBV4
========================================================
Client Mask
========================================================
MFIBV4 0x20
MFIBV6 0x0
L2FIB 0x0
sRP-pseudo-mc 0x0
UT 0x0
Prgm-Svr 0x0
P2MP 0x0
xbar 0x0
UT1 0x0
UT2 0x0
punt_lib 0x0 punt_lib 0x0
Platform programming for S, G and OLIST state:
RP/0/RSP0/CPU0:AG2-1#sh mfib vrf MLDP-14 platform route olist detail 10.1.0.2 232.11.11.11 location 0/0/CPU0
Fri Mar 12 03:31:17.067 EST
LC Type: A9K-24x10GE-SE
--------------------------------------------------------------------------
Legend:
Common SW:
Route Information
NP: NP ID B: BACL check
S: RPF Interface signal DC: Directly connected
PL: Punt to LC CPU PR: Punt to RP
PF: Punt if forwarded DR: Drop all
RI: RPF interface FS: Fabric slotmask
G: Multicast group ID M: Multicast Leaf Index
T: Table ID for lookup OC: Count of OLIST members
Base: Base of the statistics pointer NI: Not Installed
BD: Bidir
Interface Information
NP: NP ID Intf: Interface
U: uIDB index OT: OLE Type
T: Table ID IC: HW IC flag
B: HW BACL bit EU: Interface uIDB index
IB: Bundle interface EH: In HW OLIST table
OIDX: OLIST index on NP PT: Punt table entry
Base: Statistics Ptr base RM: Remote FGID (Pri/Back)
BD_ID: Bridge Domain ID if BVI RH: Raw hash
Software OLIST Information
SW OC: Software OLIST counts HW OC: Hardware OLIST counts
T: Table ID SD: Send direct flag
--------------------------------------------------------------------------
Engine: Olist Information:
NP: NP ID Intf: Interface
U: uIDB index OT: OLE Type
T: Table ID IC: HW IC flag
B: HW BACL bit EU: Interface uIDB index
IB: Bundle interface EH: In HW OLIST table
OIDX: OLIST index on NP PT: Punt table entry
Base: Statistics Ptr base RM: Remote FGID (Pri/Back)
BD_ID: Bridge Domain ID if BVI RH: Raw hash
--------------------------------------------------------------------------
Source: 10.1.0.2 Group: 232.11.11.11 Mask: 64 RPF ID : 639
Common SW Information
---------------------
------------------------------------------------------------------------------
NP B S DC PL PR PF DR BD RI FS G M T OC Base
------------------------------------------------------------------------------
0 F F F F F F F F 639 0x4 5489 2132 0 0 0x12c88
1 F F F F F F F F 639 0x4 5489 2132 0 0 0x12c88
2 F F F F F F F F 639 0x4 5489 2132 0 0 0x12c88
3 F F F F F F F F 639 0x4 5489 2132 0 0 0x12c88
4 F F F F F F F F 639 0x4 5489 2132 0 0 0x12cb0
5 F F F F F F F F 639 0x4 5489 2132 0 2 0x12cb0
6 F F F F F F F F 639 0x4 5489 2132 0 0 0x12cc8
7 F F F F F F F F 639 0x4 5489 2132 0 0 0x12cb8
---------------------------------------------------------------------------
Interface Information
------------------------------------------------------------------------------
NP Intf OT U T IC B EU IB EH OIDX PT Base RH
------------------------------------------------------------------------------
5 BE5000.1 REG 931 0 F F 931 T T/T 0 T 0x12cb8 0xef8edd84 >> verify UIDB 931
5 BE5000.2 REG 930 0 F F 930 T T/T 1 T 0x12cbb 0xef8edd84
------------------------------------------------------------------------------
Software OLIST Information
--------------------------------
NP SW OC HW OC T SD
--------------------------------
5 2 2 0 F
--------------------------------
Virtual Interface Local Presence
--------------------------------------------------------
NP Intf UL Intf Bundle Parent
--------------------------------------------------------
5 Bundle-Ether5000.1 Te0/0/0/17 Bundle-Ether5000 >> Check the member interface this S,G is forwarding
5 Bundle-Ether5000.2 Te0/0/0/17 Bundle-Ether5000
--------------------------------------------------------
Engine Information
---------------------
---------------------------------------------------------------------------
NP Intf OT U T IC B VRF Base RH
---------------------------------------------------------------------------
5 0xa03e0200 REG 41731 0 F F 0xb82c0100 0x84dd8eef
5 0xe03e0200 REG 41475 0 F F 63747 0xbb2c0100 0x84dd8eef
---------------------------------------------------------------------------
Verify UIDB programming:
RP/0/RSP0/CPU0:AG2-1#sh uidb index location 0/0/CPU0 | in 931
Fri Mar 12 03:41:21.876 EST
0/0/CPU0 0x00023ea0 Bundle-Ether5000.1 Sub-interface 931
Check Bundle Hashing for S, G:
RP/0/RSP0/CPU0:AG2-1#sh mrib platform bundle hash ip 232.11.11.11 10.1.0.2 bundle-ether5000
Fri Mar 12 03:31:19.067 EST
===================================================
(Source, Group): (10.1.0.2,232.11.11.11)
Bundle interface: Bundle-Ether500 (0x20002a0)
Bundle member: TenGigE0/0/0/17 (0x580)
Raw Hash (32b): 0x8febeed6
Lag Hash (12b): 0x00d6
Member index: 0
===================================================
RP/0/RSP0/CPU0:AG2-1#sh mrib platform bundle hash ip 232.11.21.11 10.1.0.2 bundle-ether5000
Fri Mar 12 03:31:21.067 EST
===================================================
(Source, Group): (10.1.0.2,232.11.11.11)
Bundle interface: Bundle-Ether5000 (0x20002a0)
Bundle member: TenGigE0/1/0/17 (0x5c0)
Raw Hash (32b): 0x8014609b
Lag Hash (12b): 0x009b
Member index: 1
Check BCDL client status:
RP/0/RSP0/CPU0:AG2-1#sh mrib ipv4 client
Tue Aug 31 06:35:37.639 EDT
IP MRIB client-connections
bcdl_agent:1 (client id 28)
igmp:7467 (client id 23)
ldp:17503 (client id 52)
lmrib_bcdl:1 (client id 30) >>> Expected
lmrib_bcdl:2 (client id 60) >>>> Not expected if more than 1 >> fix provided Cisco bug ID CSCvy58964
pim:7471 (client id 26)
pim6_lbl:7644 (client id 20)
pim_lbl:7471 (client id 17)
xtc:7458 (client id 33)
Towards Core
Verify MLDP status:
RP/0/RSP0/CPU0:AG2-1#sh mpls mldp status
Fri Mar 12 02:56:00.727 EST
mLDP statistics
Process status : Active, Running and Ready
Logging notifications : Disabled
Label release scan in : never
LSM ID scan in : never
RIB connection status : Connected
RIB connection open : Yes
Check the MLDP database with Global-id:
The behaviour of recursive-fec.
Here for Tail/leaf end to reach root/Head end, there is next-hop change.
On Tail End, Head end loopback is learned via AG3-2 (MID Router). So here recursive-fec plays an important role
Configuration:
mldp
address-family ipv4
recursive-fec
!
- Enable mLDP recursive fec makea Leaf (Ingress PE) to build Partitioned MDT towards the upstream router and insert the inline-RR (AG3-2) (that is, next-hop for Ingress PE/Original Root prefix) as the root.
RP/0/RSP0/CPU0:AG2-1#sh mpls mldp database root 10.44.0.10 opaquetype global-id 4 details
Fri Mar 12 03:42:20.594 EST
mLDP database
LSM-ID: 0x000D6 (LSP-ID: 0x8005D) Type: P2MP Uptime: 01:13:02
FEC Root : 10.44.0.10 >>>> This is the FEC root where Multicast Source is connected
FEC Length : 12 bytes
FEC Value internal : 0201000400000004312C000A
Opaque length : 4 bytes
Opaque value : 01 0004 00000004
Opaque decoded : [global-id 4]
Features : MBB RFEC RFWD Trace
Upstream neighbor(s) :
Recursive encode LSM-ID: 0x000D7 >>> 10.44.0.10 is reachable via 10.44.0.4 so Recursive LSM check the next command
Downstream client(s):
PIM MDT Uptime: 01:13:02
Egress intf : LmdtMLDP-14 >>>> Pointing towards mdt tunnel VRF
Table ID : IPv4: 0xe00003f9 IPv6: 0xe08003f9
RPF ID : 639 >>>> Check the RPFID it should match with mfib & mrib o/p
RD : 1000:1001 >> Check RD value
Verify the LSM ID state in the MLDP database:
RP/0/RSP0/CPU0:AG2-1#sh mpls mldp database 0x000D7 details >>>> Recursive LSM ID (Outer FEC)
Fri Mar 12 03:44:54.350 EST
mLDP database
LSM-ID: 0x000D7 (LSP-ID: 0x8005D) Type: P2MP Uptime: 01:15:36
FEC Root : 10.44.0.4
FEC Length : 25 bytes
FEC Value internal : 020700110600010312C000A000701000400000004312C0004
Opaque length : 17 bytes
Opaque value : 07 0011 0600010312C000A000701000400000004
Opaque decoded : [recursive] 10.44.0.10:[global-id 4] >>>> inner fec which is pointing towards actual source 10.44.0.10
Features : MBB RFEC RFWD Trace
Upstream neighbor(s) :
Is CSI accepting : N
10.44.0.4:0 [Active] [MBB] Uptime: 01:15:36 >>> upstream neighbor. Check ldp neighbor with hop
Local Label (D) : 29233 >>>> local label
Downstream client(s):
Recursive 0x000D6 Uptime: 01:15:36 >>> This is outer fec which is pointing towards VRF interface actual receiver (check above)
PIM MDT Uptime: 01:15:36 (from 0x000D6) >> MDT tunnel uptime
Egress intf : LmdtMLDP-14
Table ID : IPv4: 0xe00003f9 IPv6: 0xe08003f9
RPF ID : 639
RD : 1000:1001
Verify MRIB parameters for local labels for S, G:
RP/0/RSP0/CPU0:AG2-1#sh mrib mpls forwarding labels 29233 detail
Fri Mar 12 03:47:45.907 EST
LSP information (mLDP) :
LSM-ID: 0x8005D, Role: Tail >>>> Verify the LSM ID & RPF -ID with MLDP database
RPF-ID: 0x0027F, Assoc-TIDs: 0xe00003f9/0xe08003f9, MDT: LmdtMLDP-14 >> RFID is in hex convert into DEC & verify it
Incoming Label : 29233
Transported Protocol : <unknown>
Explicit Null : None
IP lookup : enabled
Platform information :
Slotmask: Primary: 0x4, Backup: 0x4 >> Verify slotmask
MGID Primary: 7751, Backup: 7752 >>> Verify the same with mfib platform route location command
Label Node Role: LMRIB_ROUTER_ROLE_EGRESS, Tunnel Type: MVPN Tunnel
Outsegment Info #1 [T/Pop]:
No info.
PlatInfo:
Primary: [Main IF: NULL (IFH: 0), UL IF: NULL (IFH: 0)]
Backup: [Main IF: NULL (IFH: 0), UL IF: NULL (IFH: 0)]
Check the RPFID and respective MGID:
RP/0/RSP0/CPU0:AG2-1#show mrib rpf-id | b 639
Fri Mar 12 03:52:43.375 EST
Addr: 127.2.0.0, RPF-ID: 639 (0x27f)
Alloc: F, IPv6 Intr: F, Stale: F, Chkpt Obj-ID: 0x0
Routes - Non-core: 1, Core: 0, Label: 2
In PD Retry: F
Platform Annot: Primary MGID: 7737, Backup MGID: 7738
Verify MPLS forwarding:
RP/0/RSP0/CPU0:AG2-1#sh mpls forwarding labels 29233
Fri Mar 12 04:00:00.695 EST
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
29233 Unlabelled mLDP/IR: 0x8005d >> verify with MPLS MLDP database LSP -ID
RP/0/RSP0/CPU0:AG2-1#sh mpls forwarding labels 29233 location 0/0/CPU0
Fri Mar 12 04:00:05.962 EST
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
29233 Unlabelled mLDP/IR: 0x8005d 289 > For LSP card, MPLS forwarding stats is not incrementing fix provided Cisco bug ID CSCwe17513
RP/0/RSP0/CPU0:AG2-1#
Check CEF forwarding:
RP/0/RSP0/CPU0:AG2-1#sh cef 10.44.0.10 >> FEC Root IP
Fri Mar 12 05:48:01.838 EST
10.44.0.10/32, version 149189, internal 0x1000001 0x0 (ptr 0x7a11058) [1], 0x0 (0x79c18c08), 0xa08 (0x79e5556c)
Updated Mar 12 02:29:03.109
Prefix Len 32, traffic index 0, precedence n/a, priority 4
via 10.44.0.4/32, 6 dependencies, recursive [flags 0x6000]
path-idx 0 NHID 0x0 [0x727a0730 0x0]
recursion-via-/32
next hop 10.44.0.4/32 via 24215/0/21
local label 29229
next hop 10.16.26.10/32 Te0/1/0/2 labels imposed {ImplNull 25509}
RP/0/RSP0/CPU0:AG2-1#sh cef 10.44.0.4 >> Next Hop for 10.44.0.10
Fri Mar 12 05:48:07.914 EST
10.44.0.4/32, version 117384, internal 0x1000001 0x0 (ptr 0x721ee2a8) [1], 0x0 (0x79c14ae4), 0xa20 (0x777e8518)
Updated Mar 11 20:41:39.060
remote adjacency to TenGigE0/1/0/2
Prefix Len 32, traffic index 0, precedence n/a, priority 1
via 10.16.26.0/32, Bundle-Ether45, 3 dependencies, weight 0, class 0, backup (Local-LFA) [flags 0x300]
path-idx 0 NHID 0x0 [0x79a002a0 0x79a00508]
next hop 10.16.26.0/32
local adjacency
local label 24215 labels imposed {ImplNull}
via 10.16.26.10/32, TenGigE0/1/0/2, 4 dependencies, weight 0, class 0, protected [flags 0x400]
path-idx 1 bkup-idx 0 NHID 0x0 [0x78eeb420 0x78ef0b50]
next hop 10.16.26.10/32
local label 24215 labels imposed {ImplNull}
RP/0/RSP0/CPU0:AG2-1#
Check the NP drops counters:
RP/0/RSP0/CPU0:ASR9006-L#sh controllers np counters np0 location 0/2/CPU0 | util egrep -i "DROP|OLE|MDF"
Fri Oct 30 03:31:18.619 UTC
16 MDF_TX_LC_CPU 131216974 491
17 MDF_TX_WIRE 1233084179074 393586. >>> Packets towards egress PHY from NP
21 MDF_TX_FABRIC 1297085738439 297619 >>> Packets towards Fabric from NP
105 RSV_DROP_EGR_UIDB_DOWN 1460 0
117 RSV_DROP_MPLS_NRLDI_NO_MATCH 413748715 0 >>
161 RSV_DROP_EGR_LAG_NO_MATCH 716 0
232 RSV_DROP_MPLS_LEAF_NO_MATCH 860077083 0
438 RSV_LSM_NO_OLE 2714 0. >>>>due to no OLE mapped or underlay Join for LSM ID
544 MDF_PIPE_LPBK 537269848146 297618
564 MDF_PUNT_POLICE_DROP 39231 0
573 MODIFY_PUNT_REASON_MISS_DROP 2 0
633 MDF_LB_TM0_LPBK0 144115523350 75159 >>> TM0 loopback utilization (Tomhawk/Typh case)
634 MDF_LB_TM0_LPBK1 799575201453 74405
635 MDF_LB_TM1_LPBK0 143959136311 74405 >>> TM1 loopback utilization (in Tomhawk case)
636 MDF_LB_TM1_LPBK1 143733432720 73661
637 MDF_TM_LP_Q_P1 6321879 10
639 MDF_TM_LP_Q_PN 1231376971969 297618
1945 PARSE_DROP_IN_UIDB_DOWN 696210 0
1948 PARSE_DROP_IPV4_DISABLED 2866 0
1979 PARSE_DROP_IPV6_DISABLED 269 0
Verify NP or FIA Drops on LC:
RP/0/RSP0/CPU0:B#sh drops all ongoing location 0/2/CPU0
=====================================
Checking for ongoing drops on 0/2/CPU0
=====================================
[np:NP0] RSV_DROP_IPV4_LEAF_NO_MATCH: +323
[np:NP0] RSV_DROP_MPLS_TXADJ_DROP: +393205
[np:NP0] RSV_EGR_LAG_NOT_LOCAL_DROP_CNT: +36
[np:NP0] MDF_RPF_FAIL_DROP: +24058109. >>>>> Check use case TZ to understand this
[np:NP1] RSV_DROP_IPV4_LEAF_NO_MATCH: +246
[np:NP1] RSV_DROP_MPLS_TXADJ_DROP: +393216
[np:NP1] RSV_EGR_LAG_NOT_LOCAL_DROP_CNT: +304
[np:NP1] MDF_RPF_FAIL_DROP: +24116093
show controller np counters:
[np:NP0] RSV_DROP_IPV4_LEAF_NO_MATCH: +29
[np:NP0] RSV_DROP_MPLS_TXADJ_DROP: +16076
[np:NP0] MDF_RPF_FAIL_DROP: +979311
[np:NP1] RSV_DROP_IPV4_LEAF_NO_MATCH: +9
[np:NP1] RSV_DROP_MPLS_TXADJ_DROP: +16080
[np:NP1] RSV_EGR_LAG_NOT_LOCAL_DROP_CNT: +11
[np:NP1] MDF_RPF_FAIL_DROP: +984429
[np:NP2] UNKNOWN_L2_ON_L3_DISCARD: +2
Note: MDF_RPF_FAIL_DROP is due to RPF fail is when the RPF interface in the route does not match with the incoming intf of the packet.
RP/0/RSP0/CPU0:PUNEKLCLAAR001#sh drops all ongoing location 0/0/CPU0 | include "RSV_DROP_ACL_DENY|TM"
Thu Dec 24 20:33:30.086 IST
[np:NP3] RSV_DROP_ACL_DENY: +109
[np:NP5] RSV_DROP_ACL_DENY: +154
[np:NP6] RSV_DROP_ACL_DENY: +74
[np:NP7] RSV_DROP_ACL_DENY: +29
[np:NP 3 TM 0] drop paks: +5909 >>> Check TM drops if it increasing rapidly check the QOS stats & RFD buffers as mentioned below
[np:NP 5 TM 0] drop paks: +909
[np:NP 6 TM 0] drop paks: +8925
[np:NP 7 TM 0] drop paks: +5491
If you observe that TM drops increase more rapidly, then collect these logs:
Show policy-map interface <> out >>>>> egress stats >>>>Check if there is any drops in queue
Collect np datalogs
Get into LC shell (run attach 0/x/cpu0)
Execute show_np –e <np_id> -d npdatalog:
Also, collect multiple(4-5) iterations of the pipeline from the LC shell.
show_np -e <np_id> -d -pipeline >>>> Only applicable for Tomhawk LC
Since Multicast traffic uses TM loopback, check if there are any drops in qoshal (example NP4 Location 4).
Sample o/p:
Tailend#show qoshal loopback-queue np 4 location 0/0/CPU0
Tue Jun 9 10:51:12.908 IST
TY Options argc:6
nphal_show_chk -p 2049loopback-queue-n 0x4
Done
Loopback Queues : NP 4
===============================================================
Multi-cast Loopback Queue NP 4
Ingress: QID 0x40 Entity: 4/0/0/4/8/0 Priority: Priority 1 Qdepth: 0
StatIDs: commit/fast_commit/drop: 0x5f0140/0x0/0x5f0141
Statistics(Pkts/Bytes):
Tx_To_TM 0/0
Total Xmt 0/0 Dropped 0/0 >>> Check Transmit & Drop packets in P1
Ingress: QID 0x41 Entity: 4/0/0/4/8/1 Priority: Priority 2 Qdepth: 0
StatIDs: commit/fast_commit/drop: 0x5f0145/0x0/0x5f0146
Statistics(Pkts/Bytes):
Tx_To_TM 0/0
Total Xmt 0/0 Dropped 0/0 >>> Check Transmit & Drop packets in P2
Ingress: QID 0x43 Entity: 4/0/0/4/8/3 Priority: Priority 3 Qdepth: 0
StatIDs: commit/fast_commit/drop: 0x5f014f/0x0/0x5f0150
Statistics(Pkts/Bytes):
Tx_To_TM 0/0
Total Xmt 0/0 Dropped 0/0 >>> Check Transmit & Drop packets in P3
Ingress: QID 0x42 Entity: 4/0/0/4/8/2 Priority: Priority Normal Qdepth: 0
StatIDs: commit/fast_commit/drop: 0x5f014a/0x0/0x5f014b
Statistics(Pkts/Bytes):
Tx_To_TM 6607491182505/9627383230841487
Total Xmt 349820789/509702036782 Dropped 6607141361716/9626873528804705 >>>> Check Transmit & Drop packets in Default Queue
To catch any bursty traffic:
np_perf -e 0 -R -o 4 -D 10 -n 5000
R is for rfd
O is for special ports (loopback comes under that)
D is duration; in the above example it is 10 secs; max you can specify is 2 days or something like that
MID Router
Towards Receivers
Check MLDP status:
RP/0/RSP1/CPU0:AG3-2#sh mpls mldp status
Fri Mar 12 08:08:37.366 UTC
mLDP statistics
Process status : Active, Running and Ready >>>> Active
Logging notifications : Disabled
Label release scan in : never
LSM ID scan in : never
RIB connection status : Connected
RIB connection open : Yes
RP/0/RSP1/CPU0:AG3-2#
Check the MLDP database:
Behaviour of recursive-fec
Here for the mid-end (inline RR) to reach the root/head-end, there is next-hop as root.
Configuration:
mldp
address-family ipv4
recursive-fec
!
Sample o/p:
RP/0/RSP1/CPU0:AG3-2#sh mpls mldp database opaquetype recursive global-id 4
Fri Mar 12 07:54:05.599 UTC
mLDP database
LSM-ID: 0x00023 (LSP-ID: 0x80023) Type: P2MP Uptime: 01:26:35
FEC Root : 10.44.0.4 (we are the root) >>> In this router, outer FEC get remove
Opaque decoded : [recursive] 10.44.0.10:[global-id 4]
Features : MBB RFWD
Upstream neighbor(s) : >>>>>>>>>>>>>>>>>>>Upstream client towards HEADEND
Recursive decode LSM-ID: 0x00024 >> Inner FEC to reach 10.44.0.10
Downstream client(s): >>>>>>>> Downstream client towards Receiver
LDP 10.44.0.5:0 Uptime: 01:26:35
Rec Next Hop : 10.44.0.5 >> Receiver end IP
Remote label (D) : 29233 >>> Remote label to reach 10.44.0.5
RP/0/RSP1/CPU0:AG3-2#sh mpls mldp database 0x00024 >>> Inner FEC (Actual LSM to reach Source)
Fri Mar 12 07:55:20.346 UTC
mLDP database
LSM-ID: 0x00024 (LSP-ID: 0x80023) Type: P2MP Uptime: 01:27:50
FEC Root : 10.44.0.10
Opaque decoded : [global-id 4]
Features : MBB RFWD
Upstream neighbor(s) :
Is CSI accepting : N
10.44.0.10:0 [Active] [MBB] Uptime: 01:27:50
Local Label (D) : 25515
Downstream client(s):
Recursive 0x00023 Uptime: 01:27:50
Check multicast PI > mrib entry/state for S, G with the help of the label:
RP/0/RSP1/CPU0:AG3-2#sh mrib mpls forwarding labels 25515 detail
Fri Mar 12 08:01:03.353 UTC
LSP information (mLDP) :
LSM-ID: 0x80023, Role: Mid >> Note LSM & Role
Incoming Label : 25515
Transported Protocol : <unknown>
Explicit Null : None
IP lookup : disabled
Platform information :
Slotmask: Primary: 0x4, Backup: 0xc
MGID Primary: 34318, Backup: 34319
Label Node Role: LMRIB_ROUTER_ROLE_MID, Tunnel Type: MVPN Tunnel
Outsegment Info #1 [M/Swap, Recursive]:
OutLabel: 29233, NH: 10.44.0.5, ID: 0x23, Sel IF: TenGigE0/0/0/3 >> verify out label with local label of 10.44.0.5
UL IF: TenGigE0/0/0/3, Node-ID: 0x8220
Backup Tunnel: Un:0x0 Backup State: Ready, NH: 0.0.0.0, MP Label: 0
Backup Sel IF: TenGigE0/1/0/0/2, UL IF: TenGigE0/1/0/0/2, Node-ID: 0x8320
PlatInfo:
Primary: [Main IF: TenGigE0/0/0/3 (IFH: 0x4000180), UL IF: TenGigE0/0/0/3 (IFH: 0x4000180)] >> Egress Interface/LC 0/0/CPU0
Backup: [Main IF: Bundle-Ether45 (IFH: 0x20002e0), UL IF: TenGigE0/1/0/0/2 (IFH: 0x60020c0)]
RP/0/RSP1/CPU0:AG3-2#
Check MGID with this command:
show mpls forwarding labels 25515 hardware egress location 0/0/CPU0
This MGID is in hex value and then converts to decimal.
Verify MGID:
RP/0/RSP1/CPU0:AG3-2#sh controllers mgidprgm mgidindex 34318 location 0/0/CPU0 >>> mgid value you got from "show mpls forwarding labels 25515 hardware egress location 0/0/CPU0"
Fri Mar 12 08:02:02.873 UTC
Device MGID-Bits Client-Last-Modified
=======================================================
XBAR-0 1 P2MP
FIA-0 1 P2MP
FIA-1 0 None
========================================================
Client Mask
========================================================
MFIBV4 0x0
MFIBV6 0x0
L2FIB 0x0
sRP-pseudo-mc 0x0
UT 0x0
Prgm-Svr 0x0
P2MP 0x1
xbar 0x0
UT1 0x0
UT2 0x0
punt_lib 0x0
RP/0/RSP1/CPU0:AG3-2#
Check forwarding rate on RSP and LC:
RP/0/RSP1/CPU0:AG3-2#sh mpls forwarding labels 25515 location 0/0/CPU0
Fri Mar 12 08:04:35.608 UTC
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
25515 29233 mLDP/IR: 0x00023 10.44.0.5 279353664958 >> Forwarding rate should increase
RP/0/RSP1/CPU0:AG3-2#sh mpls forwarding labels 25515 location 0/0/CPU0
Fri Mar 12 08:04:38.798 UTC
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
25515 29233 mLDP/IR: 0x00023 10.44.0.5 279353664958 >>> Forwarding rate should increase
Verify MPLS label programming related to S, G:
RP/0/RSP1/CPU0:AG3-2#sh mpls forwarding labels 25515 detail location 0/0/CPU0
Fri Mar 12 08:04:56.793 UTC
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
25515 mLDP/IR: 0x00023 (0x80023) >>> Verify LSM/LSP ID as per MLDP database
Updated Mar 12 06:27:29.419
mLDP/IR LSM-ID: 0x80023, MDT: 0x0
Flags:IP Lookup:not-set, Expnullv4:not-set, Expnullv6:not-set
Payload Type v4:not-set, Payload Type v6:not-set, l2vpn:not-set
Head:not-set, Tail:not-set, Bud:not-set, Peek:not-set, inclusive:not-set
Ingress Drop:not-set, Egress Drop:not-set
RPF-ID:0, Encap-ID:0
Disp-Tun:[ifh:0x0, label:-]
Platform Data [72]:
{ 0 0 0 4 0 0 0 4
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 12 0 0 0 12
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 134 14 0 0 134 15
0 0 134 14 0 0 134 15
0 0 0 1 0 0 0 2
}
mpls paths: 1, local mpls paths: 1, protected mpls paths: 1
29233 mLDP/IR: 0x00023 (0x80023) \ >>> verify LSM ID
10.44.0.5 280073746930
Updated: Mar 12 06:27:29.420
My Nodeid:0x8220
Interface Nodeids:
[ 0x8220 - - - - - - - - - ]
Interface Handles:
[ 0x4000180 - - - - - - - - - ]
Backup Interface Nodeids:
[ 0x8320 - - - - - - - - - ]
Backup Interface Handles:
[ 0x60020c0 - - - - - - - - - ]
via-label:25502, mpi-flags:0x200 tos_masks:[ primary:0x0 backup:0x0]
MPI resolution time Mar 12 06:27:29.420
Packets Switched: 581065865
Check BGP mVPN:
You can see Type 3 from the Source and Type 7 from the receiver.
RP/0/RSP1/CPU0:AG3-2#sh bgp ipv4 mvpn rd 1000:1001
Fri Mar 12 08:08:09.413 UTC
BGP router identifier 10.44.0.4, local AS number 55836
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0 RD version: 0
BGP main routing table version 11597
BGP NSR Initial initsync version 4 (Reached)
BGP NSR/ISSU Sync-Group versions 11597/0
BGP scan interval 60 secs
Status codes: s suppressed, d damped, h history, * valid, > best
i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1000:1001
*>i[1][10.44.0.5]/40 10.44.0.5 100 0 i
*>i[1][10.44.0.10]/40 10.44.0.10 100 0 i
*>i[1][10.44.0.110]/40
10.44.0.110 100 0 i
*>i[3][0][0.0.0.0][0][0.0.0.0][10.44.0.5]/120
10.44.0.5 100 0 i
*>i[3][0][0.0.0.0][0][0.0.0.0][10.44.0.10]/120
10.44.0.10 100 0 i
*>i[3][0][0.0.0.0][0][0.0.0.0][10.44.0.110]/120
10.44.0.110 100 0 i
*>i[3][32][10.1.0.2][32][232.11.11.11][10.44.0.10]/120
10.44.0.10 100 0 i
*>i[7][1000:1001][55836][32][10.1.0.2][32][232.11.11.11]/184
10.44.0.5 100 0 i
Processed 8 prefixes, 8 paths
Head End Router (Multicast Source PE)
Towards Core
Check MPLS MLDP Database:
RP/0/RSP0/CPU0:DC1-DCWAN#sh mpls mldp database root 10.44.0.10 opaquetype global-id 4
Fri Mar 12 05:27:57.124 EST
mLDP database
LSM-ID: 0x00009 Type: P2MP Uptime: 03:05:01 >> Verify LSM ID
FEC Root : 10.44.0.10 (we are the root)
Opaque decoded : [global-id 4]
Features : MBB RFWD
Upstream neighbor(s) : >>> Upstream is under VRF so it is none
None
Downstream client(s):
LDP 10.44.0.4:0 Uptime: 03:05:01 >>> Downstream client 10.44.0.4
Rec Next Hop : 10.44.0.4
Remote label (D) : 25515 >>> Verify this label with Mid router local label
PIM MDT Uptime: 03:05:01
Egress intf : LmdtMLDP-14
Table ID : IPv4: 0xe00003f8 IPv6: 0xe08003f8
HLI : 0x00009
Ingress : Yes
Local Label : 33031 (internal)
RP/0/RSP0/CPU0:DC1-DCWAN#
Check the egress core interface with the use of mrib:
RP/0/RSP0/CPU0:DC1-DCWAN#sh mrib mpls forwarding labels 33031 detail
Fri Mar 12 05:29:57.603 EST
LSP information (mLDP) :
LSM-ID: 0x00009, Role: Head, Head LSM-ID: 0x00009
Incoming Label : (33031) >>> Local Label
Transported Protocol : <unknown>
Explicit Null : None
IP lookup : disabled
Platform information :
Slotmask: Primary: 0x1, Backup: 0x1>>>>>> It gives egress LC0 as slotmask
MGID Primary: 26096, Backup: 26097 >>> Don't consider this MGID to verify the egress forwarding
Label Node Role: LMRIB_ROUTER_ROLE_INGRESS, Tunnel Type: MVPN Tunnel
Outsegment Info #1 [H/Push, Recursive]:
OutLabel: 25515, NH: 10.44.0.4, ID: 0x9, Sel IF: TenGigE0/0/1/2 >> Hash to one bundle member interface selected
UL IF: TenGigE0/0/1/2, Node-ID: 0x801
PlatInfo:
Primary: [Main IF: Bundle-Ether410 (IFH: 0x20009e0), UL IF: TenGigE0/0/1/2 (IFH: 0x140)]
Backup: [Main IF: NULL (IFH: 0x0), UL IF: NULL (IFH: 0x0)]
Alternate command:
RP/0/RSP0/CPU0:DC1-DCWAN#sh mrib mpls forwarding head-lsm-id 0x00009 detail
Fri Mar 12 05:44:46.639 EST
LSP information (mLDP) :
LSM-ID: 0x00009, Role: Head, Head LSM-ID: 0x00009
Incoming Label : (33031)
Transported Protocol : <unknown>
Explicit Null : None
IP lookup : disabled
Platform information :
Slotmask: Primary: 0x1, Backup: 0x1
MGID Primary: 26096, Backup: 26097
Label Node Role: LMRIB_ROUTER_ROLE_INGRESS, Tunnel Type: MVPN Tunnel
Outsegment Info #1 [H/Push, Recursive]:
OutLabel: 25515, NH: 10.44.0.4, ID: 0x9, Sel IF: TenGigE0/0/1/2
UL IF: TenGigE0/0/1/2, Node-ID: 0x801
PlatInfo:
Primary: [Main IF: Bundle-Ether410 (IFH: 0x20009e0), UL IF: TenGigE0/0/1/2 (IFH: 0x140)]
Backup: [Main IF: NULL (IFH: 0x0), UL IF: NULL (IFH: 0x0)]
Check mpls forwarding rate on RSP and LC - egress from DCWAN is MPLS forwarding:
RP/0/RSP0/CPU0:DC1-DCWAN#sh mpls forwarding labels 33031
Fri Mar 12 05:32:09.026 EST
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
33031 25515 mLDP/IR: 0x00009 10.44.0.4 545528292634 >> Verify forwarding rate & LSM
RP/0/RSP0/CPU0:DC1-DCWAN#sh mpls forwarding labels 33031 detail location 0/0/CPU0
Fri Mar 12 05:32:22.904 EST
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
33031 mLDP/IR: 0x00009 (0x00009)
Updated Mar 12 02:22:56.152
mLDP/IR LSM-ID: 0x00009, MDT: 0x0, Head LSM-ID: 0x00009
IPv4 Tableid: 0xe0000000, IPv6 Tableid: 0xe0800000
Flags:IP Lookup:not-set, Expnullv4:not-set, Expnullv6:not-set
Payload Type v4:not-set, Payload Type v6:not-set, l2vpn:not-set
Head:set, Tail:not-set, Bud:not-set, Peek:not-set, inclusive:not-set
Ingress Drop:not-set, Egress Drop:not-set
RPF-ID:0, Encap-ID:0
Disp-Tun:[ifh:0x0, label:-]
Platform Data [64]:
{ 0 0 0 1 0 0 0 1
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 1 0 0 0 1
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 101 240 0 0 101 241
0 0 0 1 0 0 0 1
}
mpls paths: 1, local mpls paths: 1, protected mpls paths: 0
25515 mLDP/IR: 0x00009 (0x00009) \
10.44.0.4 546248376534
Updated: Mar 12 02:22:56.150
My Nodeid:0x801
Interface Nodeids:
[ 0x801 - - - - - - - - - ]
Interface Handles:
[ 0x140 - - - - - - - - - ]
Backup Interface Nodeids:
[ - - - - - - - - - - ]
Backup Interface Handles:
[ - - - - - - - - - - ]
via-label:33009, mpi-flags:0x200 tos_masks:[ primary:0x0 backup:0x0]
MPI resolution time Mar 12 02:22:56.151
Packets Switched: 1133295387 >>> verify Traffic rate increase
RP/0/RSP0/CPU0:DC1-DCWAN#
Check CEF for next-hop shown in MLDP downstream:
RP/0/RSP0/CPU0:DC1-DCWAN#sh cef 10.44.0.4 internal
Fri Mar 12 05:39:40.543 EST
IPv4:default:0xe0000000[0x718462d8]:lsd:10.44.0.4/32[ref:1 proto:ipv4 flags:0x1000001 f2:0x0 src:lsd ver:376933][0x8abcce98]
LSPA => [ref:3 count:1 src:lsd flags:[pd create pending, updated, has-pd-li] walk-idx:0 flags:0xa20][0x727bc598]
0={
LABEL_INFO => [default [o-label:pop l-label:33009 type:0 flags:[linked to lsw, ipv4, recycle, retain] ext-flags:[none]][0x72789b68]
LABEL_INFO => [default [o-label:pop l-label:33009 type:0 flags:[owner locked, eos li, linked to lsw, ipv4, recycle, retain] ext-flags:[none]][0x727994a0]]]}
LDI => [ref:9 pl:75fe41a8 proto:mpls type:imp-eos1 lvl:1 buckets:1 slots:1 fixup:0 flags:[owner locked, added to pl, depth changed] timestamp:Mar 12 01:58:40.008][0x72811554]
slots:
{0,p0,s0,f:[]}={
NHINFO => [type:tx, linkt:link_IPv4, nh_proto:IPv4, ifh:Bundle-Ether410(0x20009e0), nh:10.16.74.0, adj:0x0, flags:[incomplete, pd bind pending]][0x73c97890]
IDB => [ref:1 ifh:Bundle-Ether410(0x20009e0) type:28/EtherBundle parent-type:0/none flags:[owner locked,proto-stats,cpuless-inserted,bundle][0x714d9030]
IDB_EXT:{MPLS IPv4:[ref:1 MPLS:[0x71a6c058] NH:2 MTU:1500 flags:[forwarding is enabled][0x75659058]],IPv4:[ref:1 IPv4:default[0x718462d8] NH:3 MTU:1500 flags:[none][0x723d3df0]],MPLS IPv6:NULL,IPv6:[ref:1 IPv6:default[0x71e63558] NH:3 MTU:1500 flags:[none][0x726ede28]]}
}
buckets:
{0,p0,s0}={[0x73c97890]}
LDI_LW => [type:imp loadinfo:[0x72811554] flags: timestamp:Mar 12 02:22:40.904][0x8aa0a3ac]
PATHLIST => [ref:12 count:1 depth:1 ldi:72811554 type:Shared flags:[mpls ext, imp, lsw] extn_flags:[] timestamp:Mar 12 01:58:40.004][0x75fe41a8]
0={
[[nh:10.16.74.0 ifh:Bundle-Ether410(0x20009e0) tbl:0xe0000000 type:implict-null flags:[none]]
[depth:1 flags:[resolved,ldi-preferred] resolves-via: ###
NHINFO => [type:tx, linkt:link_IPv4, nh_proto:IPv4, ifh:Bundle-Ether410(0x20009e0), nh:10.16.74.0, adj:0x0, flags:[incomplete, pd bind pending]][0x73c97890]
IDB => [ref:1 ifh:Bundle-Ether410(0x20009e0) type:28/EtherBundle parent-type:0/none flags:[owner locked,proto-stats,cpuless-inserted,bundle][0x714d9030]
IDB_EXT:{MPLS IPv4:[ref:1 MPLS:[0x71a6c058] NH:2 MTU:1500 flags:[forwarding is enabled][0x75659058]],IPv4:[ref:1 IPv4:default[0x718462d8] NH:3 MTU:1500 flags:[none][0x723d3df0]],MPLS IPv6:NULL,IPv6:[ref:1 IPv6:default[0x71e63558] NH:3 MTU:1500 flags:[none][0x726ede28]]}, with impnull-nh: **
NHINFO => [type:tx, linkt:link_MPLS, nh_proto:IPv4, ifh:Bundle-Ether410(0x20009e0), nh:10.16.74.0, adj:0x0, flags:[incomplete, pd bind pending]][0x73c97940]
IDB => [ref:1 ifh:Bundle-Ether410(0x20009e0) type:28/EtherBundle parent-type:0/none flags:[owner locked,proto-stats,cpuless-inserted,bundle][0x714d9030]
IDB_EXT:{MPLS IPv4:[ref:1 MPLS:[0x71a6c058] NH:2 MTU:1500 flags:[forwarding is enabled][0x75659058]],IPv4:[ref:1 IPv4:default[0x718462d8] NH:3 MTU:1500 flags:[none][0x723d3df0]],MPLS IPv6:NULL,IPv6:[ref:1 IPv6:default[0x71e63558] NH:3 MTU:1500 flags:[none][0x726ede28]]} ** ###]}
RP/0/RSP0/CPU0:DC1-DCWAN#
Verify Bundle member interfaces:
RP/0/RSP0/CPU0:DC1-DCWAN#sh int be310 | in Active
Wed Mar 10 06:37:16.756 EST
TenGigE0/0/0/0 Full-duplex 10000Mb/s Active
TenGigE0/0/0/1 Full-duplex 10000Mb/s Active
TenGigE0/0/0/4 Full-duplex 10000Mb/s Active
TenGigE0/0/0/5 Full-duplex 10000Mb/s Active
Check BCDL client status:
RP/0/RSP0/CPU0:AG2-1#sh mrib ipv4 client
Tue Aug 31 06:35:37.639 EDT
IP MRIB client-connections
bcdl_agent:1 (client id 28)
igmp:7467 (client id 23)
ldp:17503 (client id 52)
lmrib_bcdl:1 (client id 30) >>> Expected
lmrib_bcdl:2 (client id 60) >>>> Not expected if more than 1 >> fix provided Cisco bug ID CSCvy58964
pim:7471 (client id 26)
pim6_lbl:7644 (client id 20)
pim_lbl:7471 (client id 17)
xtc:7458 (client id 33)
Check mFIB:
RP/0/RSP0/CPU0:DC1-DCWAN#sh mfib vrf MLDP-14 platform route 10.1.0.2 232.11.11.11/32 location 0/0/CPU0
Fri Mar 12 09:39:34.651 EST
LC Type: A9K-MOD400-SE
--------------------------------------------------------------------------
Legend:
Common SW:
Route Information
B: BACL check S: RPF Interface signal
DC: Directly connected PL: Punt to LC CPU
PR: Punt to RP PF: Punt if forwarded
DR: Drop all RI: RPF interface
MF: MoFRR enabled TR: Tunnel route
TE: Tunnel encap CD: Conditional decap
MI: MVET index NI: Not Installed
BD: Bidir
MVPN Information (Common SW)
MLI: MVPN table index VTID: VRF Table ID
RPFID: RPF Identifier
MoFRR Information
A: Active RPF interface RI: RPF interface (or RPF ID)
WDI: Watchdog counter index
--------------------------------------------------------------------------
Source: 10.1.0.2 Group: 232.11.11.11 Mask: 64 RPF Int: BE1000.2001
Common SW Information
---------------------
MGID: 26081 MLI: 75537 Fabric Slotmask: 0x0 FGID: 0x0 >> Here no slotmask programmed as traffic is ingress under VRF & egress is MPLS forwarding
------------------------------------------------------------------------------
B S DC PL PR PF DR BD RI MF TR TE CD MI
------------------------------------------------------------------------------
F F F F F F F F 0x2013120 F F T F 0x2
---------------------------------------------------------------------------
MVPN/MLDP Encap Information:
Encap flag: T, Encap ID: 2 -->>>> Note this Encap ID
Last refcount bitmask (Enc ID): 0x0, Sets: 0, Clears: 0
Local receiver flag: F
Turnaround flag: F
Verify Encap ID:
RP/0/RSP0/CPU0:DC1-DCWAN#sh mfib encap identifier
Fri Mar 12 09:45:01.228 EST
Encap Id: 2
#Route: 1 #Encap: 1 Stale: N IP Lookup:N (MRIB Update VRF Lite(v4/v6):N/N #LocalEG:0 V6:N)
LSM ID: 9 Leaf Turnaround: True
RP/0/RSP0/CPU0:DC1-DCWAN#sh mfib encap identifier location 0/0/CPU0
Fri Mar 12 09:45:11.745 EST
Encap Id: 2
#Route: 1 #Encap: 1 Stale: N IP Lookup:N (MRIB Update VRF Lite(v4/v6):N/N #LocalEG:0 V6:N)
LSM ID: 9 Leaf Turnaround: True
Check BGP mVPN routes:
Check if BGP Type 7 is received from the receiver and Type 3 is advertised from Source to Receiver. BGP Type 3 is only advertised when data MDT is triggered.
RP/0/RSP0/CPU0:DC1-DCWAN#sh bgp ipv4 mvpn rd 1000:1001
Wed Mar 10 07:39:26.898 EST
BGP router identifier 10.44.0.10, local AS number 55836
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0 RD version: 0
BGP main routing table version 351943
BGP NSR Initial initsync version 31 (Reached)
BGP NSR/ISSU Sync-Group versions 351943/0
BGP scan interval 60 secs
Status codes: s suppressed, d damped, h history, * valid, > best
i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1000:1001 (default for vrf MLDP-14 )
*>i[1][10.44.0.5]/40 10.44.0.5 100 0 i
* i 10.44.0.5 100 0 i
*> [1][10.44.0.10]/40 0.0.0.0 0 i
*>i[1][10.44.0.110]/40
10.44.0.110 100 0 i
* i 10.44.0.110 100 0 i
*>i[3][0][0.0.0.0][0][0.0.0.0][10.44.0.5]/120
10.44.0.5 100 0 i
* i 10.44.0.5 100 0 i
*> [3][0][0.0.0.0][0][0.0.0.0][10.44.0.10]/120
0.0.0.0 0 i
*>i[3][0][0.0.0.0][0][0.0.0.0][10.44.0.110]/120
10.44.0.110 100 0 i
* i 10.44.0.110 100 0 i
*> [3][32][10.1.0.2][32][232.11.11.11][10.44.0.10]/120 >>>> Type 3 route advertised towards receiver
0.0.0.0 0 i
*>i[7][1000:1001][55836][32][10.1.0.2][32][232.11.11.11]/184 >>> Type 7 route received from receiver
10.44.0.5 100 0 i
* i 10.44.0.5 100 0 i
Processed 8 prefixes, 13 paths.
Towards Multicast Source
Check mRIB VRF:
RP/0/RSP0/CPU0:DC1-DCWAN#sh mrib vrf MLDP-14 route 10.1.0.2 232.11.11.11/32 detail
Fri Mar 12 05:17:51.828 EST
IP Multicast Routing Information Base
Entry flags: L - Domain-Local Source, E - External Source to the Domain,
C - Directly-Connected Check, S - Signal, IA - Inherit Accept,
IF - Inherit From, D - Drop, ME - MDT Encap, EID - Encap ID,
MD - MDT Decap, MT - MDT Threshold Crossed, MH - MDT interface handle
CD - Conditional Decap, MPLS - MPLS Decap, EX - Extranet
MoFE - MoFRR Enabled, MoFS - MoFRR State, MoFP - MoFRR Primary
MoFB - MoFRR Backup, RPFID - RPF ID Set, X - VXLAN
Interface flags: F - Forward, A - Accept, IC - Internal Copy,
NS - Negate Signal, DP - Don't Preserve, SP - Signal Present,
II - Internal Interest, ID - Internal Disinterest, LI - Local Interest,
LD - Local Disinterest, DI - Decapsulation Interface
EI - Encapsulation Interface, MI - MDT Interface, LVIF - MPLS Encap,
EX - Extranet, A2 - Secondary Accept, MT - MDT Threshold Crossed,
MA - Data MDT Assigned, LMI - mLDP MDT Interface, TMI - P2MP-TE MDT Interface
IRMI - IR MDT Interface
(10.1.0.2,232.11.11.11) Ver: 0x2c12 RPF nbr: 10.1.0.2 Flags: RPF MT EID,
PD: Slotmask: 0x0 >> No Slotmask as it is incoming traffic . It will get programmed when traffic is egress from node
MGID: 26081
MT Slot: 0/0/CPU0 I>>>> Ingress LC
Up: 02:55:10
RPF-ID: 0, Encap-ID: 2
Incoming Interface List
Bundle-Ether1000.2001 Flags: A, Up: 02:55:10 >> Multicast Source connected interface
Outgoing Interface List
LmdtMLDP-14 Flags: F LMI MT MA TR, Up: 02:55:10, Head LSM-ID: 0x00009 >>Verify LSM ID with MLDP dB & check egress interface
RP/0/RSP0/CPU0:DC1-DCWAN#
Verify MGID with ENCAP-ID:
RP/0/RSP0/CPU0:DC1-DCWAN#sh mrib encap-id 2
Wed Mar 17 12:50:58.421 EST
Encap ID Key : 00000110000000200100020100000000000000
Encap ID Length : 19
Encap ID Value : 2
Platform Annotation:
Slotmask: Primary: 0x1, Backup: 0x1 >>> Verify the Slot-mask point to LC 0
MGID: Primary: 43570, Backup: 43571 >>> Check this MGID
Flags (Vrflite(v4/v6),Stale,v6): N/N, N, N
Oles:
[1] type: 0x5, len: 12
LSM-ID: 0x00010 MDT: 0x2000120 Turnaround: TRUE
Primary: 0/0/CPU0[1]
Backup:
TableId: 0xe00003f8[1]
Redist History:
client id 22 redist time: 1d01h redist flags 0x0
MGID programming for egress LC (towards mdt):
RP/0/RSP0/CPU0:DC1-DCWAN#sh controllers mgidprgm mgidindex 43570 location 0/0/CPU0
Fri Mar 12 09:41:36.319 EST
Device MGID-Bits Client-Last-Modified
=======================================================
XBAR-0 10 P2MP
FIA-0 1 P2MP
FIA-1 1 P2MP
========================================================
Client Mask
========================================================
MFIBV4 0x0
MFIBV6 0x0
L2FIB 0x0
sRP-pseudo-mc 0x0
UT 0x0
Prgm-Svr 0x0
P2MP 0x3
xbar 0x0
UT1 0x0
UT2 0x0
punt_lib 0x0
Check mFIB VRF:
RP/0/RSP0/CPU0:DC1-DCWAN#sh mfib vrf MLDP-14 platform route 10.1.0.2 232.11.11.11/32 location 0/0/CPU0
Fri Mar 12 05:20:24.584 EST
LC Type: A9K-MOD400-SE
--------------------------------------------------------------------------
Legend:
Common SW:
Route Information
B: BACL check S: RPF Interface signal
DC: Directly connected PL: Punt to LC CPU
PR: Punt to RP PF: Punt if forwarded
DR: Drop all RI: RPF interface
MF: MoFRR enabled TR: Tunnel route
TE: Tunnel encap CD: Conditional decap
MI: MVET index NI: Not Installed
BD: Bidir
MVPN Information (Common SW)
MLI: MVPN table index VTID: VRF Table ID
RPFID: RPF Identifier
MoFRR Information
A: Active RPF interface RI: RPF interface (or RPF ID)
WDI: Watchdog counter index
--------------------------------------------------------------------------
Source: 10.1.0.2 Group: 232.11.11.11 Mask: 64 RPF Int: BE1000.2001 -- Ingress interface
Common SW Information
---------------------
MGID: 26081 MLI: 75537 Fabric Slotmask: 0x0 FGID: 0x0
------------------------------------------------------------------------------
B S DC PL PR PF DR BD RI MF TR TE CD MI
------------------------------------------------------------------------------
F F F F F F F F 0x2013120 F F T F 0x2
---------------------------------------------------------------------------
MVPN/MLDP Encap Information:
Encap flag: T, Encap ID: 2 >> Verify EnCap ID with mrib
Last refcount bitmask (Enc ID): 0x0, Sets: 0, Clears: 0
Local receiver flag: F
Turnaround flag: F
Check PIM:
RP/0/RSP0/CPU0:DC1-DCWAN#sh pim vrf MLDP-14 neighbor
PIM neighbors in VRF MLDP-14
Flag: B - Bidir capable, P - Proxy capable, DR - Designated Router,
E - ECMP Redirect capable
* indicates the neighbor created for this router
Neighbor Address Interface Uptime Expires DR pri Flags
10.44.0.10* LmdtMLDP-14 1d18h 00:01:18 1 (DR)
10.1.0.1* Bundle-Ether1000.2001 19:06:32 00:01:30 1 (DR) B E >> This router is DR
RP/0/RSP0/CPU0:DC1-DCWAN#sh pim vrf MLDP-14 topology 232.11.11.11 detail
Fri Mar 12 05:23:32.981 EST
IP PIM Multicast Topology Table
Entry state: (*/S,G)[RPT/SPT] Protocol Uptime Info
Entry flags: KAT - Keep Alive Timer, AA - Assume Alive, PA - Probe Alive
RA - Really Alive, IA - Inherit Alive, LH - Last Hop
DSS - Don't Signal Sources, RR - Register Received
SR - Sending Registers, SNR - Sending Null Registers
E - MSDP External, EX - Extranet
MFA - Mofrr Active, MFP - Mofrr Primary, MFB - Mofrr Backup
DCC - Don't Check Connected, ME - MDT Encap, MD - MDT Decap
MT - Crossed Data MDT threshold, MA - Data MDT Assigned
SAJ - BGP Source Active Joined, SAR - BGP Source Active Received,
SAS - BGP Source Active Sent, IM - Inband mLDP, X - VxLAN
Interface state: Name, Uptime, Fwd, Info
Interface flags: LI - Local Interest, LD - Local Dissinterest,
II - Internal Interest, ID - Internal Dissinterest,
LH - Last Hop, AS - Assert, AB - Admin Boundary, EX - Extranet,
BGP - BGP C-Multicast Join, BP - BGP Source Active Prune,
MVS - MVPN Safi Learned, MV6S - MVPN IPv6 Safi Learned
(10.1.0.2,232.11.11.11)SPT SSM Up: 03:00:51
JP: Join(now) RPF: Bundle-Ether1000.2001,10.1.0.2* Flags: MT MA >>>> MA Data MDT tunnel formed ..Ingress interface
Up: MT set (03:00:37) MDT: JoinSend Y, Cache N/N, Misc (0x0,0/0)
Cache: Add 03:00:37, Rem 1d18h. MT Cnt: Set 6, Unset 0. Joins sent 181
MDT-ifh 0x0/0x0, MT Slot none/ none
RPF-redirect BW usage: 0, Flags: 0x0, ObjID: 0x0
c-multicast-routing: PIM BGPJP: 1d18h
RPF Table: IPv4-Unicast-default
LmdtMLDP-14 03:00:51 fwd BGP >>> Signaling via BGP
Check PIM interface parameters:
RP/0/RSP0/CPU0:DC1-DCWAN#sh pim vrf MLDP-14 interface Bundle-Ether1000.2001 detail
Fri Mar 12 05:25:43.812 EST
PIM interfaces in VRF MLDP-14
IP PIM Multicast Interface State
Flag: B - Bidir enabled, NB - Bidir disabled
P - PIM Proxy enabled, NP - PIM Proxy disabled
V - Virtual Interface
BFD State - State/Interval/Multiplier
Interface PIM Nbr Hello DR
Count Intvl Prior
Bundle-Ether1000.2001 on 1 30 1 >> PIM ON Hello interval 30 sec
Primary Address : 10.1.0.1 >>> Self Address
Flags : B NP V
BFD : Off/150 ms/3
DR : this system >>> This DR
Propagation delay : 500
Override Interval : 2500
Hello Timer : 00:00:15
Neighbor Filter : -
PIM Errors/Stats:
show pim traffic
PIM Traffic Counters
Elapsed time since counters cleared: 1d01h
Received Sent
Valid PIM Packets 15759217 15214426 >>> verify the pim packets increment
Hello 9207 12336 >>> verify the pim hello Negotiations
Join-Prune 1076805 531981
Data Register 14673205 0
Null Register 73205 0
Register Stop 0 14673205
Assert 0 0
Batched Assert 0 0
Bidir DF Election 0 0
BSR Message 0 0 >>> If you have RP check this stats
Candidate-RP Adv. 0 0 >>> If you have RP check this stats
Join groups sent 0
Prune groups sent 0
Output JP bytes 0
Output hello bytes 410
Errors: >>>>> Check below paramters for errors or drops
Malformed Packets 0
Bad Checksums 0
Socket Errors 0
Subnet Errors 0
Packets dropped since send queue was full 0
Packets dropped due to invalid socket 0
Packets which couldn't be accessed 0
Packets sent on Loopback Errors 6
Packets received on PIM-disabled Interface 0
Log Collection
Collect these show techs related to multicast:
show tech-support multicast address-family ipv4
show tech-support multicast address-family ipv4 hardware
show tech-support multicast vrf <> address-family ipv4
show tech-support multicast vrf <> address-family ipv4 hardware
If you observe any NP for fabric issues or drops, capture this CLI:
show tech np
show tech fabric
Collect np datalogs:
- Get into LC shell (run
attach 0/x/cpu0) - Execute this command:
show_np –e <np_id> -d npdatalog
If you observe any process crash:
1. Upload core dump for that process file end with core.tgz and core.txt . The default file generated gets saved in harddisk:/
2. Upload show tech for that process. The Default file generated gets saved in harddisk:/showtech
If you observe any process blocked through CLI, use this command: show process blocked location all
Take:
- Follow process <JID> iteration 5 location <>
- Dumpcore running <JID/Process name> location <>
- This CLI generates dump core and uploads core dump for that process, files end with
core.tgzandcore.txt. - The default file generated gets saved in harddisk:/
- This CLI generates dump core and uploads core dump for that process, files end with
- Upload show tech for that process.
Revision History
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
03-May-2023 |
Initial Release |
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)