Troubleshoot Virtual Port-Channel (vPC) in ACI
Available Languages
Download Options
Bias-Free Language
The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
Contents
Introduction
This document describes how to identify and resolve problems that can occur with vPC in ACI.
Background Information
A virtual port channel (vPC) allows links that are physically connected to two different ACI leaf nodes to appear as a single port channel to a third device (that is, network switch, server, any other networking device that supports link aggregation technology).
vPCs consist of two ACI leaf switches designated as vPC peer switches. Of the vPC peers, one is primary and one is secondary. The system formed by the switches is referred to as a vPC domain.
No dedicated peer-link between the vPC peers; instead the fabric itself serves as the MCT.
• Peer Reachability protocol – ZMQ is utilized in lieu of CFS.
• ZMQ is an open-source high-performance messaging library that uses TCP as transport.
• This library is packaged as libzmq on the switch and linked into each application that needs to communicate with vPC peer.
Peer-reachability is not handled via a physical peer-link; instead, routing triggers are used to detect peer reachability.
• The vPC Manager registers with URIB for peer route notifications.
• When ISIS discovers a route to the peer, URIB notifies vPC manager, in turn attempts to open ZMQ socket with the peer.
• When the peer route is withdrawn by ISIS, the vPC manager is again notified by URIB, and it brings the MCT link down.
As part of upgrade best practices, it is recommended to upgrade switches in each pod in at least two separate groups so that half of leaf and spine nodes in each pod are up at any given time. An example is one group to have even numbered leaf and spine nodes, and another group to have odd numbered leaf and spines in each pod. With vPC configured devices we can make sure that at least one device is up during the upgrade by putting them in different groups. This prevents any outages during the upgrade because at least one device remains up while the other one is being upgraded.
Abbreviations
ACI: Application Centric Infrastructure
vPC : Virtual Port Channel
MCT: Multichassis EtherChannel Trunk
CFS: Cisco Fabric Services
ZMQ: Zero Messaging Queue
LACP: Link Aggregation Control Protocol
PDU: Protocol Data Unit
LAG: Link Aggregation
Prerequisite to Troubleshoot vPC Port-channels
For vPC configuration see
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/4-x/L2-configuration/Cisco-APIC-Layer2-Configuration-Guide-42x/Cisco-APIC-Layer2-Configuration-Guide-421_chapter_0111.html
vPC Validation
1. vPC Status : show vpc
FAB3-L1# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 101
Peer status : peer adjacency formed ok
vPC keep-alive status : Disabled
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 1
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled (timeout = 240 seconds)
Operational Layer3 Peer : Disabled
vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 up -
vPC status
----------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
-- ---- ------ ----------- ------ ------------
686 Po3 up success success 86
FAB3-L2# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 101
Peer status : peer adjacency formed ok
vPC keep-alive status : Disabled
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : secondary
Number of vPCs configured : 1
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled (timeout = 240 seconds)
Operational Layer3 Peer : Disabled
vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 up -
vPC status
----------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
-- ---- ------ ----------- ------ ------------
686 Po2 up success success 86
Output shows, peer adjacency is formed with vPC domain id 101, Note vPC keep alive status is disabled in ACI, because no dedicated link is needed. Po3 is UP in vPC with active vlan 86. Note that port-channel numbers can be different on vPC pair switches.
2. vPC Roles, vPC System mac and LAG ID: show vpc role
FAB3-L1# show vpc role
vPC Role status
----------------------------------------------------
vPC role : primary, operational secondary
Dual Active Detection Status : 0
vPC system-mac : 00:23:04:ee:be:65
vPC system-priority : 32667
vPC local system-mac : 00:81:c4:b1:25:4f
vPC local role-priority : 101
FAB3-L2# show vpc role
vPC Role status
----------------------------------------------------
vPC role : secondary, operational primary
Dual Active Detection Status : 0
vPC system-mac : 00:23:04:ee:be:65
vPC system-priority : 32667
vPC local system-mac : 00:5d:73:57:c4:2c
vPC local role-priority : 102This command shows that L1 is primary and L2 is secondary.
Because the end devices are connected to two different vPC switches there must be a mechanism for them to identify vPC peers as one logical device. This is achieved by use of vPC system mac in the LAG ID which is shared between the peers. This makes end device see vPC peers as one logical unit.
N3K# show lacp interface ethernet 1/24
Interface Ethernet1/24 is up
Channel group is 1 port channel is Po1
PDUs sent: 31726
PDUs rcvd: 31634
Markers sent: 0
Markers rcvd: 0
Marker response sent: 0
Marker response rcvd: 0
Unknown packets rcvd: 0
Illegal packets rcvd: 0
Lag Id: [ [(7f9b, 0-23-4-ee-be-65, 82ae, 8000, 4121), (8000, 0-a6-ca-75-6f-c1, 8000, 8000, 15d)] ]
Operational as aggregated link since Fri Sep 2 08:05:52 2022
Local Port: Eth1/24 MAC Address= 0-a6-ca-75-6f-c1
System Identifier=0x8000, Port Identifier=0x8000,0x15d
Operational key=32768
LACP_Activity=active
LACP_Timeout=Long Timeout (30s)
Synchronization=IN_SYNC
Collecting=true
Distributing=true
Partner information refresh timeout=Long Timeout (90s)
Actor Admin State=61
Actor Oper State=61
Neighbor: 0x4121
MAC Address= 0-23-4-ee-be-65
System Identifier=0x7f9b, Port Identifier=0x8000,0x4121
Operational key=33454
LACP_Activity=active
LACP_Timeout=Long Timeout (30s)
Synchronization=IN_SYNC
Collecting=true
Distributing=true
Partner Admin State=61
Partner Oper State=61
Aggregate or Individual(True=1)= 1
N3K# show lacp interface ethernet 1/25
Interface Ethernet1/25 is up
Channel group is 1 port channel is Po1
PDUs sent: 31666
PDUs rcvd: 31651
Markers sent: 0
Markers rcvd: 0
Marker response sent: 0
Marker response rcvd: 0
Unknown packets rcvd: 0
Illegal packets rcvd: 0
Lag Id: [ [(7f9b, 0-23-4-ee-be-65, 82ae, 8000, 111), (8000, 0-a6-ca-75-6f-c1, 8000, 8000, 161)] ]
Operational as aggregated link since Fri Sep 2 08:00:34 2022
Local Port: Eth1/25 MAC Address= 0-a6-ca-75-6f-c1
System Identifier=0x8000, Port Identifier=0x8000,0x161
Operational key=32768
LACP_Activity=active
LACP_Timeout=Long Timeout (30s)
Synchronization=IN_SYNC
Collecting=true
Distributing=true
Partner information refresh timeout=Long Timeout (90s)
Actor Admin State=61
Actor Oper State=61
Neighbor: 0x111
MAC Address= 0-23-4-ee-be-65
System Identifier=0x7f9b, Port Identifier=0x8000,0x111
Operational key=33454
LACP_Activity=active
LACP_Timeout=Long Timeout (30s)
Synchronization=IN_SYNC
Collecting=true
Distributing=true
Partner Admin State=61
Partner Oper State=61
Aggregate or Individual(True=1)= 1Output shows LAG ID (7f9b, 0-23-4-ee-be-65, 82ae, 8000, 4121) which is a combination of Priority as System ID (32667 in Hex), vPC system mac(00:23:04:ee:be:65), operational Key(33454 in Hex) and Port-identifier.
3. Port-channel Status : show port-channel extended
FAB3-L1# show port-channel extended
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
F - Configuration failed
-----------------------------------------------------------------------------
Group Port- BundleGrp Protocol Member Ports
Channel
-----------------------------------------------------------------------------
3 Po3(SU) 101-102 LACP Eth1/33(P)
Show port-channel extended shows more information about the state of physical links which are part of port-channel bundle.
4. TEP details and Logical Peer-link status : show system internal epm vpc
FAB3-L1# show system internal epm vpc
Local TEP IP : 10.3.208.64
Peer TEP IP : 10.3.208.67
vPC configured : Yes
vPC VIP : 10.3.16.67
MCT link status : Up
Local vPC version bitmap : 0x7
Peer vPC version bitmap : 0x7
Negotiated vPC version : 3
Peer advertisement received : Yes
Tunnel to vPC peer : Up
vPC# 686
if : port-channel3, if index : 0x16000002
local vPC state : MCEC_STATE_UP, peer vPC state : MCEC_STATE_UP
current link state : LOCAL_UP_PEER_UP
vPC fast conv : Off5. ZMQ Connection details: show system internal vpcm zmq statistics
FAB3-L1# show system internal vpcm zmq statistics
--------------------------------------------
MCECM ZMQ counters
----------------------------------------------
ZMQ server : 1
ZmQ: Registered ZmQ print callback
ZmQ: ====== Start ZMQ statistics printing ======
ZmQ: ZMQ socket type: 5, local ID: 40d0030a
ZmQ: Socket base 0x1109c3b4, #endpoints 1
ZmQ: Total 1 I/O pipes, CONNECT CNT: 0, DISCONNECT CNT: 0
ZmQ: RX CNT: 66, BYTES: 124132, ERRORS: 0
ZmQ: TX CNT: 66, BYTES: 125096, ERRORS: 0
ZmQ: Pipe tcp://10.3.208.64:5001 (ID: FD 54 flag 1 state 0): read 66 (124132 bytes) write 66 (125096 bytes) Peer I/O pipe: read 66 (125096 bytes) write 66 (124132 bytes)
ZmQ: Stream engine 0xae90049c ZMQ SOCKET 0x1109c3b4 TCP FD: 54 @ 10.3.208.67:58740
ZmQ: RX CNT: 72 BYTES: 124494 ERRORS: 0 TX CNT: 73 BYTES: 125458 ERRORS: 0
ZmQ: CONNECT CNT: 0 DISCONNECT CNT: 0
ZmQ: ====== End ZMQ statistics printing ======ZMQ statistics show state of the ZMQ session, number of times connection, occurrences of disconnections, and any errors occurred.
Troubleshoot VPC Port-channel Issues
1. Physical port is down
FAB3-L1# show vpc brief
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 101
Peer status : peer adjacency formed ok
vPC keep-alive status : Disabled
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 1
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled (timeout = 240 seconds)
Operational Layer3 Peer : Disabled
vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 up -
vPC status
----------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
-- ---- ------ ----------- ------ ------------
686 Po3 down* success success
Output shows Po3 is down.
FAB3-L1# show port-channel summary
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
F - Configuration failed
-------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
-------------------------------------------------------------------------------
3 Po3(SD) Eth LACP Eth1/33(D) We further look at the state of interfaces which are part of the port-channel. Here Eth1/33 is in Down state. LACP is configured as the bundling protocol.
FAB3-L1# show int e1/33
Ethernet1/33 is down (notconnect)
admin state is up, Dedicated Interface
Belongs to po3
Hardware: 100/1000/10000/auto Ethernet, address: 0081.c4b1.2521 (bia 0081.c4b1.2521)
MTU 9000 bytes, BW 0 Kbit, DLY 1 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 10 Gb/s
FEC (forward-error-correction) : disable-fec
Beacon is turned off
Auto-Negotiation is turned on
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped 00:08:15
Last clearing of "show interface" counters never
9 interface resets
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 0 bps, 0 pps; output rate 0 bps, 0 ppsshow interface output gives more details about interface e1/33. We can see E1/33 is down with notconnect state.
Recommended Action:
Make sure that the port is connected properly and has the correct configuration.
2. Suspend by LACP
FAB3-L1# show port-channel extended
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
F - Configuration failed
-----------------------------------------------------------------------------
Group Port- BundleGrp Protocol Member Ports
Channel
-----------------------------------------------------------------------------
3 Po3(SD) 101-102 LACP Eth1/33(s)
Output shows Eth1/33 is in suspended state. Next we look at show interface Eth1/33 for more details.
FAB3-L1# show int e1/33
Ethernet1/33 is down (suspended-due-to-no-lacp-pdus)
admin state is up, Dedicated Interface
Belongs to po3
Hardware: 100/1000/10000/auto Ethernet, address: 0081.c4b1.2521 (bia 0081.c4b1.2521)
MTU 9000 bytes, BW 0 Kbit, DLY 1 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 10 Gb/s
FEC (forward-error-correction) : disable-fec
Beacon is turned off
Auto-Negotiation is turned on
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped 00:00:13
Last clearing of "show interface" counters never
12 interface resets
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 1640 bits/sec, 0 packets/secshow interface suggests that port is suspended because of no LACP PDUs. We can further look at LACP counters and identify if LACP PDUs are being sent and received.
FAB3-L1# show lacp counters interface port-channel 3
LACPDUs Marker Marker Response LACPDUs
Port Sent Recv Sent Recv Sent Recv Pkts Err
---------------------------------------------------------------------
port-channel3
Ethernet1/33 314 264 0 0 0 0 0
FAB3-L1#
FAB3-L1#
FAB3-L1# show lacp counters interface port-channel 3
LACPDUs Marker Marker Response LACPDUs
Port Sent Recv Sent Recv Sent Recv Pkts Err
---------------------------------------------------------------------
port-channel3
Ethernet1/33 315 264 0 0 0 0 0 Output shows that the counter is only incrementing for Sent LACPDUs and Recv counter remains constant. This suggests that we did not receive LACP PDU from the remote end.
We can also look at LACP Negotiation parameters, Counters, and so on, for specific interface use "show lacp interface e1/33".
FAB3-L1# show lacp interface e1/33
Interface Ethernet1/33 is suspended
Channel group is 3 port channel is Po3
PDUs sent: 317
PDUs rcvd: 264 received
Markers sent: 0
Markers rcvd: 0
Marker response sent: 0
Marker response rcvd: 0
Unknown packets rcvd: 0
Illegal packets rcvd: 0
Lag Id: [ [(7f9b, 00-23-04-ee-be-65, 82ae, 8000, 121), (0, 0-0-0-0-0-0, 0, 0, 0)] ]
Operational as aggregated link since Mon Aug 22 09:29:53 2022
Local Port: Eth1/33 MAC Address= 00-81-c4-b1-25-4f
System Identifier=0x8000,00-81-c4-b1-25-4f
Port Identifier=0x8000,0x121
Operational key=33454
LACP_Activity=active
LACP_Timeout=Long Timeout (30s)
Synchronization=NOT_IN_SYNC
Collecting=false
Distributing=false
Partner information refresh timeout=Long Timeout (90s)
Actor Admin State=(Ac-1:To-0:Ag-1:Sy-0:Co-0:Di-0:De-1:Ex-0)
Actor Oper State=Ac-1:To-0:Ag-1:Sy-0:Co-0:Di-0:De-1:Ex-0
Neighbor: 0x0
MAC Address= 0-0-0-0-0-0
System Identifier=0x0,0x0
Port Identifier=0x0,0x0
Operational key=0
LACP_Activity=unknown
LACP_Timeout=Long Timeout (30s)
Synchronization=NOT_IN_SYNC
Collecting=false
Distributing=false
Partner Admin State=(Ac-0:To-0:Ag-0:Sy-0:Co-0:Di-0:De-0:Ex-0)
Partner Oper State=(Ac-0:To-0:Ag-0:Sy-0:Co-0:Di-0:De-0:Ex-0)
Aggregate or Individual(True=1)= 2
Futher a packet capture can also be done on the leaf for LACP packets. You can use specific filters to filter out the interface in question.
tcpdump -vvvi kpm_inb ether proto 0x8809Recommended Action:
Make sure that LACP is configured properly on the remote side and the device sends LACP PDUs on correct interface.
3. Suspend by vPC
FAB3-L1# show vpc brief
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 101
Peer status : peer adjacency formed ok
vPC keep-alive status : Disabled
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 1
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled (timeout = 240 seconds)
Operational Layer3 Peer : Disabled
vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 up -
vPC status
----------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
-- ---- ------ ----------- ------ ------------
686 Po3 down* failed vpc port channel
mis-config due to
vpc links in the 2
switches connected
to different
partners
This output shows that vPC port-channel is down because of a vPC misconfig. Observe the port-channel status.
FAB3-L1# show port-channel summary
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
F - Configuration failed
-------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
-------------------------------------------------------------------------------
3 Po3(SD) Eth LACP Eth1/33(D)
Here Eth1/33 is in Down state. Observe 'show interface e1/33' for more details.
FAB3-L1# show int e1/33
Ethernet1/33 is down (suspend-by-vpc)
admin state is up, Dedicated Interface
Belongs to po3
Hardware: 100/1000/10000/auto Ethernet, address: 0081.c4b1.2521 (bia 0081.c4b1.2521)
MTU 9000 bytes, BW 0 Kbit, DLY 1 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 10 Gb/s
FEC (forward-error-correction) : disable-fec
Beacon is turned off
Auto-Negotiation is turned on
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Switchport monitor is off
EtherType is 0x8100vPC uses LAG ID to determine if the vPC peers are connected to the same host. If there is a mismatch in the LAG ID, interfaces are suspended by vPC.
"Show vpc brief" shows that physical links in the port-channel on vPC peers are not connected to the same remote device.
LAG ID comparison can be checked with "show vpc consistency-parameters interface port-channel 3".
FAB3-L1# show vpc consistency-parameters interface port-channel 3
Type 1 : vPC will be suspended in case of mismatch
Name Type Local Value Peer Value
------------- ---- ---------------------- -----------------------
lag-id 1 [(7f9b, [(7f9b,
0-23-4-ee-be-65, 82ae, 0-23-4-ee-be-68, 82ae,
0, 0), (8000, 0, 0), (8000,
0-a6-ca-75-6f-c1, 0-a6-ca-75-6f-c1,
8000, 0, 0)] 8000, 0, 0)]
mode 1 active active
Speed 1 10 Gb/s 10 Gb/s
Duplex 1 full full
Port Mode 1 trunk trunk
Native Vlan 1 0 0
MTU 1 9000 9000
vPC card type 1 Empty Empty
Allowed VLANs - 86 86
Local suspended VLANs - - -
If there is a mismatch in the LAG-ID, ports are suspended.
Recommended Action:
Make sure that the physical links in the port-channel are connected to the same remote device.
4. LACP Suspend Individual
LACP sets a port to the suspended state if it does not receive an LACP PDU from the peer. This can cause some servers to fail to boot up as they require LACP to logically bring-up the port. You can tune behavior to individual use by disabling LACP suspend individual.
To do so, create a port channel policy in your vPC policy group, and after setting mode to LACP active, remove Suspend Individual Port. Now the ports in the vPC stay active and continue to send LACP packets.
FAB3-L1# show port-channel extended
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
b - BFD Session Wait
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
F - Configuration failed
-----------------------------------------------------------------------------
Group Port- BundleGrp Protocol Member Ports
Channel
-----------------------------------------------------------------------------
1 Po1(SD) 101-102 LACP Eth1/33(I)
Output shows that even though we did not receive LACP PDUs on Eth1/33 after LACP Suspend-Individual flag is removed, port is UP as Individual port. Note that we still send LACP PDUs from ACI leaf with this configuration. When LACP PDUs are received, the port moves back to bundled mode.
Other Errors
There are other interface errors which are not specific to vPC but are still applicable to vPC interfaces. Please refer to the links for details.
1. mcp-loop-err-disable
2. bpdu-guard-err-disable
Revision History
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
31-Oct-2022 |
Initial Release |
Contributed by Cisco Engineers
- Savinder SinghTAC
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)