Troubleshoot Router Interface Issues

Introduction

 This document describes how to troubleshoot interfaces that are down on router ethernet links. 

Prerequisites

Requirements

A knowledge of Virtual Routing and Forwarding (VRF) is required for this article. 

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Background Information

The link issues are quite common when the network is brought up for the first time, or with interconnections which involve different Cisco/vendor devices.

Troubleshoot Methodology

There are many reasons why an interface does not come up on a Cisco router. Troubleshootingcan be tedious and painful when the failure happens in the field with too many variables - the problem could be due to Layer 1 - Small Form-Factor Pluggable (SFP)/cables/connectors/patch panel or layer 2 - the port on the router (either end or one end of the link). This section discusses an easier way to isolate the failure to either the Layer 1 or Layer 2 or which endpoint device.

Interface Link Parameters must Match at Both Ends of the Link

Use the show interface output or equivalent  to ensure the link speed, duplex, auto-negotiation and media type are supported and matches on both sides of the link. To find the supported transceivers for the interface module used, this is the link.

Cisco Gigabit Ethernet Transceiver Modules Compatibility Matrix

Configure the Interface in Local Loopback

This test verfies the packet path inside the router all the way till Physical Layer Interface Module (PLIM). If the router and port ASICs are properly initialized, the interface must come up and self ping must work. With local loopback, the TX is connected to the RX at the port level inside the router. This test excludes the SFP and the cables in the packet path. Failure of the port to come up, or ping itself with a local loop, indicates a problem with the Cisco Router. If this is the case, contact Cisco TAC to troubleshoot it further.

Test the Ports and Cables with VRF

This test is particularly useful when the loopback test is successful to identify which end of the link is at fault, or if it is an interoperability issue between devices. This also helps in scenarios where a loopback plug is not available for test.

Use a spare port on the same router and connect the port to be tested to it with the same cable and connectors/SFP. The router is now looped back to itself. Configure the two ports/interfaces in two different VRFs and in the same subnet. Ping from one VRF to the other VRF.

• If the Ping fails, it points to a cable or SFP problem. Change the cables and SFP.
• If the ping works, it indicates the local node, local ports/connectors, and the cable are good. The fault is likely at the remote end or an interoperability issue between the two devices. Do the same tests at the remote device, and then contact Cisco TAC to check for any compatibility issues between the two devices.

Troubleshoot Examples

These examples are pulled for an ethernet link on  ASR1000 router. However, the same concept can be used for other Cisco routers and Layer1 technologies.

Interface Link Parameters must Match at Both Ends of the Link

GigabitEthernet0/0/0 is up, line protocol is up 
Hardware is SPA-10X1GE-V2, address is 0023.33ee.7c00 (bia 0023.33ee.7c00)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, 
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not supported 
Full Duplex, 1000Mbps, link type is auto, media type is LX
output flow-control is off, input flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:01, output 00:02:31, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
314 packets input, 24637 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles 
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 314 multicast, 0 pause input
1 packets output, 77 bytes, 0 underruns
0 output errors, 0 collisions, 6 interface resets
17 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Configure the Interface in Local Loopback

Router#show ip interface brief | include GigabitEthernet0/0/1
GigabitEthernet0/0/1 192.168.0.1 YES manual down down 
Router#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
STLD1-630-03.04-ASR1(config)#interface gigabitEthernet 0/0/1
STLD1-630-03.04-ASR1(config-if)#loopback mac 
Loopback is a traffic-affecting operation
IPv6 self ping is not supported
STLD1-630-03.04-ASR1(config-if)#
*Aug 29 17:12:21.259: %LINK-3-UPDOWN: Interface GigabitEthernet0/0/1, changed state to up
*Aug 29 17:12:20.652: %LINK-3-UPDOWN: SIP0/0: Interface GigabitEthernet0/0/1, changed state to up
*Aug 29 17:12:22.259: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0/1, changed state to up
STLD1-630-03.04-ASR1(config-if)#
STLD1-630-03.04-ASR1(config-if)#end
Router#
*Aug 29 17:12:28.684: %SYS-5-CONFIG_I: Configured from console by console

Router#show run interface gigabitEthernet 0/0/1 
Building configuration...

Current configuration : 106 bytes
!
interface GigabitEthernet0/0/1
ip address 192.168.0.1 255.255.255.0
loopback mac
negotiation auto
end

Router#ping 192.168.0.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.0.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
Router#

Test the Ports and Cables with VRF

Router#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
Router(config)#vrf definition RED
Router(config-vrf)#address-family ipv4
Router(config-vrf-af)#exit
Router(config-vrf)#vrf definition BLUE 
Router(config-vrf)#address-family ipv4 
Router(config-vrf-af)#exit
Router(config-vrf)#end
*May 16 03:41:04.173: %SYS-5-CONFIG_I: Configured from console by consoleinter
Router#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
Router(config)#interface gigabitEthernet 0/0/3
Router(config-if)#vrf forwarding RED
Router(config-if)#ip address 10.1.1.1 255.255.255.252
Router(config-if)#cdp enable
Router(config-if)#interface gigabitEthernet 0/0/2   
Router(config-if)#vrf forwarding BLUE
Router(config-if)#ip address 10.1.1.2 255.255.255.252
Router(config-if)#cdp enable
Router(config-if)#end
*May 16 03:42:02.070: %SYS-5-CONFIG_I: Configured from console by console

Router#ping vrf BLUE 10.1.1.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

Router#ping vrf BLUE 10.1.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.1, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/1 ms

Router#ping vrf RED 10.1.1.1 
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

Router#ping vrf RED 10.1.1.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
Router#
Router#show run interface gigabitEthernet 0/0/2       
Building configuration...

Current configuration : 154 bytes
!
interface GigabitEthernet0/0/2
 description Looped to 0/0/3
 vrf forwarding BLUE
 ip address 10.1.1.2 255.255.255.252
 negotiation auto
 cdp enable
end

Router#show run interface gigabitEthernet 0/0/3
Building configuration...

Current configuration : 153 bytes
!
interface GigabitEthernet0/0/3
 description Looped to 0/0/2
 vrf forwarding RED
 ip address 10.1.1.1 255.255.255.252
 negotiation auto
 cdp enable
end

Router#