Troubleshoot Port Flaps on Catalyst 9000 Series Switches

Introduction

This document describes how to identify, collect useful logs, and troubleshoot problems that can occur with Port Flaps on Catalyst 9000 switches.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on all Catalyst 9000 Series switches.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

This article was contributed by  Leonardo Pena Davila.

A port flap, usually referred to as a link flap, is a situation in which a physical interface on the switch continually goes up and down. The common cause is usually related to bad, unsupported, or non-standard cable or Small Form-Factor Pluggable (SFP) or related to other link synchronization issues. The cause for the link flaps can be intermittent or permanent.

Since link flaps tends to be a physical interference, this document explains the steps to diagnose, collect useful logs and troubleshoot problems that can occur with port flaps on Catalyst 9000 switches.

Troubleshoot

There are a number of things you can check If you have physical access to the switch to ensure the Network Modules, Cables, SFP are properly installed:

Network Modules Installation

The table describes the best practices to install  a Network Module in a Catalyst 9000 series switch:

Verify Cable and Both Sides of the Connection

These tables describe some of the possible cable issues that can cause link flaps. 

Verify SFP and SFP+ Compatibility

The Cisco portfolio of hot pluggable interfaces offers a rich set of choices in terms of speeds, protocols, reaches and supported transmission media.

You can use any combination of SFP or SFP + transceiver modules that your Catalyst 9000 Series switches device supports. The only restrictions are that each port must match the wavelength specifications on the other end of the cable and that the cable must not exceed the stipulated cable length for reliable communications.

Use only Cisco SFP transceiver modules on your Cisco device. Each SFP or SFP+  transceiver module supports the Cisco Quality Identification (ID) feature which allows a Cisco switch or router to identify and validate that the transceiver module is certified and tested by Cisco.

Tip: Refer to this link in order to verify the Cisco Optics-to-Device Compatibility Matrix

Identify Port Flaps

Use the show loggingcommand to identify a link flap event. This example shows a partial switch system log message for a link flap event with the interface TenGigabitEthernet1/0/40:

Switch#show logging | include changed
Aug 17 21:06:08.431 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/0/40, changed state to down
Aug 17 21:06:39.058 UTC: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/0/40, changed state to down
Aug 17 21:06:41.968 UTC: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/0/40, changed state to up
Aug 17 21:06:42.969 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/0/40, changed state to up
Aug 17 21:07:20.041 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/0/40, changed state to down
Aug 17 21:07:21.041 UTC: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/0/40, changed state to down
Aug 17 21:07:36.534 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/0/40, changed state to up
Aug 17 21:08:06.598 UTC: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/0/40, changed state to up
Aug 17 21:08:07.628 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/0/40, changed state to down
Aug 17 21:08:08.628 UTC: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/0/40, changed state to down
Aug 17 21:08:10.943 UTC: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/0/40, changed state to up
Aug 17 21:08:11.944 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/0/40, changed state to up

Tip: If you analyze the system message logs, you must pay attention to  the timestamp of the port flap, because it allows you to compare simultaneous events on that specifc port and validate whether or not the link flap ocurrence is expected (For example: sleep setting or other "normal" casue not necessarily an issue).

Interface Show Commands 

The show interface command gives you a lot of information that helps to identify a possible Layer 1 issue that causes a link flap event:

Switch#show interfaces tenGigabitEthernet 1/0/40
TenGigabitEthernet1/0/40 is up, line protocol is up (connected) 
Hardware is Ten Gigabit Ethernet, address is 00a5.bf9c.29a8 (bia 00a5.bf9c.29a8)
  MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-SR   <-- SFP plugged into the port
  input flow-control is on, output flow-control is unsupported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:03, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0  
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     670 packets input, 78317 bytes, 0 no buffer              
     Received 540 broadcasts (540 multicasts)
     0 runts, 0 giants, 0 throttles                           
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored     
     0 watchdog, 540 multicast, 0 pause input                 
     0 input packets with dribble condition detected          
     1766 packets output, 146082 bytes, 0 underruns           
     0 Output 0 broadcasts (0 multicasts)
     0 output errors, 0 collisions, 0 interface resets        
     0 unknown protocol drops                                 
     0 babbles, 0 late collision, 0 deferred                  
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out   

This table lists some of the counters from the show interface command:

The history command allows an interface to maintain utilization history in a graphical format similar to CPU history. This history can be maintained as either bit per second (bps) or packets per second (pps) as you can see in this example:

Switch(config-if)#history ?
 bps Maintain history in bits/second
 pps Maintain history in packets/second

Along with the rate, the user can monitor various interface counters:

Switch(config-if)#history [bps|pps] ?
  all                                Include all counters
  babbles                            Include ethernet output babbles - Babbl
  crcs                               Include CRCs - CRCs 
  deferred                           Include ethernet output deferred - Defer
  dribbles                           Include dribbles - Dribl
  excessive-collisions               Include ethernet excessive output collisions -
                                     ExCol
  flushes                            Include flushes - Flush
  frame-errors                       Include frame errors - FrErr
  giants                             Include giants - Giant
  ignored                            Include ignored - Ignor
  input-broadcasts                   Include input broadcasts - iBcst
  input-drops                        Include input drops - iDrop
  input-errors                       Include input errors - iErr
  interface-resets                   Include interface resets - IRset
  late-collisions                    Include ethernet late output collisions - LtCol
  lost-carrier                       Include ethernet output lost carrier - LstCr
  multi-collisions                   Include ethernet multiple output collisions -
                                     MlCol
  multicast                          Include ethernet input multicast - MlCst
  no-carrier                         Include ethernet output no-carrier - NoCarr
  output-broadcasts                  Include output broadcasts - oBcst
  output-buffer-failures             Include output buffer failures - oBufF
  output-buffers-swapped-out         Include output buffers swapped out - oBSwO
  output-drops                       Include output drops - oDrop
  output-errors                      Include output errors - oErr
  output-no-buffer                   Include output no buffer - oNoBf
  overruns                           Include overruns - OvrRn
  pause-input                        Include ethernet input pause - PsIn
  pause-output                       Include ethernet output pause - PsOut
  runts                              Include runts - Runts
  single-collisions                  Include ethernet single output collisions - SnCol
  throttles                          Include throttles - Thrtl
  underruns                          Include underruns - UndRn
  unknown-protocol-drops             Include unknown protocol drops - Unkno
  watchdog                           Include ethernet output watchdog - Wtchdg
  <cr>                               <cr>
SW_1(config-if)#

As with CPU history, there are graphs for the last 60 seconds, last 60 minutes and last 72 hours. Separate graphs are maintained for input and output histograms:

Switch#sh interfaces gigabitEthernet 1/0/2 history ?
 60min    Display 60 minute histograms only
 60sec    Display 60 second histograms only
 72hour   Display 72 hour histograms only
 all      Display all three histogram intervals
 both     Display both input and output histograms
 input    Display input histograms only
 output   Display output histograms only
 |        Output modifiers
 <cr> <cr>

------ Sample output ---------

Switch#show interfaces tenGigabitEthernet 1/0/9 history 60sec  



   10 
    9 
    8 
    7 
    6 
    5 
    4 
    3 
    2 
    1 
    0....5....1....1....2....2....3....3....4....4....5....5....6
              0    5    0    5    0    5    0    5    0    5    0
     TenGigabitEthernet1/0/9 input rate(mbits/sec) (last 60 seconds)


   10 
    9 
    8 
    7 
    6 
    5 
    4 
    3 
    2 
    1 
    0....5....1....1....2....2....3....3....4....4....5....5....6
              0    5    0    5    0    5    0    5    0    5    0
     TenGigabitEthernet1/0/9 output rate(mbits/sec) (last 60 seconds)

Use the show controllers ethernet-controller{interface{interface-number}}  to display per-interface (Transmit and Receive) traffic counters and errors counters  statistics read from the hardware. Use the phy keyword to display the interface internal registers or the port-info keyword to display information about the port ASIC.

This is an example of output from the show controllers ethernet-controller for a specific interface:

Switch#show controllers ethernet-controller tenGigabitEthernet 2/0/1
Transmit                  TenGigabitEthernet2/0/1               Receive                 
        61572 Total bytes                      282909 Total bytes              
            0 Unicast frames                      600 Unicast frames           
            0 Unicast bytes                     38400 Unicast bytes            
          308 Multicast frames                   3163 Multicast frames         
        61572 Multicast bytes                  244509 Multicast bytes          
            0 Broadcast frames                      0 Broadcast frames         
            0 Broadcast bytes                       0 Broadcast bytes          
            0 System FCS error frames               0 IpgViolation frames      
            0 MacUnderrun frames                    0 MacOverrun frames        
            0 Pause frames                          0 Pause frames             
            0 Cos 0 Pause frames                    0 Cos 0 Pause frames       
            0 Cos 1 Pause frames                    0 Cos 1 Pause frames       
            0 Cos 2 Pause frames                    0 Cos 2 Pause frames       
            0 Cos 3 Pause frames                    0 Cos 3 Pause frames       
            0 Cos 4 Pause frames                    0 Cos 4 Pause frames       
            0 Cos 5 Pause frames                    0 Cos 5 Pause frames       
            0 Cos 6 Pause frames                    0 Cos 6 Pause frames       
            0 Cos 7 Pause frames                    0 Cos 7 Pause frames       
            0 Oam frames                            0 OamProcessed frames      
            0 Oam frames                            0 OamDropped frames        
          193 Minimum size frames                3646 Minimum size frames      
            0 65 to 127 byte frames                 1 65 to 127 byte frames    
            0 128 to 255 byte frames                0 128 to 255 byte frames   
          115 256 to 511 byte frames              116 256 to 511 byte frames   
            0 512 to 1023 byte frames               0 512 to 1023 byte frames  
            0 1024 to 1518 byte frames              0 1024 to 1518 byte frames 
            0 1519 to 2047 byte frames              0 1519 to 2047 byte frames 
            0 2048 to 4095 byte frames              0 2048 to 4095 byte frames 
            0 4096 to 8191 byte frames              0 4096 to 8191 byte frames 
            0 8192 to 16383 byte frames             0 8192 to 16383 byte frames
            0 16384 to 32767 byte frame             0 16384 to 32767 byte frame
            0 > 32768 byte frames                   0 > 32768 byte frames      
            0 Late collision frames                 0 SymbolErr frames         <-- Usually indicates Layer 1 issues. Large amounts of symbol errors can indicate a bad device, cable, or hardware. 
            0 Excess Defer frames                   0 Collision fragments      <-- If this counter increments, this is an indication that the ports are configured at half-duplex.
            0 Good (1 coll) frames                  0 ValidUnderSize frames    
            0 Good (>1 coll) frames                 0 InvalidOverSize frames   
            0 Deferred frames                       0 ValidOverSize frames     
            0 Gold frames dropped                   0 FcsErr frames           <-- Are the result of collisions at half-duplex, a duplex mismatch, bad hardware (NIC, cable, or port)
            0 Gold frames truncated    
            0 Gold frames successful   
            0 1 collision frames                    
            0 2 collision frames       
            0 3 collision frames       
            0 4 collision frames       
            0 5 collision frames       
            0 6 collision frames       
            0 7 collision frames       
            0 8 collision frames       
            0 9 collision frames       
            0 10 collision frames      
            0 11 collision frames      
            0 12 collision frames      
            0 13 collision frames      
            0 14 collision frames      
            0 15 collision frames      
            0 Excess collision frames  

LAST UPDATE 22622 msecs AGO

Tip: You can also use the show interfaces {interface{interface-number}}  controller  command to display per-interface Transmit and Receive  statistics read from the hardware.

Use the show platform pm interface-flaps{interface{interface-number}}  to display the number of times an interface got down:

This is an example of output from the show platform pm interface-flaps{interface{interface-number}}for a specific interface:

Switch#show platform pm interface-flaps tenGigabitEthernet 2/0/1 

  Field                     AdminFields          OperFields
===============================================================
  Access Mode               Static               Static
  Access Vlan Id            1                    0
  Voice Vlan Id             4096                 0
  VLAN Unassigned                                0
  ExAccess Vlan Id          32767                
  Native Vlan Id            1                    
  Port Mode                 dynamic              access
  Encapsulation             802.1Q               Native
  disl                      auto                 
  Media                     unknown              
  DTP Nonegotiate           0                    0
  Port Protected            0                    0
  Unknown Unicast Blocked   0                    0
  Unknown Multicast Blocked 0                    0
  Vepa Enabled              0                    0
  App interface             0                    0
  Span Destination          0                    

  Duplex                    auto                 full 
  Default Duplex            auto                 
  Speed                     auto                 1000
  Auto Speed Capable        1                    1
  No Negotiate              0                    0
  No Negotiate Capable      1024                 1024
  Flow Control Receive      ON                   ON
  Flow Control Send         Off                  Off
  Jumbo                     0                    0
  saved_holdqueue_out       0                    
  saved_input_defqcount     2000                 
  Jumbo Size                1500                 

  Forwarding Vlans : none
  Current Pruned Vlans : none
  Previous Pruned Vlans : none


  Sw LinkNeg State : LinkStateUp
  No.of LinkDownEvents :    12                            <--  Number of times the interface flapped    
  XgxsResetOnLinkDown(10GE):
  Time Stamp Last Link Flapped(U) : Aug 19 14:58:00.154   <-- Last time the interface flapped
  LastLinkDownDuration(sec) 192                           <-- Time in seconds the interface stayed down during the last flap event      
  LastLinkUpDuration(sec):  2277                          <-- Time in seconds the interface stayed up before the last flap event  

Use the show idprom{interface{interface-number}}  command without keywords to display the IDPROM information for the specific interface. Use with the detail keyword to display detailed hexadecimal IDPROM information.

This is an example of output from the show idprom{interface{interface-number}} for a specific interface. The High and  Low Warning|Alarm thersholds values listed in this command output are the normal operational optical transceiver parameters. Those values can be verify from the data sheet for the specific optic. Please refer to the  Cisco Optics Datasheet 

Switch#show idprom interface Twe1/0/1 

IDPROM for transceiver TwentyFiveGigE1/0/1 :
  Description                               = SFP or SFP+ optics (type 3)
  Transceiver Type:                         = GE CWDM 1550 (107)
  Product Identifier (PID)                  = CWDM-SFP-1550  <-- 
  Vendor Revision                           = A
  Serial Number (SN)                        = XXXXXXXXXX   <-- Cisco Serial Number
  Vendor Name                               = CISCO-FINISAR
  Vendor OUI (IEEE company ID)              = 00.90.65 (36965)
  CLEI code                                 = CNTRV14FAB
  Cisco part number                         = 10-1879-03
  Device State                              = Enabled.
  Date code (yy/mm/dd)                      = 14/12/22
  Connector type                            = LC.
  Encoding                                  = 8B10B (1)
  Nominal bitrate                           = OTU-1 (2700 Mbits/s)
  Minimum bit rate as % of nominal bit rate = not specified
  Maximum bit rate as % of nominal bit rate = not specified
  The transceiver type is 107
  Link reach for 9u fiber (km)              = LR-2(80km) (80)
                                              LR-3(80km) (80)
                                              ZX(80km) (80)
  Link reach for 9u fiber (m)               = IR-2(40km) (255)
                                              LR-1(40km) (255)
                                              LR-2(80km) (255)
                                              LR-3(80km) (255)
                                              DX(40KM) (255)
                                              HX(40km) (255)
                                              ZX(80km) (255)
                                              VX(100km) (255)
  Link reach for 50u fiber (m)              = SR(2km) (0)
                                              IR-1(15km) (0)
                                              IR-2(40km) (0)
                                              LR-1(40km) (0)
                                              LR-2(80km) (0)
                                              LR-3(80km) (0)
                                              DX(40KM) (0)
                                              HX(40km) (0)
                                              ZX(80km) (0)
                                              VX(100km) (0)
                                              1xFC, 2xFC-SM(10km) (0)
                                              ESCON-SM(20km) (0)
  Link reach for 62.5u fiber (m)            = SR(2km) (0)
                                              IR-1(15km) (0)
                                              IR-2(40km) (0)
                                              LR-1(40km) (0)
                                              LR-2(80km) (0)
                                              LR-3(80km) (0)
                                              DX(40KM) (0)
                                              HX(40km) (0)
                                              ZX(80km) (0)
                                              VX(100km) (0)
                                              1xFC, 2xFC-SM(10km) (0)
                                              ESCON-SM(20km) (0)
  Nominal laser wavelength                  = 1550 nm.
  DWDM wavelength fraction                  = 1550.0  nm.
  Supported options                         = Tx disable
                                              Tx fault signal
                                              Loss of signal (standard implementation)
  Supported enhanced options                = Alarms for monitored parameters
  Diagnostic monitoring                     = Digital diagnostics supported
                                              Diagnostics are externally calibrated
                                              Rx power measured is "Average power"
  Transceiver temperature operating range   = -5 C to 75 C (commercial)
  Minimum operating temperature             = 0 C
  Maximum operating temperature             = 70 C
  High temperature alarm threshold          = +90.000 C
  High temperature warning threshold        = +85.000 C
  Low temperature warning threshold         = +0.000 C
  Low temperature alarm threshold           =  -4.000 C
  High voltage alarm threshold              = 3600.0 mVolts
  High voltage warning threshold            = 3500.0 mVolts
  Low voltage warning threshold             = 3100.0 mVolts
  Low voltage alarm threshold               = 3000.0 mVolts
  High laser bias current alarm threshold   = 84.000 mAmps
  High laser bias current warning threshold = 70.000 mAmps
  Low laser bias current warning threshold  = 4.000 mAmps
  Low laser bias current alarm threshold    = 2.000 mAmps
  High transmit power alarm threshold       =  7.4 dBm
  High transmit power warning threshold     =  4.0 dBm
  Low transmit power warning threshold      = -1.7 dBm
  Low transmit power alarm threshold        = -8.2 dBm
  High receive power alarm threshold        = -3.0 dBm
  Low receive power alarm threshold        = -33.0 dBm
  High receive power warning threshold        = -7.0 dBm
  Low receive power warning threshold        = -28.2 dBm
  External Calibration: bias current slope  = 1.000
  External Calibration: bias current offset = 0

Tip:Ensure the hardware and software version of the device are compatible with the SFP/SFP+ installed  Cisco Optics-to-Device Compatibility Matrix

This table list the various commands that can be used to troubleshoot link flaps:

Verify Cable Status with Time Domain Reflector (TDR)

The Time Domain Reflectometer (TDR) feature allows you to determine if a cable is OPEN or SHORT when it is at fault. With TDR, you can check the status of copper cables for the ports on the Catalyst 9000 Series Switches. TDR detects a cable fault with a signal that is sent  through the cable and read the signal that is reflected back. All or part of the signal can be reflected back due to defects in the cable

Use the test cable-diagnostics tdr {interface{interface-number} }to start the TDR test, then use the show cable-diagnostics tdr{interfaceinterface-number}. 

Tip: Refer to the Checking Port Status and Connectivity for further details

The example shows a TDR test result for interface Tw2/0/10:

Switch#show cable-diagnostics tdr interface tw2/0/10
TDR test last run on: November 05 02:28:43
Interface Speed Local pair  Pair length         Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Tw2/0/10 1000M Pair A       1     +/- 5 meters  Pair A      Impedance Mismatch
               Pair B       1     +/- 5 meters  Pair B      Impedance Mismatch
               Pair C       1     +/- 5 meters  Pair C      Open
               Pair D       3     +/- 5 meters  Pair D      Open

Tip: On Catalyst 9300 Series Switches, only these cable fault types are detected - OPEN, SHORT, and IMPEDANCE MISMATCH. The Normal status is displayed in case cable is properly terminated and this is done for illustrative purpose.

TDR Guidelines

This guidelines apply to the use of TDR:

• Do not change the port configuration while the TDR test is running.
• If you connect a port during a TDR test to an Auto-MDIX enabled port, the TDR result can be invalid.
• If you connect a port during a TDR test to a 100BASE-T port such as that on the device, the unused pairs (4-5 and 7-8) are reported as faulty because the remote end does not terminate these pairs.
• Due to cable characteristics, you must run the TDR test multiple times to get accurate results.
• Do not change port status (for example, remove the cable at the near or far end) because the results can be inaccurate.
• TDR works best if the test cable is disconnected from the remote port. Otherwise, it can be difficult for you to interpret results correctly.
• TDR operates across four wires. Based on on the cable conditions, the status can show that one pair is OPEN or SHORT while all other wire pairs display as faulty. This operation is acceptable because you can declare a cable faulty provided one pair of wires is either OPEN or SHORT.
• TDR intent is to determine how poorly a cable functions rather than to locate a faulty cable.
• When TDR locates a faulty cable, you can still use an offline cable diagnosis tool to better diagnose the problem.
• TDR results can differ between runs on different switch models of Catalyst 9300 Series Switches because of the resolution difference of TDR implementations. When this occurs, you must refer to an offline cable diagnosis tool.

Digital Optic Monitoring (DOM)

Digital Optical Monitoring (DOM) is an industry wide standard, intended to define a digital interface to access real-time parameters such as:

• Temperature
• Transceiver supply voltage
• Laser bias current
• Optical Tx power
• Optical Rx power

How to Enable DOM

The table list the commands you can used to turn on/off DOM for all transceivers type in the system:

Use the show interfaces {interface{interface-number}} transceiver detail command to display transceiver information:

Switch#show interfaces hundredGigE 1/0/25 transceiver detail 
ITU Channel not available (Wavelength not available),
Transceiver is internally calibrated.
mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.
++ : high alarm, + : high warning, - : low warning, -- : low alarm.
A2D readouts (if they differ), are reported in parentheses.
The threshold values are calibrated.

                                  High Alarm  High Warn  Low Warn   Low Alarm
               Temperature        Threshold   Threshold  Threshold  Threshold
Port           (Celsius)          (Celsius)   (Celsius)  (Celsius)  (Celsius)
---------      ----------------- ----------   ---------   --------- ---------
Hu1/0/25       28.8                   75.0       70.0        0.0       -5.0

                                 High Alarm   High Warn  Low Warn   Low Alarm
              Voltage            Threshold    Threshold  Threshold  Threshold
Port          (Volts)            (Volts)      (Volts)    (Volts)    (Volts)
---------     ----------------- ----------    ---------  ---------  ---------
Hu1/0/25      3.28                   3.63        3.46      3.13         2.97

                                 High Alarm   High Warn  Low Warn   Low Alarm
                Current          Threshold    Threshold  Threshold  Threshold
Port      Lane  (milliamperes)   (mA)         (mA)       (mA)       (mA)
--------- ----  ---------------  ----------   ---------  ---------  ---------
Hu1/0/25   N/A   6.2                  10.0        8.5        3.0        2.6

                 Optical         High Alarm   High Warn  Low Warn   Low Alarm
                 Transmit Power  Threshold    Threshold  Threshold  Threshold
Port      Lane   (dBm)           (dBm)        (dBm)      (dBm)      (dBm)
--------- ----   --------------- ----------   ---------  ---------  ---------
Hu1/0/25  N/A    -2.2                 1.7        -1.3       -7.3      -11.3

                 Optical          High Alarm  High Warn   Low Warn   Low Alarm
                 Receive Power    Threshold   Threshold   Threshold  Threshold
Port      Lane   (dBm)            (dBm)       (dBm)       (dBm)      (dBm)
--------- ----   ---------------  ----------  ---------   ---------  ---------
Hu1/0/25  N/A    -16.7                2.0        -1.0        -9.9      -13.9
 

Tip: In order to determine if an optical transceiver operates at the appropriate signal levels, please refer to the Cisco Optics Datasheet

Digital Optic Monitoring Syslog Messages

This section describes the most relevant threshold violation syslog messages:

Temperature Levels of SFP optics

• Explanation: This log messages is  generated when temperature is low or exceeds the normal optic operate values:

%SFF8472-3-THRESHOLD_VIOLATION: Te7/3: Temperature high alarm; Operating value: 88.7 C, Threshold value: 74.0 C.
%SFF8472-3-THRESHOLD_VIOLATION: Fo1/1/1: Temperature low alarm; Operating value: 0.0 C, Threshold value: 35.0 C.

Voltage Levels of SFP optics

• Explanation: This log messages is  generated when voltage is low or exceeds the normal optic operate values:

%SFF8472-3-THRESHOLD_VIOLATION: Gi1/1/3: Voltage high warning; Operating value: 3.50 V, Threshold value: 3.50 V.
%SFF8472-5-THRESHOLD_VIOLATION: Gi1/1: Voltage low alarm; Operating value: 2.70 V, Threshold value: 2.97 V.

Light Levels of SFP optics

• Explanation: This log messages is  generated when the light power is low or exceeds the optic operate values:

%SFF8472-3-THRESHOLD_VIOLATION: Gi1/0/1: Rx power high warning; Operating value: -2.7 dBm, Threshold value: -3.0 dBm.
%SFF8472-5-THRESHOLD_VIOLATION: Te1/1: Rx power low warning; Operating value: -13.8 dBm, Threshold value: -9.9 dBm.

Tip: For further information on DOM see Digital Optical Monitoring

Cisco Optics and Forward Error Correction (FEC)

FEC is a technique used to detect and correct a certain number of errors in a bitstream and appends redundant bits and error-checking code to the message block before transmission. As a module manufacturer, Cisco takes care to design our transceivers to comply with specifications. When the optical transceiver is operated in a Cisco host platform, the FEC is enabled by default based on the optical module type that the host software detects (See this downloadable table). In the vast majority of cases, the FEC implementation is dictated by the industry standard that the optic type supports. 

For certain custom specifications, FEC implementations vary. Refer to Understanding FEC and its Implementation in Cisco Optics  document for detailed information.

The example shows how to configure FEC and some of the available options:

switch(config-if)#fec?
  auto Enable FEC Auto-Neg
  cl108 Enable clause108 with 25G
  cl74 Enable clause74 with 25G
  off Turn FEC off

Use the show interface command to verify FEC configuration:

TwentyFiveGigE1/0/13 is up, line protocol is up (connected)
  Hardware is Twenty Five Gigabit Ethernet, address is 3473.2d93.bc8d (bia 3473.2d93.bc8d)
  MTU 9170 bytes, BW 25000000 Kbit/sec, DLY 10 usec,
      reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 25Gb/s, link type is force-up, media type is SFP-25GBase-SR
  Fec is auto     < -- The configured setting for FEC is displayed here
  input flow-control is on, output flow-control is off
  ARP type: ARPA, ARP Timeout 04:00:00 
--snip--

Note: Both sides of a link must have the same FEC encoding algorithm enabled for the link to come up.

Debug Commands

This table lists the various commands that can be used to debug Port Flaps

Caution: Use the debug commands with caution. Please be aware many debug commands have an impact on live network and only are recommended to use in a lab environment when the issue is reproduced.                                                        ;

This is a partial sample output  example of the debug commands listed in the table:

SW_2#sh debugging 
PM (platform):          
 L2 Control Infra debugging is on   <-- debug platform pm l2-control
 PM Link Status debugging is on     <-- debug platform pm link-status
 PM Vectors debugging is on         <-- debug platform pm pm-vectors
Packet Infra debugs:

Ip Address Port
------------------------------------------------------|----------

Port Manager:
  Port events debugging is on        <-- debug pm port

Condition 1: interface Te1/0/2 (1 flags triggered)
Flags: Te1/0/2

------ Sample output ---------

*Aug 25 20:01:05.791: link up/down event : link-down on Te1/0/2
*Aug 25 20:01:05.791: pm_port 1/2: during state access, got event 5(link_down)   <-- Link down event (day/time)
*Aug 25 20:01:05.791: @@@ pm_port 1/2: access -> pagp
*Aug 25 20:01:05.792: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:05.792: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:05.792: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:05.792: IOS-FMAN-PM-DEBUG-PM-VECTORS: Vp Disable: pd=0x7F1E797914B0 dpidx=10 Te1/0/2
*Aug 25 20:01:05.792: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:05.792: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:05.792: Maintains count of VP per Interface:delete, pm_vp_counter[0]: 14, pm_vp_counter[1]: 14
*Aug 25 20:01:05.792: *** port_modechange: 1/2 mode_none(10)
*Aug 25 20:01:05.792: @@@ pm_port 1/2: pagp -> dtp
*Aug 25 20:01:05.792: stop flap timer : Te1/0/2 pagp
*Aug 25 20:01:05.792: *** port_bndl_stop: 1/2 : inform yes
*Aug 25 20:01:05.792: @@@ pm_port 1/2: dtp -> present
*Aug 25 20:01:05.792: *** port_dtp_stop: 1/2
*Aug 25 20:01:05.792: stop flap timer : Te1/0/2 pagp
*Aug 25 20:01:05.792: stop flap timer : Te1/0/2 dtp
*Aug 25 20:01:05.792: stop flap timer : Te1/0/2 unknown
*Aug 25 20:01:05.792: *** port_linkchange: reason_link_change(3): link_down(0)1/2  <-- State link change 
*Aug 25 20:01:05.792: pm_port 1/2: idle during state present
*Aug 25 20:01:05.792: @@@ pm_port 1/2: present -> link_down    <-- State of the link
*Aug 25 20:01:06.791: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/0/2, changed state to down
*Aug 25 20:01:07.792: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/0/2, changed state to down
*Aug 25 20:01:11.098: IOS-FMAN-PM-DEBUG-LINK-STATUS: Received LINKCHANGE in xcvr message, if_id 10 (TenGigabitEthernet1/0/2)

*Aug 25 20:01:11.098: IOS-FMAN-PM-DEBUG-LINK-STATUS: if_id 0xA, if_name Te1/0/2, link up <-- Link became up
*Aug 25 20:01:11.098: link up/down event: link-up on Te1/0/2
*Aug 25 20:01:11.098: pm_port 1/2: during state link_down, got event 4(link_up)
*Aug 25 20:01:11.098: @@@ pm_port 1/2: link_down -> link_up
*Aug 25 20:01:11.098: flap count for link type : Te1/0/2 Linkcnt = 0 
*Aug 25 20:01:11.099: pm_port 1/2: idle during state link_up
*Aug 25 20:01:11.099: @@@ pm_port 1/2: link_up -> link_authentication
*Aug 25 20:01:11.099: pm_port 1/2: during state link_authentication, got event 8(authen_disable)
*Aug 25 20:01:11.099: @@@ pm_port 1/2: link_authentication -> link_ready
*Aug 25 20:01:11.099: *** port_linkchange: reason_link_change(3): link_up(1)1/2 
*Aug 25 20:01:11.099: pm_port 1/2: idle during state link_ready
*Aug 25 20:01:11.099: @@@ pm_port 1/2: link_ready -> dtp
*Aug 25 20:01:11.099: IOS-FMAN-PM-DEBUG-PM-VECTORS: Set pm vp mode attributes for Te1/0/2 vlan 1
*Aug 25 20:01:11.099: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:11.099: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:11.099: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:11.099: pm_port 1/2: during state dtp, got event 13(dtp_complete)
*Aug 25 20:01:11.099: @@@ pm_port 1/2: dtp -> dtp
*Aug 25 20:01:11.099: IOS-FMAN-PM-DEBUG-PM-VECTORS: Set pm vp mode attributes for Te1/0/2 vlan 1
*Aug 25 20:01:11.099: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:11.099: DTP flapping: flap count for dtp type: Te1/0/2 Dtpcnt = 0 
*Aug 25 20:01:11.099: pm_port 1/2: during state dtp, got event 110(dtp_done)
*Aug 25 20:01:11.099: @@@ pm_port 1/2: dtp -> pre_pagp_may_suspend
*Aug 25 20:01:11.099: pm_port 1/2: idle during state pre_pagp_may_suspend
*Aug 25 20:01:11.099: @@@ pm_port 1/2: pre_pagp_may_suspend -> pagp_may_suspend
*Aug 25 20:01:11.099: pm_port 1/2: during state pagp_may_suspend, got event 33(pagp_continue)
*Aug 25 20:01:11.099: @@@ pm_port 1/2: pagp_may_suspend -> start_pagp
*Aug 25 20:01:11.099: pm_port 1/2: idle during state start_pagp
*Aug 25 20:01:11.099: @@@ pm_port 1/2: start_pagp -> pagp
*Aug 25 20:01:11.100: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:11.100: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:11.100: IOS-FMAN-PM-DEBUG-PM-VECTORS: Set pm vp mode attributes for Te1/0/2 vlan 1
*Aug 25 20:01:11.100: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:11.100: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:11.100: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:11.100: *** port_bndl_start: 1/2
*Aug 25 20:01:11.100: stop flap timer : Te1/0/2 pagp
*Aug 25 20:01:11.100: pm_port 1/2: during state pagp, got event 34(dont_bundle)
*Aug 25 20:01:11.100: @@@ pm_port 1/2: pagp -> pre_post_pagp
*Aug 25 20:01:11.100: pm_port 1/2: idle during state pre_post_pagp
*Aug 25 20:01:11.100: @@@ pm_port 1/2: pre_post_pagp -> post_pagp
*Aug 25 20:01:11.100: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:11.100: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:11.100: pm_port 1/2: during state post_pagp, got event 14(dtp_access)
*Aug 25 20:01:11.100: @@@ pm_port 1/2: post_pagp -> access
*Aug 25 20:01:11.100: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:11.100: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:11.100: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:11.100: IOS-FMAN-PM-DEBUG-PM-VECTORS: Set pm vp mode attributes for Te1/0/2 vlan 1
*Aug 25 20:01:11.100: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:11.100: Maintains count of VP per Interface:add, pm_vp_counter[0]: 15, pm_vp_counter[1]: 15
*Aug 25 20:01:11.100: IOS-FMAN-PM-DEBUG-PM-VECTORS: vlan vp enable for port(Te1/0/2) and vlan:1
*Aug 25 20:01:11.101: IOS-FMAN-PM-DEBUG-PM-VECTORS: VP ENABLE: vp_pvlan_port_mode:access for Te1/0/2 
*Aug 25 20:01:11.101: IOS-FMAN-PM-DEBUG-PM-VECTORS: VP Enable: vp_pvlan_native_vlanId:1 for Te1/0/2 
*Aug 25 20:01:11.101: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:11.101: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:11.101: *** port_modechange: 1/2 mode_access(1)
*Aug 25 20:01:11.101: IOS-FMAN-PM-DEBUG-PM-VECTORS: The operational mode of Te1/0/2 in set all vlans is 1 
*Aug 25 20:01:11.101: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:11.101: IOS-FMAN-PM-DEBUG-PM-VECTORS: vp_pvlan port_mode:access vlan:1 for Te1/0/2 
*Aug 25 20:01:11.101: IOS-FMAN-PM-DEBUG-PM-VECTORS: vp_pvlan port_mode:access native_vlan:1 for Te1/0/2 
*Aug 25 20:01:11.102: IOS-FMAN-PM-DEBUG-PM-VECTORS: Success sending PM tdl message
*Aug 25 20:01:13.098: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/0/2, changed state to up
*Aug 25 20:01:14.098: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/0/2, changed state to up

Related Cisco Bugs

Related Information

Cisco Optics-to-Device Compatilbility Matrix

Cisco SFP Modules for Gigabit Ethernet Applications Data Sheet

25GE and 100GE – Enabling Higher Speeds in Enterprise with Investment Protection White Paper

Cisco CWDM SFP Solution Data Sheet

Support Innovation: How Cisco TAC is transforming documentation and simplifying self-service

Cisco Technical Support & Downloads