Troubleshoot Power over Ethernet on Catalyst 9000 Switches

Introduction

This document describes how to troubleshoot Power over Ethernet (PoE) on Catalyst 9000 PoE-capable switching platforms. 

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Catalyst 9000 Series switches

• Power over Ethernet 

Components Used

This document is not restricted to specific software and hardware versions. PoE is supported on PoE capable switch and line card models in Catalyst 9200, Catalyst 9300 and Catalyst 9400 product family. The example outputs in this document are based on a number of software and hardware versions from the Catalyst 9000 product family.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

Catalyst 9000 switches support different kinds of PoE standards.

• Pre-standard PoE devices are detected & classified at layer 1 by switch the Physical Layer Device (PHY), power is supplied at a default level, and/or higher power levels are negotiated with Cisco Discovery Protocol (CDP).
• IEEE 802.3af (PoE) and 802.3at (PoE+) powered devices (PDs) are detected by a PoE controller on Catalyst 9000 switch/line card (sometimes there are more than one), and appropriate power level can be classified prior to power on via IEEE classification, as well as negotiated later.
• Cisco UPoE (Universal Power over Ethernet) device detection and classification functions just like standards-based, but the 30W limit is raised to 60W by negotiation (post-power-up) through Link Layer Discovery Protocol (LLDP) to obtain inline power over additional wires.
• Cisco UPoE+ is based on IEEE 802.03bt & is available on selective Catalyst 9000 products and can provide up to 90W per port.

PoE switch models

• Catalyst 9000 switches and line cards with "P" in their product ID support PoE+ on a group of ports or all ports. For example, C9200L-48P-4G, C9200-24P, C9300-48P, C9400-LC-48P and so on.
• Catalyst 9000 switches and line cards with "U" in their product ID support UPoE on a group of ports or all ports. For example, C9300-24U, C9400-LC-48UX, and so on.
• Catalyst 9000 switches and line cards with "H" in their product ID support UPoE+ on a group of ports or all ports. For example, C9300-48H, C9400-LC-48H and so on.

Note: PoE capability alone does not guarantee PoE assignment. Refer to the data sheet for other restrictions & requirements like supported port range, needed power supplies and minimum software version and so on.

Terminology

• PoE - Power over Ethernet
• PoE+ - PoE+ standard increases maximum power that can be drawn by a powered device from 15.4 to 30W per port
• UPoE - Universal PoE. Cisco proprietary technology that extends the IEEE 802.at PoE standard to provide the capability to source up to 60W of power over per port
• IF_ID - Interface Identifier, internal unique value that represents a specific interface
• Platform Manager - Internal software component in Cisco IOS® XE
• Chassis Manager - Internal software component in Cisco IOS XE
• IOMD - Input Output Module Driver. Internal software component in Cisco IOS XE
• MCU - Micro Controller Unit
• PD - Powered Device (IP Phones, Access Points, Cameras and so on.)
• PSE - Power Sourcing Equipment, like a PoE capable Catalyst 9000 switch.

PoE Class

Standards-based Cisco PoE equipment conforms to the IEEE standards for five power classifications for powered devices. When Cisco PoE switch detects a powered device and grants a power request, the switch can adjust the power budget (available power) in accordance to powered-device IEEE classification.

PoE classes describe a range of power used by a specific powered device. Some powered devices require more power than others, and power classes allowed switches to manage a power budget or available power. When a powered device is detected and its class is identified, the switch allocates (reserves) the appropriate power range.

The switch can determine the IEEE power class of the powered device through application of 20V DC to the line and then measurement of the resultant current flow. IEEE-compliant powered devices produce a very specific current flow in response to the 20 VDC applied by the switch.

PoE LEDs on switchports

This table explains the meaning of LED color status on the switch.

General Troubleshooting Guidelines

Verify environmental conditions and symptoms

• Does the powered device (PD) in question not power up at all, or does it power up briefly and then power down?
• Did problem start during initial installation, or did it begin a period of time where the device worked normally?
• If problem started after the powered device worked normally, what changed? Were there any hardware or software changes? Any environmental changes (temperature, humidity, air flow, and so on)? Any electrical changes (maintenance, outage, interference, and so on)?
• Did anything occur in the local network when the problem occurred? Use the  show logging  command to review the switch log and Simple Network Management Protocol (SNMP) traps (if configured). If so, could it be related to another problem specific to that local network?
• Does the problem happen at a specific time of day or night? If so, are there any known environmental/electrical changes at that particular time/day?
• Was any network event noticed at the same time? A traffic flood, storm, loop, increased network congestion, higher than normal resource utilization (CPU, interfaces, and so on) could lead to temporary loss of connectivity between PD and another network element, which can cause the PD to reboot.

Verify specifics around powered device and switch

• What type of device is  used (Cisco legacy, 802.3af, 802.3at, UPOE?), and does the Catalyst 9000 variant in question support this type?
• Is enough inline power available from the power supply on respective switch member/line card?
• Do all ports of one switch member/line card not provide PoE or only few?
• What about ports on different PoE controllers on the same switch/line card? (Non-UPoE models have 4 ports per controller and UPoE has 2 ports per controller)
• Is more than one line card/switch member of a chassis/stack affected?
• Do only newly connected ports not provide PoE, and are already-connected ports operate OK on same switch member/line card? 
• If one of the already-connected port (PoE status OK) on same switch member/line card is bounced (shut/no shut), does the PoE functionality break or continue to work fine?
• Is data connectivity affected or it is just the PoE functionality?
• Is the problem restricted to one type/model of PD?
• Any PoE syslog messages seen?
• What are the switch models, types of linecards and PDs used?
• Does show power inline [detail] accurately reflect power status on the port?

Common PoE Problems

No PoE on Only One Port

Step 1: Verify that the powered device works on other ports and that the problem is only on one port.

Step 2: Use the show run  and show interface status commands to verify that the port is not shut down or err-disabled.

Step 3: Use the show run command to verify that the power inline never interface is not configured on the port.

Step 4: Verify that the ethernet cable from the phone to the switch port is good. Connect a known good non-PoE ethernet device (like a computer) and use the same ethernet cable to a known port that works, and make sure that it establishes a link and exchanges traffic with another host. If needed, replace the cable.

Step 5: Verify that the total cable length from the switch front panel to the powered device is not more than 100 meters. 100m includes the length of cable between two ends of patch panel (if in use).

Step 6: If patch panel is in use, connect the powered device directly to the switchport to rule out a problem with patch panel.

Step 7: If Ethernet cable is fairly long (> 50 m), disconnect the cable from the switch port. Use a shorter Ethernet cable to connect a known good data only device (like a computer) to this switch. Verify that the device establishes a data only Ethernet link and exchanges traffic with another host, or ping the IP address of switch VLAN SVI. Next, connect a powered device to this port, and see if it powers on.

Step 8: Use the show power inline and show power inline detail commands to compare the number of connected powered devices against the switch power budget (available PoE). Verify that switch power budget can power the device.

Step 9: Move to Advanced Troubleshooting section for advanced PoE troubleshooting and data collection.

No PoE on All or Group of Ports

Step 1: Use the show interface status command to verify that the ports are not shut down and not error disabled.

Step 2:  Use the show environment all, show interface status, and show power inline commands to review power status if no powered device on any port can power on. Use the show log command to review alarms reported earlier by system messages. If you see an unusual state against the power supplies, focus on that first.

Step 3: If the trouble is on all ports, the PoE section of the power supply can be defective if the switch works normally except for PoE and if non-PoE devices can establish a data ethernet link on any port. If the trouble is on a consecutive group of ports but not all ports, there could be a defective PoE subsection in the switch.

Step 4: Check logs with the command show logging. Common PoE logs are described later. If there are any logs seen from this section, interpret the information gathered and take appropriate steps.

Step 5: Bounce the interface connected to switch port. If that does not help, try a switch reload by removal of the power cord, wait for 15 seconds and power is provided to the switch again.

Step 6: Watch out for any diagnostic failures during/after bootup.

Cisco Powered Device does not Work on Cisco Poe Switch

When a functional Cisco IP Phone, Cisco wireless access point or another Cisco powered device intermittently reloads or disconnects from inline power do these steps:

Step 1: Verify all electrical connections from the switch to the powered device. Any unreliable connection results in power interruptions and intermittent powered device operations, such as powered device disconnects and reloads.

Step 2: Verify that the total cable length from the switch front panel to the powered device inclusive of patch panel (if in use) is not more than 100 meters.

Step 3: Notice what changed in the electrical environment at the switch site. What is happened at the powered device when the disconnect occurs?

Step 4: Use the show log command to review syslog and events. Examine syslog timestamps to see whether any other error messages are reported by the switch at the same time that a disconnect occurs.

Step 5: Verify that a Cisco IP Phone does not loose connectivity to the call manager immediately before the reload occurs. It can be a network problem, not a PoE problem. This can be determined by SPAN capture on the switch port while powered device disconnects and analysis of the capture file.

Step 6: If powered device allows PoE debugs or packet capture, turn them on for additional troubleshooting data points.

Step 7: Connect a non-PoE device to the port, and verify that it works. If a non-PoE device has link problems or a high error rate, the problem can be an unreliable cable connection between the switch port and the user.

Non-Cisco Powered Device does not Work on Cisco PoE Switch

When a non-Cisco powered device is connected to a Cisco PoE switch, but never powers up, or powers up and then quickly disconnects from power (powers down). Non-PoE devices work normally do these steps:

Step 1: Use the show power inline command to verify that the switch power budget (available PoE) is not depleted before or after the powered device is connected. Verify that sufficient power is available for the powered device type.

Step 2: Use the show interface status command to verify that the powered device is detected by the switch when connected.

Step 3: Use the show logging command to verify that the powered device does not cause a controller error on the port. If this occurs, it would be highlighted in a syslog.

Step 4: If powered device initially powers on and then disconnects, problem can be an initial current surge that exceeds a current-limit threshold for the switch port.

Step 5: Verify that the powered device is compatible with the Cisco switch. For example, if both units are standards-compliant, they are interoperable. CDP cannot be used to identify a non-Cisco device, and the switch must rely on accurate detection and classification through layer 1 classification or LLDP when a non-Cisco device is utilized. Ensure LLDP is operational on switch port.

Third-party Powered Device Scenarios

Scenario 1  - Attached PD is a requires more power than its class permits. But it does not support CDP/LLDP extension or it is kept disabled per organizational policy. As a result, the switchport continues to flap.

Recommendation – Configure static power

Use power inline static interface level configuration to give the maximum power to the PD irrespective of its class, PD architecture and the negotiation protocol in use. Use this step when maximum power needed by PD is not known.

C9000(config-if)#power inline static

If maximum power needed by a PD is known, this interface level configuration can be used instead.

C9000(config-if)#power inline static max <required_power>

Scenario 2 - Attached PD is PoE capable on both signal and spare pairs. But it does not support CDP/LLDP extension or it is kept disabled per organizational policy. 

 Recommendation - Configure 4 pair PoE if PD supports it.

Find out if PD supports 4 pair PoE with the command show power inline <interface> detail:

C9000#show power inline Gi1/0/1 detail
 Interface: Gi1/0/1
 Inline Power Mode: auto
 Operational status: on
 Device Detected: yes
 Device Type: Ieee PD
<snip>
Four-Pair PoE Supported: Yes  <++
Spare Pair Power Enabled: No
Four-Pair PD Architecture: Shared  <++

Configure 4 pair PoE:

Cat9K(config-if)#power inline four-pair forced  

Note: By default UPoE switch uses LLDP. Do not configure 4 pair PoE unless powered device is 4 pair capable and LLDP can not be used.

For additional troubleshooting, refer to Common PoE Syslog and Advanced Troubleshooting sections.

Scenario 3 - Class 4 device needs 30W but does not support CDP/LLDP or it is kept disabled per organizational policy. 

Recommendation - Configure 2-event classification or configure static max PoE

When a class 4 device gets detected, Cisco IOS allocates 30W without any CDP or LLDP negotiation. This means that even before the link comes up the class 4 power device gets 30W. Also, on the hardware level the switch does a 2-event classification which allows a class 4 PD to detect switch capability to provide 30W from hardware, register itself and it can move up to PoE+ level without  any CDP/LLDP packet exchange. Once 2-event is enabled on a port, you need to manually shut/no shut the port or connect the PD again to start the IEEE detection again. Power budget allocation for a class-4 device is 30W if 2-event classification is enabled on the port, else it is 15.4W.

Cat9K(config-if)#power inline port 2-event 

Note: A shut/no shut on port is needed for the power inline port 2-event command to be effective. Both switch/line card and the PD must support 2-event classification for this command to work.

Cat9K(config-if)#power inline static max <value> <++ desired amount of power in milliwatts

Common PoE Syslog, Explanation & Actions

1. CONTROLLER PORT ERROR

A port error reported by the Power over Ethernet (PoE) controller is detected by the Cisco switch. Controller error has some common variants.

1.1 Tstart Error

ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi1/0/1: Power Controller reports power Tstart error detected 

Tstart is related to inrush current when a powered device comes up on a switch port. Tstart error means that inrush current’s value measured by switch PoE controller was higher than allowed maximum.

It has been seen that this error in some cases can be related to quick plugging/unplugging of the powered device. This can happen when platform dependent PoE state-machine is in a transition state, and reinsertion of  the PD triggered a new set of state machines steps which conflict with ones in transition.

To rule this out, it is recommended to unplug the powered device connected on the port where TStart error showed up. Wait till “powered down removed” and/or “link down” syslog is seen. Plug the powered device again and see if the syslog does not re-appear.

In some cases, Tstart errors could related to longer or shorter Cat5 or Cat6 cable. Please ensure that the cable length (include cable length between patch panel ends) is within specs. Usage of a cable of different length could potentially fix the problem in some of these cases.

1.2 Power Supply Over Heat

%ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi1/0/1: Power Controller reports power supply over heat

The power inline port 2-event command could help in some cases that experience this scenario.

For this error on a Catalyst 9300L switch, review Cisco bug ID CSCvs52594 and ensure you are on Cisco IOS XE  version 16.12.3 or later.

1.3 Imax Error

%ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Te3/0/1: Power Controller reports power Imax error detected

Imax error occurs when a PoE capable port on the switch draws more power than it negotiated. Additionally, Some non-Cisco devices can have an excessive surge in current when first connected to a PoE port which could trigger Imax error.

Typically this error is seen when the powered device (PD) attached to a given port draws more power than what is negotiated through CDP/LLDP negotiation.

Try a good PD on same port and see if that helps. If the issue meets a specific PD/model, please make sure that attached powered device is IEEE compliant.

For more information see, Troubleshoot PoE Imax Errors on Catalyst 3650/3850 Switches.

1.4 Other Uncommon Controller Port Error logs

1. Power Given, but Power Controller does not Report Power Good.

%ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi1/0/20: Power given, but Power Controller does not report Power Good
%ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/20: PD removed
%ILPOWER-5-DETECT: Interface Gi1/0/20: Power Device detected: IEEE PD
%ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/20: PD removed
%ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi1/0/20: Power given, but Power Controller does not report Power Good
%ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/20: PD removed

As part of PoE detection and classification, there is a negotiation between the PSE and PD which helps the PSE determine what class the PD is. Once PoE detection and classification completes, PoE is allocated. Under ideal scenarios, after PoE allocation, the PD reports  Power Good back to the PSE and then the interface is brought up (layer 1 occurs after PoE).

If the PD fails to send the  Power Good  message or does not send the "power good" message in a timely manner, this error message is printed which results in a full restart of PoE negotiation. This can cause symptoms such as the device never fully links up or is constantly power cycling. 

To further isolate the issue, PoE debugs and traces are required from the problematic state.

2. PWRGOOD SPARE PAIR

%ILPOWER-5-PWRGOOD_SPARE_PAIR: Interface Gi1/0/1: spare pair power good

Spare pair power request made by the powered device was successful and power is available on spare pair. This is not an error message but just an indication that the powered device requested power on spare pair of the Cat5 or Cat6 cable and it was granted. No further action is needed.

3. POWER CDP SHUT

%ILPOWER-5-ILPOWER_POWER_CDP_SHUT: Interface Gi3/0/1: inline power shut
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/1, changed state to down
%LINK-3-UPDOWN: Interface GigabitEthernet3/0/1, changed state to down
%ILPOWER-7-DETECT: Interface Gi3/0/1: Power Device detected: IEEE PD
%ILPOWER-5-POWER_GRANTED: Interface Gi3/0/1: Power granted

This syslog means that inline power is shut because CDP detected that power consumption on this PoE switchport is greater than:

1. Allocation power, or
2. Hardware interface limit, or
3. User configured max power, or
4. Vailable power on this switch.

If this is a transient issue, the issue resolves itself after the switchport bounces like in the example. If there is a prevailing problem, investigate and rule out the four points mentioned earlier.

In some scenarios, this error could be seen when both CDP and LLDP are enabled on the switchport and PoE debugs reveal use of both protocols in power negotiation. You can disable LLDP to alleviate the issue:

no lldp tlv-select power-management
OR
no lldp transmit / no lldp receive

In certain rare conditions, it is observed that this log could be a result of powered device’s misbehavior. For example, PD requests a lower power value in initial negotiation and switch allocates the requested power to PD. Later same PD requests more power than earlier, higher than the previously allocated power. This triggers a CDP shut off and a port flap. Such scenarios can benefit from Perpetual PoE or Fast PoE

4. INVALID IEEE CLASS

%ILPOWER-5-INVALID_IEEE_CLASS: Interface Gi1/0/1: has detected invalid IEEE class: 8 device. Power denied
%ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD

This error is seen when connected powered device has invalid IEEE class. Switch does not power up the device. Refer to PoE Class to understand PoE classes.

If you use a non-Cisco powered device (PD), find out if the PD is the right class.

5. SHUT OVERDRAWN

%ILPOWER-3-SHUT_OVERDRAWN: Interface Gi1/0/1 is shutdown as it is consuming more than the maximum configured power (15400) milliwatts.
%ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
%PM-4-ERR_DISABLE: inline-power error detected on Gi1/0/1, putting Gi1/0/1 in err-disable state

This error means that switch decided to shut the interface down because it found the powered device consumed more than the maximum configured/negotiated power.

Ensure correct power is budgeted for this interface based on the power device electrical specifications or ratings. It is recommended to change the police cutoff power to a higher value to keep the device powered on.

If you use a non-Cisco powered device, find out the expected power needed vs what is drawn.

6. TSTART SPAREPAIR

%ILPOWER-5-TSTART_SPARE_PAIR: Interface Te3/0/1: spare pair power error: TSTART

This error means that the powered device connected to switchport tried to request power on spare Cat5 or Cat6 wire pair & switch detected a higher than expected current inrush (Tstart error) and as a result decided to shut off the power.

This error is often seen in conjunction with Imax error or other errors discussed. Do remedy procedures described for those sections that are dependent on the error seen.

7. SINGLE PAIRSET FAULT

%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
%ILPOWER-5-SINGLE_PAIRSET_FAULT: Interface Gi1/0/1: shutting down Alt-B pairset due to OVERCLS fault
%ILPOWER-5-SINGLE_PAIRSET_FAULT: Interface Gi1/0/1: shutting down Alt-B pairset due to OVERCLS fault

This error means that the dual signature powered device on the switchport has hit a critical fault on one pairset and hence that pairset is shutdown. The earlier example is taken from a UPoE+ capable powered device and switch.

8. PGOOD TIMEOUT SPARE PAIR

%ILPOWER-5-PGOOD_TIMEOUT_SPARE_PAIR: Interface Te1/0/1: spare pair power good timeout error

This error means that the powered device connected to switchport tried to request power on spare Cat5 or Cat6 wire pair but spare pair power good timeout error occurred and power on spare pair is not be supplied.

With a 802.3bt (UPoE+) switch,  remember the Cisco switch that supports IEEE 802.3bt standard for Type 3 powered devices could be in 802.3at mode by default. 802.3bt mode can be enabled through the  this configuration in global configuration mode. Note that this command power cycles the switch after the configuration. This step is not applicable for switch models that are not UPoE+ capable.

C9K(config)# hw-module switch 1 upoe-plus
!!!WARNING!!!This configuration will power cycle the switch to make it effective. Would you like to continue y/n?

Another possible solution could be to try and hardcode needed power on the switchport with the power inline static interface configuration.

In rare conditions, this error could be accompanied when a 802.2bt line card/switch is used.

%ILPOWER-5-SINGLE_PAIRSET_FAULT: Interface Gi1/0/1: shutting down Alt-B pairset due to OVERCLS fault

This would mean that the powered device is incapable to work with 802.3bt PoE system. Use a non-802.3bt PoE switch.

9. ILPOWER POWER DENY

%ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
%ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
%ILPOWER-5-ILPOWER_POWER_DENY: Interface Gi1/0/1: inline power denied. Reason: insufficient power

This error means that there is not enough power that remains in the switch to supply to the Power over Ethernet (PoE) port.

This is likely due to total inline power greater than available power. Verify power budgeting. Install more power supplies if needed. Adjustment of power supply redundancy from redundant to combined can also help. For stacked systems, stack power can be considered to pool total power across stacks.

10. CONTROLLER POST ERR

%ILPOWER-3-CONTROLLER_POST_ERR: Inline Power Feature is disabled on this switch because
     Power On Self Test (POST) failed on this switch.

Switch decided to shut off PoE because Power On Self Test (POST) failed on this switch.

Verify Power over Ethernet (PoE) controller functionality test to health status of the power-sourcing equipment. Refer to the POST section under PoE outputs and data collection for more information.

11. IEEE DISCONNECT

%ILPOWER-7-DETECT: Interface Gi2/0/1: Power Device detected: Cisco PD
%ILPOWER-5-IEEE_DISCONNECT: Interface Gi2/0/1: PD removed

This error means that the powered device is no longer connected to the switch, or the connected powered device is switched to an external AC power source which caused switch to remove PoE on the port.

In some cases this error is accompanied by other errors, like:

%ILPOWER-5-IEEE_DISCONNECT: Interface Tw1/0/1: PD removed
%ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Tw1/0/1: Power is given, but State Machine Power Good wait timer timed out
%ILPOWER-5-IEEE_DISCONNECT: Interface Tw1/0/1: PD removed

In such cases take appropriate action dependent on the other error.

12. LOG OVERDRAWN

%ILPOWER-4-LOG_OVERDRAWN: Interface Gi1/0/1 is overdrawing power. it is consuming 2346 milliwatts where as maximum configured power is (0) milliwatts.
%ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed

Interface X overdrew power. it has consumed Y milliwatts whereas maximum configured power is Z milliwatts. This is just an informational log and switch continues to provide PoE on the port unless switch runs out of power (SHUT_OVERDRAWN) or another error.

Ensure correct power is budgeted for this interface based on the powered device’s electrical specifications and ratings. It is recommended to change the police cutoff power appropriately if needed.

13. CLR OVERDRAWN

%ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi1/0/1: Power given, but State Machine Power Good wait timer timed out
%ILPOWER-4-LOG_OVERDRAWN: Interface Gi1/0/1 is overdrawing power. it is consuming 2346 milliwatts whereas maximum configured power is (0) milliwatts.
%ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
%ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: Cisco PD
%ILPOWER-5-CLR_OVERDRAWN: Interface Gi1/0/1 is NOT overdrawing power. 
                          it is consuming 2346 milliwatts whereas maximum configured value is (15400) milliwatts.

This informational log tells the user that interface X overdrew power earlier but is NOT anymore. it has consumed Y milliwatts whereas maximum configured value is Z milliwatts.

14. DET TIMEOUT SPARE PAIR

%ILPOWER-6-SET_ILPOWER: Set power allocated to POE to 17180 for slot 0
%ILPOWER-7-DETECT: Interface Gi4/0/1: Power Device detected: IEEE PD
%ILPOWER-5-POWER_GRANTED: Interface Gi4/0/1: Power granted
%ILPOWER-5-DET_TIMEOUT_SPARE_PAIR: Interface Gi4/0/1: spare pair detect timeout

This error means that powered device requested power on Cat5 or Cat6 spare wire power and in the process spare pair timeout was detected. As a result, power on spare pair is not supplied.

PoE Outputs and Data Collection

PoE Syslog

Look for any relevant error messages described under Common PoE syslog section in show logging output. For example, PoE controller error, PoE budget error, power supply problem and so on.

POST Status 

POST tests the Power over Ethernet (PoE) controller functionality test to check the chip accessibility, firmware download, and health status of the power-sourcing equipment.

C9K#show post 
Stored system POST messages:
Switch 1
---------
**snip**
POST: Inline Power Controller Tests : Begin  <++ PoE related test
POST: Inline Power Controller Tests : End, Status Passed   <++ Desirable outcome

Inline Power & Budget

Verify PoE budget and Inline Power status of a switch member/line card/interface. Use the show power inline command to review these factors:

• PoE power available per switch.
• PoE power used by all ports in the switch.
• PoE power used by each connected powered device.
• PoE power classification.

C9348U#show platform software ilpower system 1  <++ This value represents switch number for C9300/C9200 and line card number for C9400

ILP System Configuration 
    Slot: 1 
   ILP Supported: Yes 
   Total Power: 857000 
   Used Power: 8896 
   Initialization Done: Yes
   Post Done: Yes 
   Post Result Logged: No 
   Post Result: Success
   Power Summary: 
      Module: 0 
      Power Total: 857000 
      Power Used: 8896 
      Power Threshold: 80 
      Operation Status: On
   Pool: 1
      Pool Valid: Yes
      Total Power: 857000
      Power Usage: 8896

C9348U#show power inline module 1 <++ This value represents switch number for C9300/C9200 and line card number for C9400

Module   Available     Used         Remaining
         (Watts)      (Watts)        (Watts) 
------   ---------   --------       ---------
  1        857.0        8.9           848.1     <++ available PoE budget on switch 1
Interface  Admin    Oper     Power       Device         Class Max
                            (Watts) 
--------- ------ ---------- ------- ------------------- ----- ----
Gi1/0/1    off    off        0.0         n/a              n/a 60.0 
Gi1/0/2   auto    off        0.0         n/a              n/a 60.0 
Gi1/0/3   auto    off        0.0         n/a              n/a 60.0 
Gi1/0/4   auto    on         8.9       IP Phone 8851       4  60.0
**snip**

C9348U#show power inline gigabitEthernet 1/0/4 

Interface  Admin   Oper      Power      Device          Class  Max
                            (Watts)     
--------- ------ ---------- ------- ------------------- -----  ----
Gi1/0/4    auto     on         8.9     IP Phone 8851       4   60.0    <++ Oper status is typically "on". Other states are bad/faulty/off and so on

C9348U#show power inline gigabitEthernet 1/0/4 detail 
 Interface: Gi1/0/4
 Inline Power Mode: auto
 Operational status: on <++ Success
 Device Detected: yes <++ Success
 Device Type: Cisco IP Phone 8851       <++ Success
 IEEE Class: 4 <++ Success
 Discovery mechanism used/configured: Ieee and Cisco
 Police: off

 Power Allocated 
 Admin Value: 60.0
 Power drawn from the source: 8.9 <++ Success
 Power available to the device: 8.9 <++ Success

 Actual consumption
 Measured at the port: 3.4 <++ Success
 Maximum Power drawn by the device since powered on: 3.8

 Absent Counter: 0
 Over Current Counter: 0
 Short Current Counter: 0
 Invalid Signature Counter: 0
 Power Denied Counter: 0

 Power Negotiation Used: CDP
 LLDP Power Negotiation --Sent to PD-- --Rcvd from PD--
       Power Type:         -                 -
       Power Source:       -                 -
       Power Priority:     -                 -
       Requested Power(W): -                 -
       Allocated Power(W): -                 -

Four-Pair PoE Supported: Yes
Spare Pair Power Enabled: No 

C9348U#show power inline police gigabitEthernet 1/0/4

Interface  Admin    Oper      Admin     Oper     Cutoff   Oper 
State     State    Police     Police     Power    Power 
--------- ------ ---------- ---------- ---------- ------ ----- 
Gi1/0/4   auto      on         none      n/a       n/a     3.4    <++ Verify Operating Power

C9348U#show platform software ilpower port gigabitEthernet 1/0/4
ILP Port Configuration for interface Gi1/0/4
  Initialization Done: Yes
   ILP Supported: Yes 
   ILP Enabled: Yes
   POST: Yes
   Detect On: No
   PD Detected Yes
   PD Class Done No
   Cisco PD: No
   Power is On: Yes
   Power Denied: No
   PD Type: IEEE
   PD Class: IEEE4
   Power State: OK
   Current State: NGWC_ILP_LINK_UP_S     <++ Success
   Previous State: NGWC_ILP_LINK_UP_S
   Requested Power: 8896
   Short: 0
   Short Cnt: 0
   Cisco PD Detect Count: 0
   Spare Pair mode: 0
   Spare Pair Arch: 1
   Signal Pair Pwr alloc: 0
   Spare Pair Power On: 0
   PD power state: 0
   Timer:
      Bad Power: Stopped
      Power Good: Stopped
      Power Denied: Stopped
      Cisco PD Detect: Stopped
      IEEE Detect: Stopped
      IEEE Short: Stopped
      Link Down: Stopped
      Vsense: Stopped

PoE Diagnostics 

With online diagnostics, you can test and verify the hardware functionality of a device while the device is connected to a live network. Online diagnostics contain packet-switching tests that check different hardware components and verify the data path and control signals. Online diagnostics detect problems related but not limited to:

• PoE hardware components
• Interfaces
• Solder joints & board integrity

Here are some diagnostic tests that can be used. These can be run ondemand unlike POST that runs only during bootup. Before the test, read the information in the table to understand potential impact.

* Under review by Cisco if this could be made non-disruptive in future.

** Non-Disruptive test, safe to run during production.

 Catalyst 9200

C9200L-24P-4X-A#diagnostic start switch 1 test DiagPoETest    <++ 1 is switch number, use respective switch number in question
Diagnostic[switch 1]: Running test(s) 6 may disrupt normal system operation and requires reload
Do you want to continue? [no]: yes     <++ hit yes, this is non-disruptive. Enhancement is being tracked to remove warning message

*Jun 10 10:22:06.718: %DIAG-6-TEST_RUNNING: switch 1: Running DiagPoETest{ID=6} ...
*Jun 10 10:22:06.719: %DIAG-6-TEST_OK: switch 1: DiagPoETest{ID=6} has completed successfully

C9200L-24P-4X-A#sh diagnostic result switch 1 test DiagPoETest
Current bootup diagnostic level: minimal

Test results: (. = Pass, F = Fail, U = Untested)
6) DiagPoETest ---------------------> .      <++ expected result is pass "."

Catalyst 9300

C9348U-1#diagnostic start switch 1 test DiagPoETest <++ 1 is switch number, use respective switch number in question
Diagnostic[switch 1]: Running test(s) 8 may disrupt normal system operation and requires reload
Do you want to continue? [no]: yes << use with caution, this is disruptive test
C9348U-1#
*Mar 7 06:28:39 CET: %DIAG-6-TEST_RUNNING: switch 1: Running DiagPoETest{ID=8} ...
*Mar 7 06:28:39 CET: %DIAG-6-TEST_OK: switch 1: DiagPoETest{ID=8} has completed successfully
C9348U-1#

C9348U-1#show diagnostic result switch 1 test DiagPoETest
Current bootup diagnostic level: minimal
Test results: (. = Pass, F = Fail, U = Untested)
8) DiagPoETest ---------------------> . <++ expected result is pass "."

Catalyst 9400

C9400#diagnostic start module 3 test TestPoe <++ 3 is line card number, use respective line card number in question
*Jun 10 10:15:23.835: %SYS-5-CONFIG_I: Configured from console by console
test94#
*Jun 10 10:15:26.118: %DIAG-6-TEST_RUNNING: module 3: Running TestPoe{ID=5} ...
*Jun 10 10:15:26.119: %DIAG-6-TEST_OK: module 3: TestPoe{ID=5} has completed successfully


C9400#sh diagnostic result module 3 test TestPoe
Current bootup diagnostic level: minimal

Test results: (. = Pass, F = Fail, U = Untested
5) TestPoe -------------------------> . <++ expected result is pass "."

Advanced Troubleshooting

This section contains PoE debugs and platform specific information that is useful to troubleshoot PoE issues. Some of these outputs do not make sense or would not be available in a human readable format to the end user. These have been found safe to run in production & would be useful if provided to Cisco TAC when a PoE is troubleshot.

InlinePower (ILP) Debugs for PoE

ILpower (ILP) is an internal Cisco IOS XE software component that runs within Cisco IOS Dameon (Cisco IOSd). ilpower implements PoE state machine which governs various steps of PoE functionality. Next, is an ilpower diagram which can be used as a reference in conjunction with Cisco IOSd debugs.

Examine debugs from each state machine step to understand at which step is the functionality broken. Compare these debugs from a PoE port that works and a PoE port that does not work with same/similar PDs can also useful to identify anomalies.

1. Start these debugs:

debug condition interface GigabitEthernet <> <++ Specify interface number for conditional debugging. This helps to limit impact on CPU.
debug ilpower event 
debug ilpower controller
debug ilpower powerman

2. Shut the port in question.

3. Turn off logging console and terminal monitor (no logging console from global configuration mode and term no mon form user Exec mode).

4. Back up the logging output if needed, since the next step resets the logging buffer. Example- show logging | redirect flash:showlogbackup.txt

5. Make sure logging buffer level is set to debugging. Increase the logging buffer size to at least 50K (logging buffer 50000). It is important to remember that this step clears historical logs.

6. Enable the conditional debugging and clear logging (clear logging).

7. Unshut the port in question and wait for 30-40 sec at least for PoE negotiation.

8. Turn off debugging with undebug all and collect the show logging to understand the debugs. 

9. Undo all changes made in steps 2-7. 

This is how a succesful PoE transaction typically looks:

*Mar 6 22:18:33.493: ILP:: ilp enabled in hwidb Gi1/0/4
*Mar 6 22:18:33.493: ILP notify LLDB-TLV: lldp power class tlv:
*Mar 6 22:18:33.493: (curr/prev) pwr value 15400/0
*Mar 6 22:18:33.493: ILP:: ILP CLI 'no shut' handling ( Gi1/0/4 ) Okay
*Mar 6 22:18:33.493: ILP:: Sending poe coredump msg to slot:1
*Mar 6 22:18:33.493: ILP::
Sending E_ILP_GET_DEBUG_CORE_DUMP IPC message from RP to platform
*Mar 6 22:18:33.493: ILP:: ilp hwidb Gi1/0/4 admstate 2
*Mar 6 22:18:33.493: ILP:: ilp hwidb Gi1/0/4 admstate auto, start detect 2
*Mar 6 22:18:33.493: ILP:: ILP CLI 'no shut' handling ( Gi1/0/4 ) Okay
*Mar 6 22:18:33.493: ILP:: ilp enabled in hwidb Gi1/0/4
*Mar 6 22:18:33.494: ILP:: Gi1/0/4: State=NGWC_ILP_SHUT_OFF_S-0, Event=NGWC_ILP_CLI_START_DETECT_EV-17
*Mar 6 22:18:33.494: ILP:: START_DETECT_EV, shutoff_state Gi1/0/4
*Mar 6 22:18:33.494: ILP:: Sending poe detect msg to slot:1 port:4
*Mar 6 22:18:33.494: ILP::
Sending E_ILP_START_IEEE IPC message from RP to platform


*Mar 6 22:18:34.617: ILP:: ILP:get_all_events: num_port: 1, if_id: 4
*Mar 6 22:18:34.617: ILP:: interface in get_all_events: Gi1/0/4, slot 1, port 4
*Mar 6 22:18:34.617: ILP:: ilp event CLASS DONE            <++ Classification done
*Mar 6 22:18:34.617: ILP:: posting ilp slot 1 port 4 event 1 class 4
*Mar 6 22:18:34.617: ILP:: ilp fault 0
*Mar 6 22:18:34.618: ILP:: Gi1/0/4: State=NGWC_ILP_DETECTING_S-2, Event=NGWC_ILP_IEEE_CLASS_DONE_EV-1
*Mar 6 23:18:34 CET: %ILPOWER-7-DETECT: Interface Gi1/0/4: Power Device detected: IEEE PD
*Mar 6 22:18:34.618: (Gi1/0/4) data power pool 1           <++ power is taken from a single pool on the PSE called pool 1
*Mar 6 22:18:34.618: Ilpower PD device 3 class 7 from interface (Gi1/0/4)
*Mar 6 22:18:34.618: (Gi1/0/4) state auto
*Mar 6 22:18:34.618: (Gi1/0/4) data power pool: 1, pool 1
*Mar 6 22:18:34.618: (Gi1/0/4) curr pwr usage 30000
*Mar 6 22:18:34.618: (Gi1/0/4) req pwr 30000               <++ requested power is 30W i.e 30000 mw
*Mar 6 22:18:34.618: (Gi1/0/4) total pwr 857000            <++ total current available PoE on switch 1 is 875000 mw
*Mar 6 22:18:34.618: (Gi1/0/4) power_status OK
*Mar 6 22:18:34.618: ilpower new power from pd discovery Gi1/0/4, power_status ok
*Mar 6 22:18:34.618: Ilpower interface (Gi1/0/4) power status change, allocated power 30000
*Mar 6 22:18:34.618: ILP notify LLDB-TLV: lldp power class tlv:
*Mar 6 22:18:34.618: (curr/prev) pwr value 30000/0         <++ current value 30W and previous value was 0
*Mar 6 22:18:34.618: ILP::
Sending E_ILP_USED_POE IPC message from RP to platform

*Mar 6 22:18:34.618: ILP:: Update used poe power 30000 to platform_mgr for slot 1
*Mar 6 22:18:34.618: ILP:: Sending icutoff current msg to slot:1 port:4
*Mar 6 22:18:34.618: ILP::
Sending E_ILP_SET_ICUTOFF IPC message from RP to platform
*Mar 6 22:18:34.618: ilpower_notify_lldp_power_via_mdi_tlv Gi1/0/4 pwr alloc 30000
*Mar 6 22:18:34.618: Gi1/0/4 AUTO PORT PWR Alloc 255 Request 255
*Mar 6 22:18:34.618: Gi1/0/4: LLDP NOTIFY TLV:             <++ values are pushed down to software in form of TLV (type-length-value)
(curr/prev) PSE Allocation: 25500/0
(curr/prev) PD Request : 25500/0
(curr/prev) PD Class : Class 4/ <++ class 4 device, 30W from PSE
(curr/prev) PD Priority : low/unknown
(curr/prev) Power Type : Type 2 PSE/Type 2 PSE
(curr/prev) mdi_pwr_support: 15/0
(curr/prev Power Pair) : Signal/
(curr/prev) PSE Pwr Source : Primary/Unknown
*Mar 6 22:18:34.619: ILP:: Sending ieee pwr msg to slot:1 port:4
*Mar 6 22:18:34.619: ILP::
Sending E_ILP_APPROVE_PWR,DENY IPC message from RP to platform
*Mar 6 22:18:34.619: ILP:: ILP Power Accounting REQ_PWR ( Gi1/0/4 ) Okay sys_used=30000
*Mar 6 22:18:34.619: ILP::
Sending E_ILP_SET_ICUTOFF IPC message from RP to platform
*Mar 6 22:18:34.619: ILP:: Sending icutoff current msg to slot:1 port:4
*Mar 6 22:18:34.619: ILP::
Sending E_ILP_SET_ICUTOFF IPC message from RP to platform
*Mar 6 22:18:34.619: ILP::
Sending E_ILP_SET_ICUTOFF IPC message from RP to platform
*Mar 6 22:18:34.619: ILP:: Sending icutoff current msg to slot:1 port:4
*Mar 6 22:18:34.619: ILP::
Sending E_ILP_SET_ICUTOFF IPC message from RP to platform
*Mar 6 22:18:34.619: ILP::
Sending E_ILP_SET_ICUTOFF IPC message from RP to platform
*Mar 6 22:18:34.619: ILP:: Sending icutoff current msg to slot:1 port:4
*Mar 6 22:18:34.619: ILP::
Sending E_ILP_SET_ICUTOFF IPC message from RP to platform

*Mar 6 22:18:34.909: ILP:: Rx Response ILP msg: response_code 12, sw_num 1
*Mar 6 22:18:34.909: ILP:: ILP msg: received E_ILP_GET_POWER_SENSE
*Mar 6 22:18:34.909: ILP:: ILP:pwr_sense: num_ports: 48, switch_num: 1
*Mar 6 22:18:34.910: ILP:: ILP:Gi1/0/4:power real 0, min 0, max 0, police 0, overdraw: 0
*Mar 6 23:18:35 CET: %SYS-5-CONFIG_I: Configured from console by console

*Mar 6 22:18:35.205: ILP:: ILP:get_all_events: num_port: 1, if_id: 4
*Mar 6 22:18:35.206: ILP:: interface in get_all_events: Gi1/0/4, slot 1, port 4
*Mar 6 22:18:35.206: ILP:: ilp event PWR GOOD
*Mar 6 22:18:35.206: ILP:: posting ilp slot 1 port 4 event 2 class 0
*Mar 6 22:18:35.206: ILP:: ilp fault 0
*Mar 6 22:18:35.206: ILP:: Gi1/0/4: State=NGWC_ILP_IEEE_PD_DETECTED_S-4, Event=NGWC_ILP_PWR_GOOD_EV-2
*Mar 6 23:18:35 CET: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/4: Power granted
*Mar 6 23:18:35 CET: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/4, changed state to down
*Mar 6 22:18:39.318: ILP:: ilpsm posting link up event Gi1/0/4
*Mar 6 22:18:39.319: ILP:: Gi1/0/4: State=NGWC_ILP_LINK_UP_S-6, Event=NGWC_ILP_PHY_LINK_UP_EV-20
*Mar 6 23:18:41 CET: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/4, changed state to up
*Mar 6 22:18:41.317: ILP:: ilp enabled in hwidb Gi1/0/4
*Mar 6 23:18:42 CET: %SYS-5-LOG_CONFIG_CHANGE: Console logging: level debugging, xml disabled, filtering disabled
*Mar 6 23:18:42 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/4, changed state to up
**snip**

Catalyst 9200 Specific Data Collection

1. Collect show tech-support PoE.

C9200#show tech-support poe | redirect flash:shtechPOE9200.txt

2. Retrieve IFM mapping for respective switch member. Ensure you use the correct switch number on which the PoE problem exists. This is useful for TAC to interpret other collected outputs.

C9200#show platform software fed switch 1 ifm mappings 

Interface            IF_ID Inst Asic Core Port SubPort Mac Cntx LPN GPN Type Active
GigabitEthernet1/0/1  0x7   0    0    0    4     0     12    4   1   1  NIF   Y
GigabitEthernet1/0/2  0x8   0    0    0    5     0      4    5   2   2  NIF   Y
GigabitEthernet1/0/3  0x9   0    0    0    6     0     14    6   3   3  NIF   Y
GigabitEthernet1/0/4  0xa   0    0    0    7     0     13    7   4   4  NIF   Y
**snip**

3. Collect Traces. This CLI creates a binary file in flash. It can be decoded by Cisco TAC to investigate deeper.

C9200#request platform software trace archive

C9200#dir flash: | in tar
48602 -rw- 404145 Jun 9 2020 03:12:36 +00:00 C9200L-48P-4X-1_1_RP_0_trace_archive-20200609-031235.tar.gz <++ upload to TAC case
C9200#

4. Collect further PoE Registers. This CLI creates a file in flash. It can be analyzed by Cisco TAC to investigate deeper.

C9200#show controllers power inline
For logs refer to /flash/poe_controller_logs_*

C9200#dir flash: | in poe
32472   -rw-            33566   Dec 4 2021 09:12:10 +00:00  poe_controller_logs_sw2_Sat-Dec-04-21-09:12:10-UTC

Note: This CLI is officially supported on 17.6.x onwards.

Catalyst 9300 Specific Data Collection

1. Collect show tech-support PoE.

C9300#show tech-support poe | redirect flash:shtechPOE9300.txt

2. Useful show commands (also present in show tech poe) that can be individually collected and examined.

show clock 
show version 
show running-config 
show env all 
show power inline 
show power inline police 
show interface status 
show platform software ilpower details 
show stack-power budgeting 
show stack-power detail 
show controllers ethernet-controller phy detail 
show controllers power inline module 1 
show platform frontend-controller version 0 1 
show platform frontend-controller manager 0 1 
show platform frontend-controller subordinate 0 1 
show platform software ilpower system 1 
show power inline Gi<> detail 

3. Collect frontend-controller version and controller dump.

3.1 Show platform frontend-controller version 0 <switch number>.

C9348U#show platform frontend-controller version 0 1   <++ 1 is switch number here, use your respective switch number in question
Switch 1 MCU:
Software Version  129
System Type        6
Device Id          2
Device Revision    0
Hardware Version   41
Bootloader Version 17

3.2 Show controllers power inline module <switch number>.

show controllers power inline module 1 <++ 1 is switch number, use respective switch no. in question

3.3 Read controller registers.

     test frontend-controller read-poe <MCU no> module <switch member#>

You must use the console access to print this output. Collect this output for all MCUs on the switch in question.

Note: For a UPoE module MCU number is 1-24 and for POE+ module MCU number is 1 -12.

test frontend-controller read-poe 1 module 1 <++ MCU #1 of switch 1,use respective switch number as applicable
test frontend-controller read-poe 2 module 1 <++ MCU #2 of switch 1,use respective switch number as applicable
test frontend-controller read-poe 3 module 1 <++ MCU #3 of switch 1,use respective switch number as applicable
... 
...
test frontend-controller read-poe 12 module 1 <++ MCU #12 of switch 1,use respective switch number as applicable
... 
...     <++ Output for MCU 13-24 is applicable only to UPoE devices
...
test frontend-controller read-poe 24 module 1 

Sample Output-

C9300#test frontend-controller read-poe 24 module 1
Switch 1 Power controller instance 24
Switch number:1

Basic registers:
0x08 0xF6 0x00 0x00 0x01 0x01 0x00 0x00 
0x00 0x00 0x00 0x00 0x06 0x00 0x00 0x00 
0x00 0x2C 0x02 0x0F 0x11 0xF0 0xC0 0x80 
0x00 0x00 0x10 0x1B 0x10 0x01 0x00 0x00 
0x00 0x00 0x10 0x02 0x00 0x00 0x00 0x00 
0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 
0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 
0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00

Extended registers:
0xFF 0xFF 0x00 0x00 0x00 0x00 0x00 0xA8 
0x00 0x69 0x03 0x00 0x00 0x00 0x00 0x00 
0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 
0x00 0x00 0x00 0x00 0x15 0x16 0x60 0xFF 
0x00 0x00 0x00 0x02 0xAA 0x00 0x00 0x00 
0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 
0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 
0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00

4. Retrieve IFM mapping for respective switch member. Ensure to use correct stackwise switch number on which PoE problem exists. This is useful for TAC to interpret other collected outputs.

C9348U#show platform software fed switch 1 ifm mappings 
Interface            IF_ID  Inst  Asic  Core  Port  SubPort  Mac  Cntx  LPN  GPN  Type  Active
GigabitEthernet1/0/1 0x8     1     0     1     0      0      26    6     1    1    NIF   Y
GigabitEthernet1/0/2 0x9     1     0     1     1      0       6    7     2    2    NIF   Y
GigabitEthernet1/0/3 0xa     1     0     1     2      0      28    8     3    3    NIF   Y
GigabitEthernet1/0/4 0xb     1     0     1     3      0      27    9     4    4    NIF   Y
**snip**

5. Collect platform manager traces for TAC

5.1 Set PoE Trace level to verbose. Use respective switch number in question.

Prior to Cisco IOS XE version 16.11.x

set platform software trace platform-mgr switch <switch_num> r0 redearth verbose set platform software trace platform-mgr switch <switch_num> r0 poe verbose
Cisco IOS XE version 16.11.x onwards

set platform software trace chassis-manager switch <switch_num> r0 re_poe verbose set platform software trace chassis-manager switch <switch_num> r0 redearth verbose

set platform software trace chassis-manager switch 1 r0 re_poe verbose 
set platform software trace chassis-manager switch 1 r0 redearth verbose

5.2 Shut/no shut the port in question.

interface gi1/0/4
sh 
no shut <++ wait 2-4 sec before issuing no shut

5.3 Wait for 20-30 seconds.

5.4 Collect Traces.

The command request platform software trace archive creates a binary file in flash of Primary switch, must be decoded by TAC.

C9K#request platform software trace archive 

C9K#dir flash: | in tar 
434284 -rw- 7466248 June 07 2020 13:45:54 +01:00 DUT_1_RP_0_trace_archive-20191125-134539.tar.gz   <++ upload this to TAC case

5.5 Reset the Trace level to Info.

Prior to Cisco IOS XE version 16.11.x

set platform software trace platform-mgr switch <switch_num> r0 redearth info set platform software trace platform-mgr switch <switch_num> r0 poe info
Cisco IOS XE version 16.11.x onwards

set platform software trace chassis-manager switch <switch_num> r0 re_poe info set platform software trace chassis-manager switch <switch_num> r0 redearth info

Catalyst 9400 Specific Data Collection

1. Collect show tech-support PoE.

C9400#show tech-support poe | redirect bootflash:showtechpoe9400.txt

2. Useful show commands (also present in show tech PoE) that can be individually collected and examined.

show clock 
show version 
show running-config 
show env all 
show power inline 
show power inline police 
show interface status 
show platform software ilpower details 
show controllers ethernet-controller phy detail 
show power inline upoe-plus (applicable to modules supporting UPoE+ like C9400-LC-48H)
**snip**

3. Collect platform specific information.

show platform software iomd redundancy show platform show tech-support platform | redirect bootflash:showtechplatform9400.txt

4. Collect Port Register dumps.

test platform hard poe get <line card slot #> global test platform hard poe get <line card#> port <port# in question for PoE>

test platform hard poe get 3 global <++ line card slot number 3, use respective line card number
test platform hard poe get 3 port 1 <++ line card slot number 3, port 1, use respective line card/port number

C9400#test platform hard poe get 2 global
Global Register for slot 2
0x00FFFFFF    0x00FFFFFF    0x80001304    0x000000C1
0x00000000    0x00000700    0x0FFD0FFD    0x00000015
0x0000000E    0x00000000    0x005AD258    0x00003A0A
0x00000700    0x00000000    0x00000000    0x00000000
0x00000000    0x00000000    0x00000000    0x00000000

 POE FW loaded successfully <-- success
 POE health status : GOOD   <-- success
 POE PSE FW ver :19
 POE Abstraction layer FW ver = 14

5. Retrieve IFM mapping for ports. This is useful for TAC to interpret other collected outputs.

show platform software fed active ifm mappings

C9400#show platform software fed active ifm mappings 

Interface            IF_ID Inst Asic Core Port SubPort Mac Cntx LPN GPN Type Active
GigabitEthernet1/0/1  0x8   0    0    0    0    0      4    4    1  101 NIF   Y
GigabitEthernet1/0/2  0x9   0    0    0    1    1      4    4    2  102 NIF   Y
GigabitEthernet1/0/3  0xa   0    0    0    2    2      4    4    3  103 NIF   Y
**snip**

6. Collect IOMD traces.

6.1 Set IOMD Trace level to verbose. Use respective module number in question.

set platform software trace iomd <module_number>/0 poe verbose.

set platform software trace iomd 3/0 poe verbose <++ Here 3 is line card slot#, use respective slot number as applicable

6.2 Shut/no shut the port in question.

conf t
interface gi3/0/1
shut
! wait 2-4 sec before issuing no shut
no shut

6.3 Wait 40-60 sec.

6.4 Collect Traces.

The command request platform software trace archive creates a binary file in flash of Primary switch, must be decoded by TAC.

C9400#dir bootflash: | in tar
194692 -rw- 50261871 Jun 9 2020 02:53:36 +00:00 test94_RP_0_trace_archive-20200609-025326.tar.gz <++ upload this file to TAC case

6.5 Reset the Trace level to Info.

set platform software trace iomd <module number>/0 poe info

set platform software trace iomd 3/0 poe info <++ Here 3 is line card slot#, use respective slot number as applicable

Last Resort / Intrusive Recovery Steps

If PoE does not recover through any of the steps mentioned and appears to be due to soft failure, other steps can be tried to attempt a recovery. Do note that these steps are intrusive and could cause a potential downtime. They can also erase data that is typically needed to root cause the problem. If root cause is important, please reach out to TAC and collect needed information before these steps. 

1. Refer to Recommended Cisco IOS XE releases for Catalyst 9000 switches and upgrade to the recommended release. Recommended releases contain fixes and optimizations that could potentially solve an issue known and resolved in past.

2. If stack-power is in use,  remove the stack-power cables temporarily before any of these steps.

3. Try a reload of the switch member/line card in question

4. In a stackwise system (C9200, C9300), hard power cycle the member/active switch in question.  This step is also needed if you perform an MCU reset.

5. To hard reset, unplug all input power cables to the stack and let it power dowm. Wait 10 seconds and plug the power cables again. For Catalyst 9400, try a hard reseat of the line card. Physically unseat the line card, wait for a few seconds and seat the card back.

6. If it is a high availability (HA) setup and the issue exists in multiple members of a stack or multiple line cards of a C9400 chassis, please try HA failover/SSO (redundancy force-switchover)

7.  If the issue persists and the switch member in question is part of a stack, try these steps:

             a. Take the member switch out of stack and boot it standalone mode. See if that helps to recover PoE on that member switch.
             b. If not, power off the member (standalone/ when out of stack), wait for 3-5 minutes before power is provided again.
    
 8. For C9400, you can move the line card in question to a different slot or chassis, if feasible.

Related Information

• Technical Support & Documentation - Cisco Systems
• Cisco Catalyst 9200 Series Switches Data Sheet
• Cisco Catalyst 9300 Series Switches Data Sheet
• Cisco Catalyst 9400 Series Switch Data Sheet
• Cisco Catalyst 9400 Series Switch Line Cards Data Sheet
• Recommended Cisco IOS XE releases for Catalyst 9000 switches
• End-of-Sale and End-of-Life Announcement for the Cisco IOS XE 16.6.x
• End-of-Sale and End-of-Life Announcement for the Cisco IOS XE 16.9.x