Troubleshooting PowerOn Auto Provisioning

Switch Does Not Come Up in Time for POAP to Complete

If the switch does not come up in a reasonable duration for POAP to complete, connect to the switch through the serial line and check to see if it is stuck at the following prompt: 

Waiting for system online status before starting POAP ...
Waiting for system online status before starting POAP ...
Waiting for system online status before starting POAP ...

System is not fully online. Skip POAP? (yes/no)[n]:

You can continue with POAP by entering no at the prompt. If POAP does not start properly on the second attempt, proceed with the normal setup by entering yes at the prompt when it returns.

POAP Fails

Take these actions if any of the following PowerOn Auto Provisioning (POAP) errors appear:

Problem

Log Example

Solution

POAP does not get aborted or POAP abort is stuck at the “Disabling POAP” log

Abort Auto Provisioning and continue with normal setup ?(yes/no)[n]: yes

  1. Use Ctrl+c or Ctrl+z to abort the POAP process and enter the switch.

  2. If the above solution fails, power cycle the switch.

  3. Abort POAP at an earlier prompt

Note

 

After aborting POAP and performing the necessary configurations or maintenance, you can save the configuration and reboot the switch to ensure it starts normally without entering POAP.

POAP DHCP offer is not accepted

2022 Nov 17 11:55:59 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [FOX2249PGK1-D4:C9:3C:85:7D:BF] - Missing Nexthop information, Option(242)

2022 Nov 17 11:55:59 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [FOX2249PGK1-D4:C9:3C:85:7D:BF] - Missing RT Prefix information, Option(243)

2022 Nov 17 11:55:59 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: [FOX2249PGK1-D4:C9:3C:85:7D:BF] - Missing bootfile url, Option(59)

Add the prompted missing DHCP option as printed on the console to the DHCP server configuration.

POAP script does not get copied

The error message is printed after “Copy Failed”

2022 Mar 10 22:46:52 switch %$ VDC-1 %$ %USER-1-SYSTEM_MSG: S/N[FDO25020XF4]-MAC[A0:3D:6E:EE:D8:40] - Command is : terminal dont-ask ; terminal password <removed> ; copy http://netcode.any.fbinfra.net/dl/image/cisco_nxos_9.3.8/nxos.9.3.8.bin bootflash:/nxos.9.3.8.bin.tmp vrf management - /script.sh

2022 Mar 10 22:47:22 switch %$ VDC-1 %$ %USER-1-SYSTEM_MSG: S/N[FDO25020XF4]-MAC[A0:3D:6E:EE:D8:40] - Copy failed: "\nERROR: ld.so: object '/isan/lib/libutils.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.\nERROR: ld.so: object '/isan/lib/libsandbox.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLA

 Make sure that the file name mentioned in the bootfile URL is correct, and the file is stored in the location as mentioned in the copy command output.

POAP script errors out with no error message

178b171b535356627f7517e7a4c89d25 2022 Jun 9 00:17:55 switch %$ VDC-1 %$ %POAP-2-POAP_SCRIPT_STARTED_MD5_VALIDATED: [FOC232800YF-08:4F:A9:E4:95:37] - POAP script execution started(MD5 validated) 2022 Jun 9 00:17:56 switch %$ VDC-1 %$ %POAP-2-POAP_FAILURE: [FOC232800YF-08:4F:A9:E4:95:37] - POAP Script execution failed

Run the python script on a Linux machine or Cisco Nexus switch using the python3 command to capture the syntax errors.

Once you've found the syntax error, resolve the error with the information provided.

POAP script fails with error

2023 Apr 26 16:59:00 switch %$ VDC-1 %$ %USER-1-SYSTEM_MSG: - Executing configure terminal ; show crypto ca trustpoints - /script.sh^M^M

2023 Apr 26 16:59:01 switch %$ VDC-1 %$ %USER-1-SYSTEM_MSG: - Trustpoint already present. Please check. Exiting USB script. - /script.sh^M^M

2023 Apr 26 16:59:02 switch %$ VDC-1 %$ %POAP-2-POAP_FAILURE: [FDO25110HUV-F8:7A:41:55:30:9F] - POAP Script execution failed^M^M

2023 Apr 26 16:59:07 switch %$ VDC-1 %$ %POAP-2-POAP_FAILURE: [FDO25110HUV-F8:7A:41:55:30:9F] - POAP Script execution failed

Examine the specific error message that precedes this line “Script execution failed”.

The error message will typically provide details about what went wrong and help you identify the part of the script that needs to be addressed.

Resolve the error with the information provided in the logs.

Configuration is missing after POAP replay

root@switch(config)# route-map test % Incomplete command at ‘^’ marker ret=-19

Use the show startup-config log command and check for the missing configuration.

Configure the missing configuration until the issue is resolved.

  • Stop the POAP process to continue with the normal switch bring-up steps. It might take a few minutes for a full stop of POAP, so be patient. 

    2013 Oct 29 22:24:59 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Assigned IP address: 172.23.40.221
2013 Oct 29 22:24:59 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Netmask: 255.255.255.0
2013 Oct 29 22:24:59 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: DNS Server: 172.21.157.5
2013 Oct 29 22:24:59 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Default Gateway: 172.23.40.1
2013 Oct 29 22:24:59 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Script Server: 172.23.40.6
2013 Oct 29 22:24:59 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Script Name: /pxelinux.0
2013 Oct 29 22:25:09 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: The POAP Script download has started
2013 Oct 29 22:25:09 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: The POAP Script is being downloaded from [copy tftp://172.23.40.6//pxelinux.0 bootflash:scripts/script.sh vrf management ]
2013 Oct 29 22:25:10 switch %$ VDC-1 %$ %POAP-2-POAP_FAILURE: POAP boot file download failed.
2013 Oct 29 22:25:10 switch %$ VDC-1 %$ %POAP-2-POAP_FAILURE: POAP DHCP discover phase failed
2013 Oct 29 22:25:12 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Abort Power On Auto Provisioning and continue with normal setup ?(yes/no)[n]:
2013 Oct 29 22:25:46 switch %$ VDC-1 %$ %POAP-2-POAP_DHCP_DISCOVER_START: POAP DHCP Discover phase started
2013 Oct 29 22:25:46 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Abort Power On Auto Provisioning and continue with normal setup ?(yes/no)[n]:

Abort Auto Provisioning and continue with normal setup ?(yes/no)[n]: yes
  • Check the failure reason in the log files. Two POAP log files are saved on the bootflash. Logs from the POAP process are stored in the file that ends with poap_pid _init.log as shown below. The failure reason should appear toward the end of this file. 
    bash-4.2# tail 20131029_222312_poap_5367_init.log -n 3
Tue Oct 29 22:27:41 2013:poap_net_rx_pkt: Droppping the pakcet due to Ethernet hdrparsing error on if_index - 5000000
Tue Oct 29 22:27:41 2013:DEST IP is not Broadcast
Tue Oct 29 22:27:41 2013:poap_net_rx_pkt: Droppping the pakcet due to Ethernet hdrparsing error on if_index - 5000000
  • Check to see if the POAP script file that is downloaded from your DHCP or TFTP server fails in the process of running. Depending on the stage of the failure, the device might proceed with the normal setup or reboot. 
    2013 Oct 29 22:42:34 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Assigned IP address: 172.23.40.181
2013 Oct 29 22:42:34 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Netmask: 255.255.255.0
2013 Oct 29 22:42:34 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: DNS Server: 172.21.157.5
2013 Oct 29 22:42:34 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Default Gateway: 172.23.40.1
2013 Oct 29 22:42:34 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Script Server: 172.23.40.6
2013 Oct 29 22:42:34 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Script Name: poap.py
2013 Oct 29 22:42:45 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: The POAP Script download has started
2013 Oct 29 22:42:45 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: The POAP Script is being downloaded from [copy tftp://172.23.40.6/poap.py bootflash:scripts/script.sh vrf management ]
2013 Oct 29 22:42:46 switch %$ VDC-1 %$ %POAP-2-POAP_SCRIPT_DOWNLOADED: Successfully downloaded POAP script file
2013 Oct 29 22:42:46 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Script file size 21965, MD5 checksum 1bd4b86892439c5785a20a3e3ac2b0de
2013 Oct 29 22:42:46 switch %$ VDC-1 %$ %POAP-2-POAP_SCRIPT_STARTED_MD5_NOT_VALIDATED: POAP script execution started(MD5 not validated)
2013 Oct 29 22:47:57 switch %$ VDC-1 %$ %POAP-2-POAP_FAILURE: POAP script execution aborted
  • The POAP script file logs are written to a file under the bootflash scheme. The filename starts with poap.log. If there are multiple file logs, look at the one with the most recent time stamp for any errors. 
    bash-4.2# tail poap.log.22_42_46
CLI : show file volatile:poap.cfg.md5.poap_md5 | grep -v '^#' | head lines 1 | sed 's/ .*$//'
INFO: md5sum 46684d8f8b7c5ffac3b37ac8560928e5 (.md5 file)
CLI : show file volatile:poap.cfg md5sum
INFO: md5sum 46684d8f8b7c5ffac3b37ac8560928e5 (recalculated)
CLI : show system internal platform internal info | grep box_online | sed 's/[^0-9]*//g'
INFO: Setting the boot variables
CLI : config terminal ; boot nxos bootflash:poap/system.img
CLI : copy running-config startup-config
CLI : copy volatile:poap.cfg scheduled-config
INFO: Configuration successful