Troubleshooting Initial Startup Problems

Your Cisco ASR 1000 Series Router went through extensive testing before leaving the factory. However, if you encounter problems starting the router, use the information in this chapter to help isolate the cause of the problems. This chapter contains the following sections:

The procedures in this chapter assume that you are troubleshooting the initial system startup and that your router is in the original factory configuration.

If you have removed or replaced components or changed any default settings, the recommendations in this chapter might not apply. Make sure to review the safety warnings listed in the Regulatory Compliance and Safety Information for the Cisco ASR 1000 Series Aggregation Services Routers publication that accompanied your Cisco ASR 1000 Series Router before using the troubleshooting procedures in this chapter.


Note
For operations and maintenance information, such as checking the LEDs, checking status using show commands, what to do when installation is not successful, that is specific to the Cisco ASR 1000 Series Aggregation Services Routers, see Cisco ASR 1000 Series Aggregation Services Routers Operations and Maintenance Guide .

Troubleshooting Overview

This section describes the methods used in troubleshooting the router. The troubleshooting methods are organized according to the major subsystems in the router.

If you are unable to solve a problem on your own, you can contact a Cisco customer service representative for assistance. When you call, have the following information ready:

  • Date you received the router and the chassis serial number (label located on the chassis, see xref “Serial Number Label Location” section.
  • Installed SPAs.
    • Use the show platform command to determine which SPAs are installed if possible.
  • Cisco software release number.
    • Use the show version command to determine this information if possible.
  • Brief description of the symptoms and steps you have taken to isolate and solve the issue.
  • Maintenance agreement or warranty information.

Online Troubleshooting Resources

In addition to following the subsystems approach to troubleshooting, a variety of online troubleshooting resources are available.

  • Cisco ASR 1000 Series Aggregation Services Routers Troubleshooting Guide contains information to help you troubleshoot problems with the Cisco ASR 1000 Series Router.
  • Cisco.com registered users can access various troubleshooting tools such as Software Advisor, Cisco IOS Error Message Decoder Tool, and Output Interpreter Tool after logging in at http://www.cisco.com/en/US/support/tsd_most_requested_tools.html.

General Troubleshooting Tips

The following table lists general troubleshooting tips.


Note
You must always have two power supplies installed on the Cisco ASR 1000 Series Routers to ensure sufficient cooling for the box. The system fans are inside the power supply and must spin for cooling. Since all the system fans can be powered by one power supply, it is not required for the second power supply to be powered on, but it must be installed.
Table 1. General Startup Troubleshooting Tips

Symptom

Corrective Action

System fails to power on

Check that:

  • All power cords are properly connected to the Cisco ASR 1000 Series Router and at the power connection end.
  • The power switch is in the | ON position.

System fails to boot up properly

If your system has power, check the STATUS LED on the ASR 1000 Series route processor and make certain all connections are secure. See Table 2-8 for more information about the LEDs.

Power problems

If the two power supplies are both showing GREEN on their Output LEDs, then a failure of one power supply will not result in a system failure.

Only one operating power supply is required for the Cisco ASR 1000 Series Router system to operate. Having only one supply powered is a supported configuration. If both supplies have an OUTPUT FAIL LED red, then the system will fail.

Troubleshooting Using a Subsystem Approach

To solve a system problem, try to isolate the problem to a specific subsystem. Compare current router behavior with expected router behavior. Because a startup issue is usually attributable to one component, it is most efficient to examine each subsystem, rather than trying to troubleshoot each router component.

For troubleshooting purposes in this chapter, the router consists of the following subsystems:

  • Power subsystem—Includes the following components:

AC input or DC input power supplies, also called power entry modules (PEMs).

  • Processor subsystem—The Cisco ASR 1000 series RP, ESPs, and SIPs have onboard processors. The RP downloads software to each board in the system over the Ethernet Out of Band Channel (EOBC). There is a status LED on each board (RP, ESP, SIP) that indicates the progress of loading software. Note that on routers such as the Cisco ASR 1001 Router and the Cisco ASR 1002-X Router that have an integrated route processor, embedded services processor, and SIP, there is a single status LED to show the loading status of the system. The LED is red if ROMMON does not boot. If the board has booted ROMMON successfully, the LED is yellow. If operation software (IOS) has downloaded successfully, the LED is green.
  • Cooling subsystem—Consists of three fans in each of the Cisco ASR 1006 Router and Cisco ASR 1004 Router power supplies and two fans in each of the Cisco ASR 1002 Router, Cisco ASR 1002-F Router, and Cisco ASR 1002-X Router power supplies. On the Cisco ASR 1001 Router, each PEM has its own fan and the system itself has a separate fan tray. The fans draw in air from each of the chassis and PEMs in a front to back direction.

Normal Router Startup Sequence

You can generally determine when and where the power supply failed during the startup sequence by checking the status LEDs on the power supply modules.

In a normal router startup sequence, the following sequence of events and conditions occur:

  1. The fan in each PEM receives power and begins drawing air through the power supply. The power supply PWR OK indicator is on and reflects power supply status.
  2. As the power on and boot process progresses for the Cisco ASR 1000 Series RPs, ASR 1000 ESPs, and each installed SIP, the status of each card is indicated by LEDs. On routers that have an integrated route processor, embedded services processor, and SIP, the status LED shows the status of the process.

Troubleshooting the Power Subsystem

Use the information in the following table to isolate problems with the power system.

Table 2. Troubleshooting the Power System

Symptom

Possible Cause

Possible Solution

System begins power on

System has failed to boot.

There is a power fault in the chassis.

The status LED stays amber during the entire boot process and then turns green when the software has completely booted.

System does not power on.

AC power cable not fully seated at system or at the wall outlet (power source).

Turn the router power switch to the standby position (|) and reseat the AC power cable at the system or at the wall outlet (power source).

DC power cable not turned on at the panel board of the circuit breaker.

Turn the router power switch to the standby position (|) and locate the circuit breaker on the panel board that services the DC circuit; switch the circuit breaker to the on position.

AC internal power cables are not fully seated from the power supply

Turn the router power switch to the standby position (|) and remove and reinsert the power supply cable.

System does not power on.

DC internal power cables are not fully seated on the terminal block.

Check that all ground cables are properly seated in the terminal block on the DC power supply.

Power source is faulty.

Turn the power source switch off, connect the power cable to another power source, if available, and turn the router power switch back on.

Faulty power cable.

Turn the router power switch to the standby position (|), (for DC power, also turn the circuit breaker to the Off position and tape it to that position), remove the cable and replace it.

Faulty power supply.

If the system still fails to come up when the power supply is connected to a different power source with a new power cable, the power supply is probably faulty. Contact a service representative.

System powers off, no STATUS LED, and no operating fans.

Power supply failure.

**If the system is operating with only one power supply, it powers off after five minutes.

**Except for the Cisco ASR 1001 Router, four power supplies must be installed in the chassis at all times, with a minimum of two power supplies (one per zone) connected to the mains in order to power on the system and ensure sufficient cooling. The system fans are inside the power supply units and must spin for cooling. Because all the system fans can be powered by one power supply, the second power supply unit does not have to be powered on, but must be connected. If you remove a power supply from a system that has four power supplies that are connected and powered on, the system can run only for a maximum of five minutes before shutting down. However, because the fans and power elements are independent within the power supply, the replacement power supply does not have to be energized within five minutes. The only requirement is that the power supply be installed in the chassis in order to energize the fans and maintain proper system cooling.

Troubleshooting the Cooling Subsystem

Use the information in the following table to isolate problems with the cooling subsystem.

Table 3. Troubleshooting the Cooling Subsystem

Symptom

Possible Cause

Possible Solution

System shuts down, some fans may or may not continue to rotate, and this error message is displayed: 


Queued messages:
%ENVM-1-SHUTDOWN: Environmental Monitor initiated shutdown

This error message indicates that the system has detected an overtemperature condition or out-of-tolerance power condition inside the chassis.

  • One or more fans are not operating.
  • The fans are operating too slowly.
  • The power supply is not operating.

To determine if the fans are operating, listen for them. In noisy environments, place your hand on the rear of the chassis to feel if air is being forced out the vents.

Contact a service representative.

Multiple fan failure

Fan speed dropped below the fan fail speed.

  • Identify and log the failed fan PEM location.
  • Replace the PEM.

Heated exhaust air from other equipment is entering the routers inlet vents.

Move other equipment or the router to ensure proper airflow.

System shuts down, and the following error message is displayed: 


Queued messages:
%ENVM-1-SHUTDOWN: Environmental Monitor initiated shutdown

This error message indicates that the system has detected an over-temperature condition or out-of-tolerance power condition inside the chassis.

Note 
The system fans may continue to operate although the system shuts down.

The error message could indicate a faulty component or temperature sensor. Before the system shuts down, use the show env all command to display the internal chassis environment.

Contact a service representative.

If an environmental shutdown results from an out-of-tolerance power condition, the system shuts down.

If the system still fails to come up when the power supply is connected to a different power source with a new power cable, the power supply is probably faulty. Contact a service representative.

Troubleshooting the Shared Port Adapter

Use the information in the following table to isolate problems with the shared port adapter.

Table 4. Troubleshooting the Shared Port Adapter

Symptom

Possible Cause

Possible Solution

Shared Port adapter ENABLED LED is off.

The shared port adapter might have pulled away from the system board.

Reseat the adapter in its slot (you do not have to turn off the system power when removing or replacing port and service adapters). After the system reinitializes the interfaces, the ENABLED LED on the shared port adapter should go on. Ensure that the adapter is correctly replaced, with the screws fully tightened.

If the ENABLED LED remains off, the system detected a processor hardware failure. (This LED should be on in normal operation.) Contact a service representative for instructions.

Troubleshooting the Upgrade

The following table lists tips to resolve troubleshooting errors that you may encounter during an upgrade.

Table 5. Upgrade-Related Troubleshooting Tips

Symptom

Cause

Corrective Action

An error message similar to the following example is displayed after you start the upgrade: 


validate_package: SHA-1 hash:
        calculated 9526c1bf:10341089:84ecbb0d:cb12a344:b696af14
        expected   93315a74:57061354:d514ff0c:8b25f8f8:842afb4b
SHA-1 hash doesn't match                                       
application image failed to run

There is a mismatch in the size of the image file that you are using and the expected size of the image file. Note that this error may occur, regardless of the file system to which you copy the image file.

Ensure that the size of the image file that you are using is the same as the expected file size. If required, download the image file again and retry the upgrade.

An error message similar to the following example is displayed when you try to boot the system, and then the router switches to the rommon prompt: 


Directory an_image.bin not found
Unable to locate an_image.bin directory
Unable to load an_image.bin
boot: error executing "boot harddisk:an_image.bin"
autoboot: boot failed, restarting

The image file name specified in the boot command is invalid.

Perform the following steps:

  1. Verify that the image file is present in the harddisk:, bootflash:, or external USB device (usb0: or usb1:) by using the dir file-system command at the rommon prompt.
  2. When you run the dir file-system command, if a message similar to "Please reset before continuing" is displayed, reconfigure the configuration-register to 0x0 and then run the reset command to allow the router to enter the rommon prompt without reattempting to boot using an invalid image. 

rommon> dir harddisk:
Please reset before continuing <<<<<
rommon> confreg 0x0
rommon> reset
  1. Boot the router from the rommon prompt by using a command similar to the following example: 

rommon> BOOT=harddisk:asr1000rp1-ipbasek9.03.05.01.S.152-1.S1.bin
  1. To save the current environment variable setting, run the sync command as follows: 

rommon> sync
  1. Ensure that autoboot has been enabled by using the confreg 0x2102 command at the rommon prompt, and then run the reset command at the same prompt.

If there is no valid image in either the local file system or the external file system, you can run the boot tftp: command to install the image that is on the TFTP server. To do this, ensure that the management Ethernet interface is physically connected to the default gateway, configure the ROMmon variables with the appropriate values, and then run the boot tftp: command as shown in the following example: 


rommon >
IP_SUBNET_MASK=255.255.255.0
TFTP_SERVER=192.0.2.2
TFTP_FILE=asr1000rp1-ipbasek9.03.05.01.S.152-1.S1.bin
DEFAULT_GATEWAY=192.0.2.1
IP_ADDRESS=192.0.2.26
rommon > boot tftp:

Autoboot has been enabled by using the config-register 0x2102 command. The following error message is displayed when the router restarts automatically: 


no valid BOOT image found
Final autoboot attempt from default boot device...
Located l2tp_rmcd_alg                             
Image size 10271 inode num 12, bks cnt 3 blk size 8*512
#                                                      
Boot image size = 10271 (0x281f) bytes
                                      
.
.
.
Boot image size = 11262 (0x2bfe) bytes
Unknown image structure
Located test
Image size 11506 inode num 63, bks cnt 3 blk size 8*512

The boot system command has not been configured by running a command similar to the following example: 


boot system bootflash:asr1000rp1-ipbasek9.03.05.01.S.152-1.S1.bin

Perform the following steps:

  1. Press the Break key, or run the break command from the console terminal. If Break is enabled, the router enters the rommon mode. Proceed to the last step. If Break is disabled, power cycle the router (turn the router off or unplug the power cord, and then restore power after waiting for 30 seconds).
  2. Within 30 seconds of restoring power to the router, press the Break key or run the break command. This action causes the router to enter the rommon mode and display the rommon prompt.
  3. Install a previously working version of the image from bootflash: or harddisk:. Alternatively, copy onto a USB stick the image to which you want to upgrade the router and then install that image on the router.

Replacing or Recovering a Lost Password

This section describes how to recover a lost enable or console login password, and how to replace a lost enable secret password on your Cisco ASR 1000 Series Router.


Note
It is possible to recover the enable or console login password. The enable secret password is encrypted, however, and must be replaced with a new enable secret password.

Overview of the Password Recovery Procedure

Following is an overview of the steps in the password recovery procedure:

SUMMARY STEPS

  1. If you can log in to the router, enter the show version command to determine the existing configuration register value.
  2. Press the Break key to get to the bootstrap program prompt (ROM monitor). You might need to reload the system image by power cycling the router.
  3. Change the configuration register so the following functions are enabled:
  4. Power cycle the router by turning power off and then back on.
  5. Log in to the router and enter the privileged EXEC mode.
  6. Enter the show startup-config command to display the passwords.
  7. Recover or replace the displayed passwords.
  8. Change the configuration register back to its original setting.

DETAILED STEPS

Step 1

If you can log in to the router, enter the show version command to determine the existing configuration register value.

Step 2

Press the Break key to get to the bootstrap program prompt (ROM monitor). You might need to reload the system image by power cycling the router.

Note 
To recover a lost password if the Break function is disabled on the router, you must have physical access to the router.
Caution 

If you are going to power cycle a Cisco ASR 1013 Router, we recommend that you first perform a graceful reload on the router. Power cycling the router without first performing a graceful reload might cause a loss of data stored in the NVRAM. In other words, the configuration file might be lost. Note that this is not observed when a power failure occurs because each active power supply in the two power supply zones is power cycled at the same time during a power failure. If there is a chance that the router might be power cycled without a graceful reload, we recommend that you use the boot config file-system:configuration-file nvbypass command to specify a file system other than the NVRAM for storing the configuration file. The following are examples:Router(config)# boot config harddisk:config_file.cfg nvbypass Router(config)# boot config bootflash:configuration_data.cfg nvbypass

Step 3

Change the configuration register so the following functions are enabled:

  1. Break

  2. Ignore startup configuration

  3. Boot from flash memory

    Note 
    The key to recovering a lost password is to set the configuration register bit 6 (0x0040) so that the startup configuration (usually in NVRAM) is ignored. This allows you to log in without using a password and to display the startup configuration passwords.
Step 4

Power cycle the router by turning power off and then back on.

Note 
When powering off the router, wait for 30 seconds before powering it on again.
Step 5

Log in to the router and enter the privileged EXEC mode.

Step 6

Enter the show startup-config command to display the passwords.

Step 7

Recover or replace the displayed passwords.

Step 8

Change the configuration register back to its original setting.

Details of the Password Recovery Procedure

Perform the following steps to recover or replace a lost enable, enable secret, or console login password:

SUMMARY STEPS

  1. Attach an ASCII terminal to the console port on your router.
  2. Configure the terminal to operate at 9600 baud, 8 data bits, no parity, and 1 stop bit (9600 8N1).
  3. If you can log in to the router as a nonprivileged user, enter the show version command to display the existing configuration register value. Note the value for use later and proceed to Step 6. If you cannot log in to the router at all, go to the next step.
  4. Press the Break key or send a Break from the console terminal. If Break is enabled, the router enters the ROM monitor, indicated by the ROM monitor prompt (rommon1>). Proceed to Step 6. If Break is disabled, power cycle the router (turn the router off or unplug the power cord, and then restore power after waiting 30 seconds). Then proceed to Step 5.
  5. Within 60 seconds of restoring the power to the router, press the Break key or send a Break. This action causes the router to enter the ROM monitor and display the ROM monitor prompt (rommon1>).
  6. Set the configuration register using the configuration register utility; enter the confreg command at the ROM monitor prompt as follows:
  7. Answer yes to the enable ignore system config info? question, and note the current configuration register settings.
  8. Initialize the router by entering the reset command as follows:
  9. Enter no in response to the System Configuration Dialog prompts until the following message is displayed:
  10. Press Return. The user EXEC prompt is displayed as follows:
  11. Enter the enable command to enter privileged EXEC mode. Then enter the show startup-config command to display the passwords in the configuration file as follows:
  12. Scan the configuration file display looking for the passwords (the enable passwords are usually near the beginning of the file, and the console login or user EXEC password is near the end). The passwords displayed look something like this:
  13. Enter the configure memory command to load the startup configuration file into running memory. This action allows you to modify or replace passwords in the configuration.
  14. Enter the privileged EXEC configure terminal command to enter configuration mode:
  15. Change all three passwords using the following commands:
  16. You must configure all interfaces to be not administratively shut down as follows:
  17. Use the config-register command to set the configuration register to the original value noted in Step 3 or Step 8, or to the factory default value 0x2102 as follows:
  18. Press Ctrl-Z (hold down the Control key while you press Z) or enter end to exit configuration mode and return to the EXEC command interpreter.
  19. Enter the copy running-config startup-config command to save the new configuration to NVRAM.
  20. Enter the reload command to reboot the router.
  21. Log in to the router with the new or recovered passwords.

DETAILED STEPS

Step 1

Attach an ASCII terminal to the console port on your router.

Step 2

Configure the terminal to operate at 9600 baud, 8 data bits, no parity, and 1 stop bit (9600 8N1).

Step 3

If you can log in to the router as a nonprivileged user, enter the show version command to display the existing configuration register value. Note the value for use later and proceed to Step 6. If you cannot log in to the router at all, go to the next step.

Step 4

Press the Break key or send a Break from the console terminal. If Break is enabled, the router enters the ROM monitor, indicated by the ROM monitor prompt (rommon1>). Proceed to Step 6. If Break is disabled, power cycle the router (turn the router off or unplug the power cord, and then restore power after waiting 30 seconds). Then proceed to Step 5.

Step 5

Within 60 seconds of restoring the power to the router, press the Break key or send a Break. This action causes the router to enter the ROM monitor and display the ROM monitor prompt (rommon1>).

Step 6

Set the configuration register using the configuration register utility; enter the confreg command at the ROM monitor prompt as follows:

Example:


rommon1> confreg 0x2142
Step 7

Answer yes to the enable ignore system config info? question, and note the current configuration register settings.

Step 8

Initialize the router by entering the reset command as follows:

Example:


rommon2> reset

The router initializes, the configuration register is set to 0x142, and the router boots the system image from flash memory and enters the System Configuration Dialog prompt as follows:

Example:


--- System Configuration Dialog --
Step 9

Enter no in response to the System Configuration Dialog prompts until the following message is displayed:

Press RETURN to get started!

Step 10

Press Return. The user EXEC prompt is displayed as follows:

Router>

Step 11

Enter the enable command to enter privileged EXEC mode. Then enter the show startup-config command to display the passwords in the configuration file as follows:

Router# show startup-config

Step 12

Scan the configuration file display looking for the passwords (the enable passwords are usually near the beginning of the file, and the console login or user EXEC password is near the end). The passwords displayed look something like this:

enable secret 5 $1$ORPP$s9syZt4uKn3SnpuLDrhuei

enable password 23skiddoo

.

.

line con 0

password onramp

The enable secret password is encrypted and cannot be recovered; it must be replaced. The enable and console login passwords may be encrypted or clear text. Proceed to the next step to replace an enable secret, console login, or enable password. If there is no enable secret password, note the enable and console login passwords, if they are not encrypted, and proceed to Step 17.

Caution 

Do not execute the next step unless you have determined you must change or replace the enable, enable secret, or console login passwords. Failure to follow the steps as shown might cause you to erase your router configuration.

Step 13

Enter the configure memory command to load the startup configuration file into running memory. This action allows you to modify or replace passwords in the configuration.

Router# configure memory

Step 14

Enter the privileged EXEC configure terminal command to enter configuration mode:

Hostname# configure terminal

Step 15

Change all three passwords using the following commands:

Hostname(config)# enable secret newpassword1

Hostname(config)# enable password newpassword2

Hostname(config)# line con 0

Hostname(config-line)# password newpassword3

Change only the passwords necessary for your configuration. You can remove individual passwords by using the no form of the above commands. For example, entering the no enable secret command removes the enable secret password.

Step 16

You must configure all interfaces to be not administratively shut down as follows:

Hostname(config)# interface gigabitethernet 0/0

Hostname(config-int)# no shutdown

Enter the equivalent commands for all interfaces that were originally configured. If you omit this step, all interfaces are administratively shut down and unavailable when the router is restarted.

Step 17

Use the config-register command to set the configuration register to the original value noted in Step 3 or Step 8, or to the factory default value 0x2102 as follows:

Hostname(config)# config-register 0x2102

Step 18

Press Ctrl-Z (hold down the Control key while you press Z) or enter end to exit configuration mode and return to the EXEC command interpreter.

Caution 

Do not execute the next step unless you have changed or replaced a password. If you skipped Step 13 through Step 16, skip to Step 20. Failure to observe this caution causes you to erase your router configuration file.

Step 19

Enter the copy running-config startup-config command to save the new configuration to NVRAM.

Step 20

Enter the reload command to reboot the router.

Step 21

Log in to the router with the new or recovered passwords.

What to do next

This completes the steps for recovering or replacing a lost enable, enable secret, or console login password.

Recovering the Password When a Standby RP is Included in the System

If a standby RP is included in the system, perform the following procedure to recover the password:


Note
You need not perform the password recovery procedure on the standby RP. After you bypass the NVRAM (startup-config) on the active RP, subsequent reloads of the standby RP synchronize the running-config of the standby RP with the running-config of the active RP.
  1. Connect to the console port of the standby RP. It is recommended that you create a console port connection to the standby RP and not use the console port of the active RP.
  2. Reset both RPs to the ROMmon prompt.

Do not reboot the standby RP at this point. Leave it at the ROMmon prompt.

  1. Perform the password recovery procedure on the active RP.
  2. After the running-config of the active RP has been modified with the password removed from the running-config, save the new running-config to NVRAM (that is, copy the contents of the running-config to the startup-config).
  3. Start the standby RP.

The standby RP synchronizes its configuration with that of the active RP. The new configuration does not include a password.