Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Deploy

Configure

Download

Install

Verify

Trigger Event

Validate Action

Introduction

This document describes how to deploy a Diagnostic Signature (DS) in order to automatically collect the diagnostics data that is required to troubleshoot issues with the Cisco Integrated Services Routers (ISRs), Cisco Aggregation Services Routers (ASRs), and Cisco Catalyst 6500 Series switches and 7600 Series routers.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The show commands that are used throughout this document were captured from a Cisco ISR 3945 Series router that runs Cisco IOS® Version 15.4(2)T3.

The DS support is available in the Cisco IOS versions that are listed in this table:

Platform

Software Version

ISR 1900, 2900, 3900 Series routers

Cisco IOS 15.4(2)T and later

ISR 4300, 4400 Series routers

Cisco IOS 15.5(2)S, IOS XE 3.15 and later

ASR 1000 Series routers

Cisco IOS 15.5(2)S, IOS XE 3.15 and later

Cisco Cloud Services Router 1000V Series

Cisco IOS 15.5(2)S, IOS XE 3.15 and later

7600 Series routers

Cisco IOS 15.3(3)S and later

Catalyst 6500 Series switches

Cisco IOS 15.1(2)SY3 and later

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Background Information

The DSs are XML files that contain information about problem trigger events and actions to be taken in order to remediate or troubleshoot. They are created by the Cisco Technical Assistance Center (TAC) and can be deployed via Smart Call Home, or you can manually load the file onto the device. You can use the Diagnostic Signature Lookup Tool (DSLT) in order to find the right DS for the given type of issue.

In this document, you will learn how to find and deploy a DS in order to troubleshoot PVDM-3 DSP crashes. This DS performs these actions:

1. Configures the router in order to generate Digital Signal Processor (DSP) crashdumps.

2. Detects DSP crashes via the writing out DSP dump to file <file name> syslog message.

3. Collects the relevant show commands.

4. Sends the collected data via email to attach@cisco.com.

At the time of installation, you are prompted to enter the maximum number of DSP crashdump files to be created, the case number to be used in the email subject, and the email address to be notified of the DSP crash occurrence.

Deploy

This section describes how to deploy the DS.

Configure

Note: The DSs are a part of the call-home feature in the Cisco IOS.

The first step that you must complete in order to deploy the DS is to configure the call-home feature. Once the call-home feature is enabled, the DS support is automatically enabled and configured for the use of the CiscoTAC-1 profile by default.

Here is an example configuration for Smart Call Home in the Cisco IOS:

service call-home

call-home

contact-email-addr router@cisco.com

mail-server 192.0.2.33 priority 1

http-proxy "192.0.2.60" port 3128

no syslog-throttling

profile "CiscoTAC-1"

active

destination transport-method http

no destination transport-method email

Router#show call-home diagnostic-signature

Current diagnostic-signature settings:

Diagnostic-signature: enabled

Profile: CiscoTAC-1 (status: ACTIVE)

Environment variable:

Not yet set up

Downloaded DSes:

Last Update

DS ID    DS Name   Revision Status     (GMT+00:00)

-------- --------- -------- ---------- ------------

Router#

Download

The second step is to find the signature ID or download the DS XML file. In order to do this, provide the platform, product, problem scope, problem type, and software version in Diagnostic Signature Lookup Tool as shown here:

Use this information in order to download the DS via Smart Call Home with the DS ID:

call-home diagnostic-signature download 10492

Use this information in order to manually load the DS file onto the device:

Router#copy ftp://192.0.2.10/DS_10492.xml flash:

Destination filename [DS_10492.xml]?

Accessing ftp://192.0.2.10/DS_10492.xml...!

[OK - 3804/4096 bytes]

3804 bytes copied in 0.476 secs (7992 bytes/sec)

Router#

Router#call-home diagnostic-signature load flash:DS_10492.xml

Load file flash:DS_10492.xml success

Router#

Here is a high-level view of individual blocks within the DS file:

Router#show call-home diagnostic-signature 10492

ID              : 10492

Name            : DS_PVDM3_DSP_Crash_Event_1

Functionality:

This DS configures DSP crash dump generation, identifies crash event and

collects show commands required to troubleshoot and identify root cause.

This DS will have no impact on the performance of the router.

Prompts:

Variable: ds_number_of_files               Prompt: Number of crashdump files

to be stored in the flash    (1-5)

Type: integer    Range: 1..5

Variable: ds_case_number                   Prompt: Enter TAC Case Number

(Case number to which diagnostics data need to be uploaded)

Type: regexp    Pattern: 6[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]

Variable: ds_user_email                    Prompt: Enter Notification Email-Address

(Email address to which problem occurrence needs to be notified)

Type: regexp    Pattern: [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]+

Prerequisite:

Type            : CMD

Element List    :

CMD : config t

CMD : voice dsp crash-dump destination flash:dsp_crashdump

CMD : voice dsp crash-dump file-limit $ds_number_of_files

Event:

Action Tag      : a1

Event Tag       : e1

Type            : syslog

Syslog Pattern  : .*writing out DSP dump to file ([^[:space:]]+).*

Includes action steps that may impact device state: No

Action:

Action Tag      : a1

Type            : EMAILTO

Email To        : attach@cisco.com,$ds_user_email

Subject         : DSP Crashdump

Attach SR       : $ds_case_number

Element List    :

DATA&colon; show version

DATA&colon; show voice dsp group all

DATA&colon; show call active voice brief

DATA&colon; show call active video brief

DATA&colon; show call active fax brief

DATA&colon; show sccp connection

DATA&colon; show dspfarm all

DATA&colon; dir $ds_dsp_crashdump_file

DATA&colon; show run

Postrequisite:

Type            : CMD

Element List    :

CMD : config t

CMD : no voice dsp crash-dump destination flash:dsp_crashdump

CMD : no voice dsp crash-dump file-limit $ds_number_of_files

CMD : end

Router#

Install

Note: This step is required only for signatures that are in the pending status after download.

After you have configured and downloaded the DS, you must then install it:

Router#show call-home diagnostic-signature

Current diagnostic-signature settings:

Diagnostic-signature: enabled

Profile: CiscoTAC-1 (status: ACTIVE)

Environment variable:

Not yet set up

Downloaded DSes:

Last Update

DS ID    DS Name                         Revision Status     (GMT-04:00)

-------- ------------------------------- -------- ---------- -------------------

10492    DS_PVDM3_DSP_Crash_Event_1      1.0      pending    2015-06-04 20:01:24

Router#

During the installation process, you are prompted with questions that are defined in the Prompts section:

Router#call-home diagnostic-signature install 10492

Number of crashdump files to be stored in the flash (1-5) 5

Enter TAC Case Number (Case number to which diagnostics data need to be uploaded)

600000001

Enter Notification Email-Address (Email address to which problem occurrence needs

to be notified) attach@cisco.com

All prompt variables are configured successfully.

Router#

Router#show call-home diagnostic-signature

Current diagnostic-signature settings:

Diagnostic-signature: enabled

Profile: CiscoTAC-1 (status: ACTIVE)

Environment variable:

Not yet set up

Downloaded DSes:

Last Update

DS ID    DS Name                         Revision Status     (GMT-04:00)

-------- ------------------------------- -------- ---------- -------------------

10492    DS_PVDM3_DSP_Crash_Event_1      1.0      registered 2015-06-04 20:01:24

Router#

Once the DS is registered, the actions that are specified in the Prerequisite section are executed. In this example, the commands that are related to DSP crashdump generation are configured:

Router#show run | section voice dsp

voice dsp crash-dump file-limit 3

voice dsp crash-dump destination flash:dsp_crashdump

Router#

Verify

This section describes how to verify that the DS is installed and operates correctly.

Trigger Event

Cisco recommends that you simulate the problem trigger in order to ensure that the DS works correctly. For example, you can simulate a DSP crash via the test voice driver command, as shown here:

Router#test voice driver

Enter Voice NM  slot number : 0

C29xx/C39xx Debugging Section;

1 - FPGA Registers Read/Write

2 - 5510 DSP test

3 - DSPRM test

5 - IOCTRL TDM Registers Read/Write

6 - IOCTRL HDLC Registers Read/Write

7 - IOCTRL TDM Memory Read/Write

8 - get conn store address

9 - TDM PLL Read/Wrire

10 - SP2600 DSP test

11 - Quit

Select option : 10

SP2600 DSP Testing Section:

1 - Display Device Information

2 -  Reset 1 DSP

3 -  Reset All DSPs

4 -  Download DSP Firmware

5 -  JTAG Read DSP Memory

6 -  JTAG Write DSP Memory

7 -  Keepalive Enable/Disable

8 -  Display DSP Keepalive Status

9 -  Simulate DSP Crash

10 - ACK Testing

11 - Set Mbrd_dsp_debug Value

12 - PLD watch dog timers Enable/Disable

13 - Send Status_Request DSP Message

14 - Display Host and DSP MAC Address

15 - Display PLD and BOOTLOADER Version

16 - GigE enable/disable port

17 - Reset TDM port

18 - Show ports receiving oversubscription tone

19 - Display firmware build string

20 - Simulate All ARM Crash

21 - Simulate All ARM Crash after All DSS Crash

22 - Read PVDM PLD register

23 - Write PVDM PLD register

24 - Import DSP command file

25 - Switch DSP application between HR image and Streamware

26 - Show video capabilities of a DSP

27 - QUIT

Select option : 9

(1=DSP, 2=ARM) :1

Enter DSP id : 1

Enter Mode:

Mode 1: Simulates Assert Condition

Mode 2: Simulates Endless loop

Mode 3: Stop High Level Responses to Commands

Enter Mode: 1

SP2600 DSP Testing Section:

1 - Display Device Information

2 -  Reset 1 DSP

3 -  Reset All DSPs

4 -  Download DSP Firmware

5 -  JTAG Read DSP Memory

6 -  JTAG Write DSP Memory

7 -  Keepalive Enable/Disable

8 -  Display DSP Keepalive Status

9 -  Simulate DSP Crash

10 - ACK Testing

11 - Set Mbrd_dsp_debug Value

12 - PLD watch dog timers Enable/Disable

13 - Send Status_Request DSP Message

14 - Display Host and DSP MAC Address

15 - Display PLD and BOOTLOADER Version

16 - GigE enable/disable port

17 - Reset TDM port

18 - Show ports receiving oversubscription tone

19 - Display firmware build string

20 - Simulate All ARM Crash

21 - Simulate All ARM Crash after All DSS Crash

22 - Read PVDM PLD register

23 - Write PVDM PLD register

24 - Import DSP command file

25 - Switch DSP application between HR image and Streamware

26 - Show video capabilities of a DSP

27 - QUIT

Select option : 27

C29xx/C39xx Debugging Section;

1 - FPGA Registers Read/Write

2 - 5510 DSP test

3 - DSPRM test

5 - IOCTRL TDM Registers Read/Write

6 - IOCTRL HDLC Registers Read/Write

7 - IOCTRL TDM Memory Read/Write

8 - get conn store address

9 - TDM PLL Read/Wrire

10 - SP2600 DSP test

11 - Quit

Select option : 11

Router#

Here is the output from the show log command:

032517: Jun  5 00:02:46.300: writing out DSP dump to file

flash:dsp_crashdump-1433462566-1

032517: Jun  5 00:02:46.300: writing out DSP dump to file

flash:dsp_crashdump-1433462566-1, sequence

032517: Jun  5 00:02:46.300: writing out DSP dump to file

flash:dsp_crashdump-1433462566-1, timestamp

032532: Jun  5 00:02:46.344: DS-ACT-TRACE: call_home_ds_regexp_paren_str_get[2571],

run regular expression once with pattern .*writing out DSP dump to file

([^[:space:]]+).*

032534: Jun  5 00:02:46.344: DS-ACT-TRACE: : writing out DSP dump to file flash:

dsp_crashdump-1433462566-1

032551: Jun  5 00:02:46.348: CALL-HOME-TRACE: Event 41 description <032517:

Jun 5 00:02:46.300: writing out DSP dump to file flash:dsp_crashdump-1433462566-1>

Enter the show call-home diagnostic-signature statistics command in order to verify whether the problem event trigger has been detected by the DS:

Router#show call-home diagnostic-signature statistics

Triggered/     Average Run  Max Run

DS ID     DS Name                          Max/Deinstall  Time(sec)    Time(sec)

--------  -------------------------------  -------------  -----------  ---------

10492     DS_PVDM3_DSP_Crash_Event_1       0/0/N          0.000        0.000

3900-12#

3900-12#

3900-12#

3900-12#show call-home diagnostic-signature statistics

Triggered/     Average Run  Max Run

DS ID     DS Name                          Max/Deinstall  Time(sec)    Time(sec)

--------  -------------------------------  -------------  -----------  ---------

10492     DS_PVDM3_DSP_Crash_Event_1       1/0/N          15.152       15.152

Router#

Validate Action

The final step that you must complete in order to verify the DS deployment is to validate whether actions such as these are performed correctly:

- Command execution

- Script execution

- Collected data transmission via Email or Smart Call Home with collected data

In this example, an email is sent to attach@cisco.com with the output of the DSP-related show commands.