Last Updated: March 2009
The Service Diagnostics feature provides a bundled set of Tool Command Language (Tcl) scripts and Embedded Event Manager (EEM) policies written and tested by subject matter experts to facilitate diagnosing common networking issues in the areas of Border Gateway Protocol (BGP), Open Shortest Path First (OSPF), and Quality of Service (QoS). A new feature called Embedded Menu Manager (EMM)-available in Cisco IOS Software Release 12.4(20)T and later Cisco IOS images-may be used to guide the user in installing and deploying these scripts and policies.
Definitions
Introduction to Service Diagnostics
The concept behind Service Diagnostics is to automate some of the vast troubleshooting experiences of Cisco engineers by using the existing scripting capabilities and embedded management tools in Cisco IOS. Cisco has been adding and enhancing such tools as EEM, Embedded Syslog Manager (ESM), and Embedded Resource Manager (ERM) to Cisco IOS over the past few years. This feature is meant to be the "glue" that combines one or more of these tools to automate common diagnostic scenarios.
The goal is to isolate the end user from the rigors of Tcl scripting, and/or EEM policy writing, and provide a simple interface for deploying and receiving feedback from scenario-specific troubleshooting scripts. Service Diagnostics provides CLI Tcl shell user interfaces as well as EMM Menu Definition Files (MDF's) for deploying troubleshooting scenarios. The scripts are posted on the Cisco Beyond Website under the Diagnostic category http://www.cisco.com/go/ciscobeyond.
Deploying Service Diagnostic Scenarios
Installing Scripts
If your image has the EMM feature, it is much easier to use the MDF vs. the ZIP file. If not, you must use the tclsh helper scripts.
Installation Using tclsh CLI parser mode
1. Download the diagnostic scripts from Cisco Beyond Website: http://www.cisco.com/go/ciscobeyond
Scripts will be organized into diagnostic scenarios, ie: sdiag_bgp_1.0.zip
2. Unzip the scenario archive.
3. On the router's disk create a directory to store all the scripts.
4. Example:
Router# mkdir svc-diag
5. The files to be copied to the dir svc-diag (per scenario) are:
BGP
· sdiag_bgp.tcl
· bgp_neighbor_loss.tcl
· bgp_neighbor_formation.tcl
· bgp_neighbor_route.tcl
· sdiag_bgp_display.tcl
· sdiag_bgp_undeploy.tcl
· collectEmailParameters.tcl
OSPF
· ospf_adj_duplicate_rid.tcl
· ospf_intf_down_detached.tcl
· ospf_miss_area_id.tcl
· ospf_stuck_cases.tcl
· ospf_timer_expired.tcl
· sdiag_ospf.tcl
· sdiag_ospf_display.tcl
· sdiag_ospf_undeploy.tcl
QoS
· sdiag_qos.tcl
· sdiag_qos_display.tcl
· sdiag_qos_undeploy.tcl
· qos_drop_packet.tcl
Resource (CPU, memory, buffer monitoring)
· sdiag_resource.tcl
· sdiag_resource_display.tcl
· sdiag_resource_undeploy.tcl
· collectEmailParameters.tcl
6. copy tftp disk#:/svc-diag
7. The router will prompt for :-
a. Address or name of remote host []?
b. Source filename []? Give the full path where the script has to be copied from
c. Destination filename []?Give the path as disk#:/svc-diag/filename.tcl
8. Create a directory to store the user libraries and the template to send email
Example:Router# mkdir user_library
9. Copy the following support files from tftp to disk following the same procedure as above to the user_library:
BGP
· email_template
· userlib.tcl
· diag_lib.tcl
· sdiag_lib.tcl
· bgp_lib.tcl
· tclIndex
· lib.tcl
OSPF
· email_template_cmd
· email_setup.tcl
· diag_lib.tcl
· tclIndex
· lib.tcl
· ospf_lib.tcl
· acl_lib.tcl
QOS
· email_template_cmd
· email_setup.tcl
· diag_lib.tcl
· tclIndex
· lib.tcl
Resource (CPU, memory, buffer monitoring)
· diag_lib.tcl
· userlib.tcl
· email_template
Note: The Diagnostic Tcl scripts posted on Cisco Beyond will be digitally signed. If your Cisco IOS router image supports Tcl script signing, you may use this feature to verify the integrity of the downloaded scripts. For more information please see the topic "Signed Tcl Scripts" here http://www.cisco.com/en/US/products/ps6441/products_feature_guide09186a00808d65fe.html.
Installation Using EMM
1. Download the MDF containing the diagnostic scripts from Cisco Beyond Website: http://www.cisco.com/go/ciscobeyond
e.g., sdiag_bfp.mdf
Note: The MDF contains all the scripts and policy files needed for a scenario. It is like a ZIP, but also contains menu definitions to guide the user.
2. Copy the MDF to a file system that the router has access to, e.g. local disk:, flash:, or tftp: server.
3. From privileged exec-mode, launch EMM
Router# emm mdf tftp://my_tftp_server/sdiag_bgp.mdf
You will be presented with a main menu as follows:
=====================================================================
BGP Diagnostics
Enter ? for help or ?# for item help
---------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Setup EEM Environment
3. Deploy BGP Neighbor Loss Diagnostic Script
4. Deploy BGP Neighbor Formation Problem Diagnostic Script
5. Deploy BGP Route Problem Diagnostic Script
6. Deploy All BGP Scripts
7. Remove Diagnostic Policies
8. Display Diagnostic Policy Configuration
9. Exit
Enter selection [9]:
4. Press the number "1" (no Enter key is needed). You will be prompted for directories for the EEM user library and user policies as follows:
Enter ? for help
Enter a directory to store the BGP diagnostic policies in the form of a URL
(excluding filename, e.g. disk0:/svc_diag
Enter value [disk0:/svc_diag]:
Enter ? for help
Enter a directory for the user library files in the form of a URL
(excluding filename, e.g. disk0:/user_lib
Enter value [disk0:/user_lib]:
Note: The MDF will query the router's available file systems and present a default directory that had sufficient free space to contain the diagnostic scripts and policies. Press the "Enter" key to accept the default.
Deploying Scenarios
Deploy the script via tclsh with parameters "notification" "configuration history option" "event history option" "user policy directory" "user library directory" where:
The value for notification can be "email or syslog or all"
The value for configuration history option, event history option can be "TRUE or FALSE"
The value for user policy and user library directories is the respective full path where the scripts and library files are stored
For each scenario, the following sections document an example command line using tclsh.
BGP Neighbor Loss
Router#tclsh disk#:/sdiag_bgp.tcl bgp_neighbor_loss.tcl email TRUE TRUE disk2:/ disk2:/user_lib
where
sdiag_bgp.tcl is the registration script
bgp_neighbor_loss.tcl is the EEM policy script
the notification is email
the configuration history option is set to TRUE
the event history option is set to TRUE
the user policy directory is disk2:
the user library directory is disk2:/user_lib
BGP Neighbor Formation:
Router#tclsh disk#:/sdiag_bgp.tcl bgp_neighbor_formation.tcl email TRUE TRUE disk2:/ disk2:/user_lib 120
where
sdiag_bgp.tcl is the registration script
bgp_neighbor_formation.tcl is the EEM policy script
the notification is email
the configuration history option is set to TRUE
the event history option is set to TRUE
the user policy directory is disk2:
the user library directory is disk2:/user_lib
the timer value is 120.
BGP Neighbor Route:
Router#tclsh disk#:sdiag_bgp.tcl bgp_neighbor_route.tcl email TRUE TRUE disk#: disk#:/user_lib 120
where
sdiag_bgp.tcl is the registration script
bgp_neighbor_route.tcl is the EEM policy script
the notification is email
the configuration history option is set to TRUE
the event history option is set to TRUE
the user policy directory is disk2:
the user library directory is disk2:/user_lib
the timer value is 120.
QoS
tclsh disk0:sdiag_qos.tcl qos_drop_packet.tcl syslog FALSE FALSE disk0: inputfile disk0:/drop_file disk0:/svc_diag disk0:/user_lib
OSPF Stuck Cases
tclsh disk0:sdiag_ospf.tcl ospf_stuck_cases.tcl email TRUE TRUE disk0:/svc_diag disk0:/user_lib
OSPF Timer Expired
tclsh disk0:sdiag_ospf.tcl ospf_timer_expired.tcl email TRUE TRUE disk0:/svc_diag disk0:/user_lib
CPU Resource
tclsh <disk#:>/<dir_name>/sdiag_resource.tcl <cpu> <notification> <configurationHistory> <EventHistory> <user_pol_dir> <user_lib_dir> <process_name or interrupt> <high_threshold> <low_threshold> <time_interval>
Memory Resource
tclsh <disk#:>/<dir_name>/sdiag_resource.tcl <memory> <notification> <configurationHistory> <EventHistory> <user_pol_dir> <user_lib_dir> <process_name> <high_threshold> <low_threshold> <time_interval> <resource_allocation>
Buffer Resource
tclsh <disk#:>/<dir_name>/sdiag_resource.tcl <buffer> <notification> <configurationHistory> <EventHistory> <user_pol_dir> <user_lib_dir> <process_name> <high_threshold> <low_threshold> <time_interval> <resource_allocation>
Deployment Using EMM
1. After installation, select menu item "2" and answer the questions regarding EEM environment variables, e.g. _email_to, _email_server, etc.
=====================================================================
BGP Diagnostics
Enter ? for help or ?# for item help
---------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Setup EEM Environment
3. Deploy BGP Neighbor Loss Diagnostic Script
4. Deploy BGP Neighbor Formation Problem Diagnostic Script
5. Deploy BGP Route Problem Diagnostic Script
6. Deploy All BGP Scripts
7. Remove Diagnostic Policies
8. Display Diagnostic Policy Configuration
9. Exit
Enter selection [9]:
2. After setting up the EEM environment, select the menu item corresponding to the desired diagnostic. You will be prompted to select the type of reporting desired:
Select the type of notification to be sent when a problem is diagnosed
1. email
2. syslog
Enter choice:
If you select email reporting, you may add more information to the report, such as configuration change history:
Do you want to log configuration history?
1. Yes
2. No
Enter choice:
Displaying Inputs
To display the inputs that have been given, invoke the file:
BGP:
· Router#tclsh disk#:/svc-diag/sdiag_bgp_display.tcl bgp_neighbor_loss.tcl (OR)
where
sdiag_bgp_display.tcl is the script that displays the BGP inputs on the router related to the BGP- EEM policy script passed as the argument
bgp_neighbor_loss.tcl is the BGP neighbor loss policy script
· Router#tclsh disk#:/svc-diag/sdiag_bgp_display.tcl bgp_neighbor_formation.tcl (OR)
where
sdiag_bgp_display.tcl is the script that displays the BGP inputs on the router related to the BGP- EEM policy script passed as the argument
bgp_neighbor_formation.tcl is the BGP neighbor formation policy script
· Router#tclsh disk#:/svc-diag/sdiag_bgp_display.tcl bgp_neighbor_route.tcl
where
sdiag_bgp_display.tcl is the script that displays the BGP inputs on the router related to the BGP- EEM policy script passed as the argument
bgp_neighbor_route.tcl is the BGP route policy script
Resource
· tclsh disk#:/dir_name/sdiag_resource_display.tcl <cpu>
OR
· tclsh disk#:/dir_name/sdiag_resource_display.tcl <memory>
OR
· tclsh disk#:/dir_name/sdiag_resource_display.tcl <buffer>
Display Status Using EMM
Simply select the menu item entitled "Display ..." In the sample BGP MDF below, type the number "8".
====================================================================
BGP Diagnostics
Enter ? for help or ?# for item help
---------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Setup EEM Environment
3. Deploy BGP Neighbor Loss Diagnostic Script
4. Deploy BGP Neighbor Formation Problem Diagnostic Script
5. Deploy BGP Route Problem Diagnostic Script
6. Deploy All BGP Scripts
7. Remove Diagnostic Policies
8. Display Diagnostic Policy Configuration
9. Exit
Enter selection [9]:
Removing Scenarios
To completely remove all the inputs related to a scenario, invoke the file:
BGP:
· Router#tclsh disk#:/svc-diag/sdiag_bgp_undeploy.tcl bgp_neighbor_loss.tcl (OR)
where
sdiag_bgp_undeploy.tcl is the script that removes the BGP inputs on the router related to the BGP- EEM policy script passed as the argument
bgp_neighbor_loss.tcl is the BGP neighbor loss policy script
· Router#tclsh disk#:/svc-diag/sdiag_bgp_undeploy.tcl bgp_neighbor_formation.tcl (OR)
where
sdiag_bgp_undeploy.tcl is the script that removes the BGP inputs on the router related to the BGP- EEM policy script passed as the argument
bgp_neighbor_formation.tcl is the BGP neighbor formation policy script
· Router#tclsh disk#:/svc-diag/sdiag_bgp_undeploy.tcl bgp_neighbor_route.tcl
where
sdiag_bgp_undeploy.tcl is the script that removes the BGP inputs on the router related to the BGP- EEM policy script passed as the argument
bgp_neighbor_route.tcl is the BGP neighbor formation policy script
QoS
Router#tclsh disk#:/svc-diag/sdiag_qos_undeploy.tcl qos_drop_packet.tcl
OSPF
Router#tclsh disk#:/svc-diag/sdiag_ospf_undeploy.tcl ospf_stuck_cases.tcl
Resource:
FOR CPU
tclsh <disk#:>/<dir_name>/ <sdiag_resource_undeploy.tcl> <cpu> <process name or interrupt>
FOR MEMORY
tclsh <disk#:>/<dir_name>/ <sdiag_resource_undeploy.tcl> <memory> <process name>
FOR BUFFER
tclsh <disk#:>/<dir_name>/ <sdiag_resource_undeploy.tcl> <buffer> <process name>
Removal Using EMM
Simply select the menu item entitled "Remove Diagnostic Policies". In the sample BGP MDF below, type the number "7".
=====================================================================
BGP Diagnostics
Enter ? for help or ?# for item help
---------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Setup EEM Environment
3. Deploy BGP Neighbor Loss Diagnostic Script
4. Deploy BGP Neighbor Formation Problem Diagnostic Script
5. Deploy BGP Route Problem Diagnostic Script
6. Deploy All BGP Scripts
7. Remove Diagnostic Policies
8. Display Diagnostic Policy Configuration
9. Exit
Enter selection [9]:
BGP Diagnostics
The BGP configuration includes BGP peers configured using direct ip address and peer-group configurations.
BGP Neighbor Loss
Triggers/Symptoms
Syslog message on the router indicating that the BGP neighbor x.x.x.x going from UP to DOWN.
The reasons could be:
1. The interface on the neighbor or the local router is shut.
2. Access-list is configured on the local or the neighbor router that is explicitly or implicitly denying the tcp, ip or udp packets from the other.
3. Any BGP configuration on the neighbor router is removed.
Actions/Outputs
We will see one of the following messages:
*Oct 16 09:34:09.904: %HA_EM-6-LOG: tmpsys:/eem_policy/bgp_neighbor_loss.tcl:
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP NEIGHBOR LOSS IS:
Neighbor x.x.x.x has gone down and does not seem to be reachable through ping. Check network connectivity.
*Oct 16 09:34:09.904: %HA_EM-6-LOG: tmpsys:/eem_policy/bgp_neighbor_loss.tcl:
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP NEIGHBOR LOSS IS:
Neighbor x.x.x.x has gone down. However, configured neighbor is reachable through ping. Check BGP configuration at the peer and any access-list restrictions between the peers.
*Oct 16 09:34:09.904: %HA_EM-6-LOG: tmpsys:/eem_policy/bgp_neighbor_loss.tcl:
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP NEIGHBOR LOSS IS:
Neighbor x.x.x.x has gone down and does not seem to be reachable through ping. Check network connectivity, BGP configuration at peer and any access-list restrictions between the peers
BGP Neighbor Formation Problem
Triggers/Symptoms
"Show ip bgp neighbor" reflects BGP neighbor remains at ACTIVE, IDLE or CONNECT. The reasons could be:
1. The interface on the local or neighbor router is down.
2. Access-list is configured on the local or the neighbor router that is explicitly or implicitly denying the tcp, ip or udp packets from the other.
3. For iBGP neighbor "update-source Loopback" is not configured OR is misconfigured on the local or neighbor router.
4. For eBGP neighbor "update-source Loopback" or "ebgp-multihop is not configured OR is misconfigured on the local or neighbor router.
5. For indirectly connected eBGP or iBGP neighbors the static route or ip routing between the peers may be missing.
Actions/Outputs
We will see one of the following messages:
*Oct 16 09:34:09.904: %HA_EM-6-LOG: tmpsys:/eem_policy/bgp_neighbor_formation.tcl:
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP NEIGHBOR FORMATION IS:
Neighbor x.x.x.x does not seem to be reachable through ping. Check network connectivity, BGP configuration at the peer and any access list restrictions between the peers.
*Oct 16 09:34:09.904: %HA_EM-6-LOG: tmpsys:/eem_policy/bgp_neighbor_formation.tcl:
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP NEIGHBOR FORMATION IS:
Neighbor x.x.x.x is reachable through ping. Check BGP configuration at the peer and any access list restrictions between the peers.
*Oct 16 09:34:09.904: %HA_EM-6-LOG: tmpsys:/eem_policy/bgp_neighbor_formation.tcl:
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP NEIGHBOR FORMATION IS:
Neighbor x.x.x.x does not seem to be reachable through ping. Check network connectivity and BGP configuration at the peer and/or any access list restrictions between the peers.
*Oct 16 09:34:09.904: %HA_EM-6-LOG: tmpsys:/eem_policy/bgp_neighbor_formation.tcl:
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP NEIGHBOR FORMATION IS:
Check BGP configuration and BGP neighbor reachability for neighbor $ipaddr at the peer and/or any access-list restrictions between the peers.
*Oct 16 09:34:09.904: %HA_EM-6-LOG: tmpsys:/eem_policy/bgp_neighbor_formation.tcl:
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP NEIGHBOR FORMATION IS:
The configuration 'neighbor x.x.x.x update-source Loopback' may be required on the local router.
The configuration 'neighbor x.x.x.x ebgp-multihop' may be required on the local router.
*Oct 16 09:34:09.904: %HA_EM-6-LOG: tmpsys:/eem_policy/bgp_neighbor_formation.tcl:
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP NEIGHBOR FORMATION IS:
Check network connectivity to the BGP neighbor x.x.x.x
BGP Route Problem
Triggers/Symptoms
This can be due to BGP route loss, route not being advertised to peer or route not getting originated as the routes are not installed in the routing table.
The reasons could be:
1. In the BGP, configuration synchronization is turned on (local router).
2. The configuration "next-hop-self" command is missing on the local router.
3. There is a mismatch in the prefix mask between the BGP configuration and the routing table.
4. All the routes in the bgp table are not valid and are not best path routes.
Actions/Outputs
We will see one of the following messages:
*Oct 16 09:34:09.904: %HA_EM-6-LOG: tmpsys:/eem_policy/bgp_neighbor_route.tcl:
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP ROUTE PROBLEM IS:
There is a mismatch between prefix mask in the IP routing table and BGP configuration network x.x.x.x mask x.x.x.x
*Oct 16 09:34:09.904: %HA_EM-6-LOG: tmpsys:/eem_policy/bgp_neighbor_route.tcl:
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP ROUTE PROBLEM IS:
Synchronization is turned on in the BGP configuration. Prefixes might not be installed on the Routing table if they are not known through IGP
*Oct 16 09:34:09.904: %HA_EM-6-LOG: tmpsys:/eem_policy/bgp_neighbor_route.tcl:
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP ROUTE PROBLEM IS:
BGP Next-hop x.x.x.x of bgp prefix ix.x.x.x/y is not reachable as ping fails; next-hop-self need to be configured in the bgp configuration of the router
*Oct 16 09:34:09.904: %HA_EM-6-LOG: tmpsys:/eem_policy/bgp_neighbor_route.tcl:
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP ROUTE PROBLEM IS:
THE ROUTES: x.x.x.x which is not marked as best path prefixes in the BGP table are not installed in the routing table and so will not be advertised to peer.
*Oct 16 09:34:09.904: %HA_EM-6-LOG: tmpsys:/eem_policy/bgp_neighbor_route.tcl:
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP ROUTE PROBLEM IS:
Neighbor x.x.x. is not being advertised any routes.
*Oct 16 09:34:09.904: %HA_EM-6-LOG: tmpsys:/eem_policy/bgp_neighbor_route.tcl:
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP ROUTE PROBLEM IS:
Best path route exists, Check for other reasons why Routes are not advertised.
QoS Diagnostics
Triggers/Symptoms
QoS script is an application to support qos drop counters. Currently qos drop counters for policymap, classmap, traffic shapping, police, matchstatement etc. are displayed by the mib graphic browser, this is not easily caught when a packet drop happens. Using qos policy can inform a user when a drop happens, without looking at the mib browser manually.
The script runs with an input-file user given and report drop counters based on the template input-file. The user can find drop counters in the template file and it can be customized if user wants an diff_only counter from policymap or a raw counter from policymap or classmap or an rawmatchstatement, etc. The user needs to specify which interface and which direction he/she wants the drop counter.
The inputfile has four fields. "diff_only counter, class name, interface name, input/output direction", where the first field is if the user wants to have diff_only counters, diff_only counter is the new drop counter between the last report time and this new report time. The second field is for which policymap/classmap/matchstatement the user wants to have a drop counter reported (the classname can be a policymap, classmap or matchstatement), the third field is for which interface the user wants to have a drop counter reported and the forth field is for which service policy direction the user wants the drop counter reported.
When diff only is specified, all classmap diff counters for the interface at the service policy user specified, will be reported, so the second field for class_name does not matter.
An example of the user input file:
yes qosc1 gi0/0 input
This one will report all different drop counters in interface giga0/0 at input service policy during this 15 minutes.
The other example can be:
no PingTest e0/0 input
PingTest is the policymap name in the interface e0/0. Our qos script will report which drop counter PingTest will have at interface e0/0 on input direction.
The other example can be:
no PingPackets gi0/0 output
PingPackets is the classmap name in the interface e0/0 here.
when 2 classmaps as inputs are given by user in inputfile as the following:
"no qosc1 giga0/0 input
no qosc1 giga0/0 output"
Actions/Outputs
We will see the different output based on different inputfile, For first example of the inputfile, we will see the following report:
*Dec 13 10:27:31.765: %HA_EM-6-LOG: tmpsys:/eem_policy/qos_drop_packet.tcl:
THE SERVICE DIAGNOSTIC MESSAGE FOR QOS DROP COUNTER is:
Interface gi0/0, service policy direction "input" on policymap "qospol1":
New drop during this 15 minutes for classmap qosc1 is 732087551, its match statement is "protocol ip"
--------------------------------------------------------------
*Dec 13 10:27:31.765: %HA_EM-6-LOG: tmpsys:/eem_policy/a.tcl:
Email result will not be sent, please look at console or buffer on service diagnostic messages
--------------------------------------------------------------
For last example of the inputfile, we will see the following output:
*Dec 12 16:28:53.361: %HA_EM-6-LOG: tmpsys:/eem_policy/qos_drop_packet.tcl:
THE SERVICE DIAGNOSTIC MESSAGE FOR QOS DROP COUNTER is:
Interface gi0/0, service policy direction "input" on policymap "qospol1":
CLASSMAP <name>: qosc1 <description>: matchAll <drop packet>: 24212663 <pre-policybyte>: 1453197660 <post-policybyte>: 437880 <match statement>: protocol ip
Interface gi0/0, service policy direction "output" on policymap "qospol10":
CLASSMAP <name>: qosc1 <description>: matchAll <drop packet>: 1326 <pre-policybyte>: 722721 <post-policybyte>: 643161 <match statement>: protocol ip
--------------------------------------------------------------
*Dec 12 16:28:53.369: %HA_EM-6-LOG: tmpsys:/eem_policy/a.tcl:
Email result will not be sent, please look at console or buffer on service diagnostic messages
The output reported 2 classmap drop counters based on user input file:
This first classmap qosc1 is a classmap in interface giga0/0 on input direction.
The second classmap qosc1 is a classmap in interface giga0/0 on output direction.
OSPF Diagnostics
OSPF Neighbor Loss
Triggers/Symptoms
The OSPF neighbor loss scenario has the following 3 policies:
ospf_intf_down_detached.tcl
This policy checks the syslog message for interface down and detached.
%OSPF-5-ADJCHG:.*DOWN, Neighbor Down: Interface down or detached"
The possible reason can be:
1. The interface shut in its own router is configured (admin down)
2. An interface shut in remote end is configured and near end has a Down Down
3. Passive-interface is configured.
ospf_miss_area_id.tcl
The policy checks the syslog error message for miss area id:
%OSPF-4-ERRRCV: Received invalid packet: mismatch area ID"
It gives addition information to explain additionally on the message and give what the user has configured in the router.
ospf_timer_expired.tcl
The policy checks syslog message for timer_expired:
%OSPF-5-ADJCHG:.*DOWN, Neighbor Down: Dead timer expired
The possible reason can be:
1. Remote end shut in Ethernet interface.
Please note, when remote end in Fastethernet interface, the syslog
message is not Dead timer expired. Instead it is
"\%OSPF-5-ADJCHG:.*DOWN, Neighbor Down: Interface down or detached"
2. ip address list is configured.
3. ospf is removed from the remote end.
Actions/Outputs
The examples for neighbor loss output policies can be:
*Nov 20 11:29:32.331: %OSPF-5-ADJCHG: Process 107, Nbr 132.0.0.1 on GigabitEthernet0/0 from FULL to DOWN, Neighbor Down: Interface down or detached
*Nov 20 11:29:33.531: %HA_EM-6-LOG: tmpsys:/eem_policy/ospf_intf_down_detached.tcl:
THE SERVICE DIAG MESSAGE FOR INTERFACE DOWN DETACHED is:
OSPF Neighbor 132.0.0.1 down is caused by interface shut down. Please "no shut" the interface GigabitEthernet0/0 to bring the adjacency up
--------------------------------------------------------------
*Nov 20 11:29:33.535: %HA_EM-6-LOG: tmpsys:/eem_policy/ospf_intf_down_detached.tcl:
Email result will not be sent, please look at console or buffer on service diagnostic messages
--------------------------------------------------------------
OSPF Neighbor Formation Problem
Triggers/Symptoms
The OSPF neighbor formation scenario includes two scripts:
ospf_adj_duplicate_rid.tcl
The script detects the syslog message:
%OSPF-4-DUP_RTRID_NBR: OSPF detected duplicate router-id .* on interface
and gives addition information for which kind of case can cause the duplicate router id.
ospf_stuck_cases.tcl
The script is a timer based script to check if the router is stuck at one of the following states:
1. Stuck at attempt state, which can be caused by wrong neighbor configuration
For example, the following configuration can cause router stuck at attempt state:
router ospf 107
log-adjacency-changes
network 131.107.0.0 0.0.255.255 area 0
neighbor 131.107.5.11 ----- The neighbor is not existing and the neighbor should be 131.107.5.2
2. Stuck at init state, can be caused by the access list in the remote end side. The script can provide a suggestion for this case.
3. Stuck at exstart/exchange state, can be caused by an mtu problem.
Actions/Outputs
For ospf neighbor formation problem, the screen output can be:
For duplicate router rid is:
THE SERVICE DIAG MESSAGE FOR ADJ DUPLICATE RID is:
The message happens when two routers are configured with the same router id. Check the router ids on them to make sure they have individual router ids and restart the ospf protocol by doing "no router ospf <ospf_id>" and "router ospf <ospf_id>" in configuration mode or "clear ip ospf <process_id> process" in super user mode
--------------------------------------------------------------
*Nov 15 09:29:30.635: %HA_EM-6-LOG: tmpsys:/eem_policy/ospf_adj_duplicate_rid.tcl:
Email result will not be sent, please look at console or buffer on service diagnostic messages
--------------------------------------------------------------
For stuck at INIT:
THE SERVICE DIAG MESSAGE FOR STUCK CASES is:
OSPF neighbor 131.107.1.2 is stuck at INIT might be due to access list on remote end blocking OSPF hellos or authentication config is present on one side, Please check the access-list or enable authentication on both sides.
--------------------------------------------------------------
*Nov 27 10:55:07.067: %HA_EM-6-LOG: tmpsys:/eem_policy/ospf_stuck_cases.tcl:
Email result will not be sent, please look at console or buffer on service diagnostic messages
--------------------------------------------------------------
For stuck at EXCHANGE:
THE SERVICE DIAG MESSAGE FOR STUCK CASES is:
OSPF neighbor 131.107.1.2 is stuck at EXCHANGE might be due to unmatched mtu, the stuck interface GigabitEthernet0/0 has mtu value 3456, Please check mtu value in remote side to make sure they are synchronized
--------------------------------------------------------------
*Nov 14 15:29:27.559: %HA_EM-6-LOG: tmpsys:/eem_policy/ospf_stuck_cases.tcl:
Email result will not be sent, please look at console or buffer on service diagnostic messages
--------------------------------------------------------------
stuck at ATTEMPT:
THE SERVICE DIAG MESSAGE FOR STUCK CASES is:
Stuck at ATTEMPT might be due to wrong neighbor configuration. The neighbor address 131.107.5.11 might not exist. Please check the correct one and reconfigure it
--------------------------------------------------------------
*Nov 27 10:43:18.083: %HA_EM-6-LOG: tmpsys:/eem_policy/ospf_stuck_cases.tcl:
Email result will not be sent, please look at console or buffer on service diagnostic messages
Resource Diagnostics
Triggers/Symptoms
CPU, Memory, or buffer resource diagnostics are all triggered by crossing of user settable usage thresholds. Service Diagnostic scripts in this area use two embedded IOS features - ERM and EEM. The scripts dynamically generate an EEM policy that uses an ERM event detector in the area to be monitored.
Actions/Outputs
The actions/outputs of these policies are limited to reporting the threshold crossings. The value-add (over using ERM directly) is the email notification capability, as well as providing a wrapper for quickly configuring ERM (which is fairly complex). A typical message follows:
*Feb 25 16:44:25.235: %HA_EM-6-LOG: tmpsys:/eem_policy/sdiag_Exec.tcl:
THE SERVICE DIAGNOSTIC MESSAGE FOR RESOURCE CPU MONITORING IS :
Process Exec with PID 227 exceeded the configured CPU utilization threshold 20
Please note, caveats:
1. Sometimes there is an EEM Tcl Error which is due to email_template not being restaged. Since this issue does not have definite steps to be reproduced the workaround for this issue is to do the following on the router:
Router#> no event manager directory user library disk#:/<user library name>
Router#> event manager directory user library disk#:/<user library name>
2. All the BGP scripts will work only with IPv4.
3. There are three sev 3 bugs CSCsx65581, CSCsx53550, CSCsx47799 and 2 sev 4 bugs CSCsx60168, CSCsx65614 that are to be resolved.
4. When the BGP configuration on the router is huge, the existing BGP policies may exceed the maxrun time value for the policy, hence the user needs to change the maxrun timer value in the policy according to their BGP configurations.