IP Simple Network Management Protocol (SNMP) Causes High CPU Utilization

Introduction

This document explains how to troubleshoot high CPU utilization in a router due to the SNMP ENGINE process running in the router, especially in low end routers.

Prerequisites

Requirements

Readers of this document should have knowledge of these topics:

• SNMP

• Cisco IOS

Components Used

The information in this document is based on routers running Cisco IOS^® Software.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

For more information on document conventions, refer to the Cisco Technical Tips Conventions.

High CPU Utilization Caused by the SNMP ENGINE Process

Sometimes, messages like this might appear in the router console:

%SNMP-3-CPUHOG: Processing [chars] of [chars]

They mean that the SNMP agent on the device has taken too much time to process a request.

You can determine the cause of high CPU use in a router by using the output of the show process cpu command.

The following example shows the output of the show process cpu command.

cacuk#show process cpu 
CPU utilization for five seconds: 0%/0%; 
one minute: 0%;
five minutes: 0% 

PID	Runtime(ms)	Invoked	uSecs	5Sec	1Min	5Min	TTY	Process
1	68	258816	0	0.00%	0.00%	0.00%	0	Load Meter
2	0	1	0	0.00%	0.00%	0.00%	0	OSPF Hello
3	788132	131480	5994	0.00%	0.00%	0.00%	0	Check heaps
4	0	1	0	0.00%	0.00%	0.00%	0	Chunk Manager
5	56	131	427	0.00%	0.00%	0.00%	0	Pool Manager
69	202700	421730	480	0.00%	0.01%	0.00%	0	IP SNMP
71	1193648	211250	5650	0.00%	0.19%	0.15%	0	SNMP ENGINE

The debug snmp command checks which Object IDs (OIDs) or Management Information Base (MIBs) are being queried at the time.

Note: Running a debug over a production network may overwhelm the router.

There are two probable causes for high CPU utilization occuring in relation to Simple Network Management Protocol (SNMP):

1. Large route and/or ARP tables being polled by the NMS station

2. Polling certain MIBs

Large Route and/or ARP Tables Polled by the NMS Station

The Network Management station queries routers for their entire route table to learn about other networks. It uses this information to find other routers and query them about their knowledge of networks around them. In this fashion, the management station can learn the topology of the entire network.

The router stores the route table in a hashed format, more conducive to quick route searches. However, SNMP responses for the route are required to be returned in lexicographical order per RFC1213. Therefore, for each SNMP request the router receives, the hash table must be sorted lexicographically before a SNMP response PDU can be built. The larger the route table, the more CPU intesive the sort.

SNMP is a low priority process as far as the CPU scheduler is concerned, so another process requiring CPU resources takes priority. Therefore, while CPU spikes occur in this scenario, they should not affect performance.

Recommendation 1: Use snmp-server view Commands

To avoid performance issues, force the router to prematurely end queries for the route table from the network management system server. Configure the router to respond with a complete message as soon as it receives the start of a request for the route table, as follows:

snmp-server view cutdown iso included

!--- ISO is used as a starting point as it is the snmp 'root'.

 
snmp-server view cutdown 1.3.6.1.2.1.4.21 excluded 
snmp-server view cutdown 1.3.6.1.2.1.4.22 excluded 
snmp-server view cutdown 1.3.6.1.2.1.3 excluded
snmp-server view cutdown 1.3.6.1.6.3.15 excluded

!--- The additional object snmpUsmMIB is excluded.

snmp-server view cutdown 1.3.6.1.6.3.16 excluded

!--- The additional object snmpVacmMIB is excluded.

snmp-server view cutdown 1.3.6.1.6.3.18 excluded

!--- The additional object snmpCommunityMIB is excluded.
 
snmp-server community public view cutdown RO
snmp-server community private view cutdown RW

Caution: In the above example, snmpUsmMIB is excluded, snmpVacmMIB is excluded, and snmpCommunityMIB is excluded. Together these objects can be used to get information about configured users and community strings to gain administrative access to the device. It is recommended that these objects be exlcluded from view on any device which can be accessed by public users.

This configuration blocks requests to retrieve the route table (ipRouteTable) and the Address Resolution Protocol (ARP) table (ipNetToMediaTable), but allows all other requests through. If you have an older versions of Cisco IOS® Software, it won't recognize the MIB object ipRouteTable, so use the following configuration instead:

snmp-server view cutdown iso included

!--- ISO is used as a starting point as it is the snmp 'root'.

snmp-server view cutdown 1.3.6.1.2.1.4.21 excluded 
snmp-server view cutdown 1.3.6.1.2.1.4.22 excluded 
snmp-server view cutdown at excluded
snmp-server view cutdown 1.3.6.1.6.3.15 excluded

!--- The additional object snmpUsmMIB is excluded.

snmp-server view cutdown 1.3.6.1.6.3.16 excluded

!--- The additional object snmpVacmMIB is excluded.

snmp-server view cutdown 1.3.6.1.6.3.18 excluded

!--- The additional object snmpCommunityMIB is excluded.
 
snmp-server community public view cutdown RO 
snmp-server community private view cutdown RW

Caution: In the above example, snmpUsmMIB is excluded, snmpVacmMIB is excluded, and snmpCommunityMIB is excluded. Together these objects can be used to get information about configured users and community strings to gain administrative access to the device. It is recommended that these objects be exlcluded from view on any device which can be accessed by public users.

In both of these examples, you can substitute your own community strings.

The outcome of these configurations is that the router no longer returns either the ARP table or the IP route table when queried. This prevents SNMP network discoveries from creating CPU spikes on the router in question, but it also removes a degree of manageability from the router.

Routers that previously were not exhibiting CPU spikes may begin to do so if there is a change in their routing table. The amount of cycles required to reply to the IP route table requests is a function of the number of routes in the routing table. If the number of routes increases, CPU utilization increases as well.

Recommendation 2: Turn on CEF

A change was made in the Cisco IOS code to allow SNMP to query the Cisco Express Forwarding (CEF) table for routing entries if CEF switching is used. This significantly improves the situation. With CEF enabled, the SNMP agent responds to a get-next/get-bulk operation for the routing or ARP tables with information from the Forwarding Information Base (FIB). The FIB is stored in lexicographical order and no sorting is needed. Without CEF enabled, the SNMP agent responds with information from the Routing Information Base (RIB), which must be sorted into lexicographical order causing high-CPU.

Follow the bug ID link below and see detailed bug information.

CSCdk54265 (registered customers only) - CPU HOG gets generated when polling routing table via SNMP from Network Management Station.

Polling Certain MIBs

Network Management Applications frequently retrieve information from devices using SNMP. One example would be Resource Manager Essentials (RME), an applications within the CiscoWorks 2000 suite of products. RME can retrieve flash information defined in the CISCO-FLASH-MIB. If the device has an ATA flash disk, it can cause CPU utilization to jump.

Follow the bug ID link below and see detailed bug information.

CSCdt97325 (registered customers only)

To verify and apply the workaround, follow these steps:

1. Verify if the device has an ATA Flash Card in one of its slots:

   Router#show disk1: 
   ******** ATA Flash Card Geometry/Format Info ******** 
   
   ATA CARD GEOMETRY 
       Number of Heads:        12 
       Number of Cylinders     906 
       Sectors per Cylinder    63 
       Sector Size             512 
       Total Sectors           684936 
   
   ATA CARD FORMAT 
       Number of FAT Sectors  84 
       Sectors Per Cluster    32 
       Number of Clusters     21372 
       Number of Data Sectors 684117 
       Base Root Sector       169 
       Base FAT Sector        1 
       Base Data Sector       201 
   
   Router# 
   
   

   If you're not sure or if you do not get output for the show diskX command, run the show tech command and do a search for disk.

2. Once you know you have an ATA Flash Disk, you can apply the following workaround to prevent the SNMP queries on the FLASH-MIB:

   Router(config)#snmp-server view <any_word> iso included
   
   !--- ISO is used as a starting point as it is the snmp 'root'.
    
   Router(config)#snmp-server view <any_word> ciscoFlashMIB excluded 
   
   !--- The additional object snmpUsmMIB is excluded.
   
   Router(config)#snmp-server view <any_word> 1.3.6.1.6.3.15 excluded
   
   !--- The additional object snmpVacmMIB is excluded.
   
   Router(config)#snmp-server view <any_word> 1.3.6.1.6.3.16 excluded
   
   !--- The additional object snmpCommunityMIB is excluded.
   
   Router(config)#snmp-server view <any_word> 1.3.6.1.6.3.18 excluded
   Router(config)#snmp-server community <read_community_string> view <any_word> ro 
   Router(config)#snmp-server community <write_community_string> view <any_word> rw 
   Router(config)#exit
   Router#

   Note: Choose the same word on every line for the <any_word> in the above configuration.

   Caution: In the above example, snmpUsmMIB is excluded, snmpVacmMIB is excluded, and snmpCommunityMIB is excluded. Together these objects can be used to get information about configured users and community strings to gain administrative access to the device. It is recommended that these objects be exlcluded from view on any device which can be accessed by public users.

3. Verify that it is correctly implemented:

   Router#show run 
   ... 
   snmp-server view <any_word> iso included
   
   !--- ISO is used as a starting point as it is the snmp 'root'.
   
   snmp-server view <any_word> ciscoFlashMIB excluded
   snmp-server view <any_word> internet.6.3.15 excluded
   
   !--- The additional object snmpUsmMIB is excluded.
   
   snmp-server view <any_word> internet.6.3.16 excluded
   
   !--- The additional object snmpVacmMIB is excluded.
   
   snmp-server view <any_word> internet.6.3.18 excluded 
   
   !--- The additional object snmpCommunityMIB is excluded.
   
   snmp-server community <write_community_string> view <any_word> RW 
   snmp-server community <read_community_string> view <any_word> RO
   

   Note: <any_word>, <write_community_string>, and <read_community_string> are those that are in the configuration.

   Caution: In the above example, snmpUsmMIB is excluded, snmpVacmMIB is excluded, and snmpCommunityMIB is excluded. Together these objects can be used to get information about configured users and community strings to gain administrative access to the device. It is recommended that these objects be exlcluded from view on any device which can be accessed by public users.

4. Issue the show proc cpu command to verify if the CPU-Util for SNMP goes down and look for the IP_SNMP process.

Note: The bug CSCdt97325 has also been fixed in some later Cisco IOS releases, so check the bug for any details.

Other bugs related to polling MIBs:

Follow the Cisco bug ID links below and see detailed bug information.

• CSCdm67427 (registered customers only) - Polling the ATM sub interface returns CPU HOG message from the device.

• CSCdu63734 (registered customers only) - Flash MIB makes too many calls to ifs.

• CSCdu48652 (registered customers only) - Flash MIB query hangs voice calls on 7200.

• CSCds53368 (registered customers only) - Problems with ciscoFlashPartitionEntry object in CISCO-FLASH-MIB.

• CSCdu55091 (registered customers only) - 2500 snmpgetnext for certain SNMP objects causes trace back.

• CSCdx54836 (registered customers only) - SNMP polling on Flash MIB causes high cpu utilization on the switch.

Related Information

• Troubleshooting High CPU Utilization on Cisco Routers
• Technical Support & Documentation - Cisco Systems