Troubleshoot CPU, Memory, and Files Usage of Tasks in StarOS

Available Languages

Download Options

  • PDF     (353.7 KB)
    View with Adobe Reader on a variety of devices
  • ePub     (358.9 KB)
    View in various apps on iPhone, iPad, Android, Sony Reader, or Windows Phone
  • Mobi (Kindle)     (178.0 KB)
    View on Kindle device or Kindle app on multiple devices
Updated:March 9, 2022
Document ID:217730

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

    Introduction

    This document describes how a task's resource usage works in StarOS and provides a list of logs that need to be collected to troubleshoot CPU/Memory/File high usage events. On StarOS, the Resource Management Subsystem (resctrl / resmgr) assigns a set of resource limits for each task in the system. It monitors each task's resource usage in order to ensure it stays within the limit. When a task has exceeded its limits, the Syslog or Simple Network Management Protocol (SNMP) traps are generated to notify the network operations.

    Resource Monitoring Mechanism

    There are a lot of tasks run on StarOS, for example sessmgr/aaamgr/vpnmgr and so on. Each task is set a limit for CPU/Memory/File usage and the limits are monitored by resource management. The limits can be different by task type (sessmgr and aaamgr have different limits), StarOS version, and hardware type. Also, the limits are defined by the system and are not configurable by users.

    The description for each task on StarOS can be found in the StarOS Tasks chapter of the System Administration Guide.

    The basic resource usage information can be found in the output of the show task resources CLI command.

    Troubleshoot CPU, Memory, and Files Usage of Tasks in StarOS - Show Task Resources Command Output

    Field Description
    cputime used CPU usage of task
    cputime allc Allocated CPU usage limit for task
    memory used Memory usage of task
    memory alloc Allocated memory usage limit for task
    files used Files usage of task
    files allc Allocated files usage of task
    status Status of task: good / warn / over

    It is important to understand that the purpose is to keep watch on resources and does not limit the task functionality. The task must be able to work even after it consumes more CPU/Memory/Files than the limit. Syslog and SNMP traps are generated when the limit is crossed, but it does not always indicate an issue.

    Suspected Cause

    In many cases, a temporary usage spike is not a problem. But if it is persistent, for example, a task's CPU usage stays at 100% or memory usage continues to grow and never be reduced, such cases need to be investigated.

    The typical causes for a temporary spike are:

    • CLI command which generates huge output (CLI task)
    • Amount of log information held in the system (evlogd task)

    The cases that need to be investigated are:

    • High CPU usage caused by the internal infinite loop (CPU usage stays at 100%)
    • The constant increase of memory usage by a memory leak or fragmentation

    The examples of the SNMP traps in sessmgr, npudrv and CLI facilities are shown here:

    Mon Aug 26 11:32:19 2013 Internal trap notification 1221 (MemoryOver) facility sessmgr instance 16 card 1 cpu 0 allocated 204800 used 220392
    Mon Aug 26 11:32:29 2013 Internal trap notification 1222 (MemoryOverClear) facility sessmgr instance 16 card 1 cpu 0 allocated 1249280 used 219608

    Fri Dec 20 13:52:20 2013 Internal trap notification 1217 (MemoryWarn) facility npudrv instance 401 card 5 cpu 0 allocated 112640 used 119588
    Fri Dec 20 14:07:26 2013 Internal trap notification 1218 (MemoryWarnClear) facility cli instance 5011763 card 5 cpu 0 allocated 56320 used 46856

    Wed Dec 25 12:24:16 2013 Internal trap notification 1220 (CPUOverClear) facility cli instance 5010294 card 5 cpu 0 allocated 600 used 272
    Wed Dec 25 12:24:16 2013 Internal trap notification 1216 (CPUWarnClear) facility cli instance 5010294 card 5 cpu 0 allocated 600 used 272

    Wed Dec 25 17:04:56 2013 Internal trap notification 1215 (CPUWarn) facility cli instance 5010317 card 5 cpu 0 allocated 600 used 595
    Wed Dec 25 17:05:36 2013 Internal trap notification 1216 (CPUWarnClear) facility cli instance 5010317 card 5 cpu 0 allocated 600 used 220

    CPU Usage

    When the CPU task usage is close or over the limit, the CPUWarn and CPUOver SNMP traps are generated along with the Syslog warning.

    SNMP Traps

    Internal trap notification 1215 (CPUWarn) facility sct instance 0 card 8 cpu 0 allocated 500 used 451
    Internal trap notification 1219 (CPUOver) facility cli instance 5010046 card 5 cpu 0 allocated 600 used 609

    In the CPUOver example, the instance number 5010046 consumes 60.9% CPU usage while the limit is 60%.

    Syslog

    [resmgr 14502 warning] [2/0/2352 <rmmgr:20> _resource_cpu.c:2876] [software internal system] The task ipsecmgr-202 is over it's cputime limit. Allocated 50.0%, Using 51.8%

    Note: This Syslog is a warning level and is not generated with the default logging setting. If this needs to be generated, the logging setting for resmgr must be configured as a warning.

    Memory Usage

    When memory task usage is close or over the limit, the MemoryWarn and MemoryOver SNMP traps are generated along with the Syslog warning.

    SNMP Traps

    Internal trap notification 1217 (MemoryWarn) facility cli instance 5005588 card 5 cpu 0 allocated 66560 used 70212
    Internal trap notification 1221 (MemoryOver) facility cli instance 5010046 card 5 cpu 0 allocated 66560 used 89940

    In the MemoryOver example, the instance number 5010046 consumes 89940 memory while the limit is 66560.

    Syslog

    [resmgr 14500 warning] [8/0/4054 <rmmgr:80> _resource_cpu.c:3622] [software internal system syslog] The task bulkstat-0 is over its memory limit. Allocated 46080K, Using 48120K

    Note: This Syslog is a warning level and is not generated with the default logging setting. If this needs to be generated, the logging setting for resmgr must be configured as a warning.

    Files Usage

    The files indicates the number of open files, or the file descriptor task uses. There is no SNMP trap for the file's usage, but a Syslog is generated when the limit is crossed.

    2013-May-28+14:16:18.746 [resmgr 14517 warning] [8/0/4440 <rmmgr:80> _resource_cpu.c:3558] [software internal system syslog] The task cli-8031369 is over its open files limit. Allocated 2000, Using 2499

    Information Needed to Troubleshoot Issues

    This section describes what information needs to be collected before you open a new Technical Assistance Center (TAC) Service Request when further investigation is needed. The log that needs to be collected is different based on the type of usage.

    Note: In addition to the list of commands, the output of the command show support detail is always required.

    CPU Usage

    Enter these commands in the StarOS CLI and capture the output:

    • show task resources
    • show task resource max
    • show snmp trap history
    • show logs
    • show profile facility <task name> instance <instance number> depth 4

    Note: The show profile command is a hidden-mode CLI command.

    Memory Usage

    Enter these commands in the StarOS CLI and capture the output:

    • show task resources
    • show task resource max
    • show snmp trap history
    • show logs

    Collect heap and system heap commands multiple times at regular intervals, for example, every 15 minutes and four outputs.

    • show messenger proclet facility <task name> instance <instance number> heap
    • show messenger proclet facility <task name> instance <instance number> system heap

    Note: The show messenger proclet command a hidden-mode CLI command.

    Files Usage

    Enter these commands in the StarOS CLI and capture the output:

    • show task resources
    • show task resource max
    • show snmp trap history
    • show logs

    Revision History

    Revision Publish Date Comments
    1.0
    09-Mar-2022
    Initial Release
    TAC Authored

    Contributed by Cisco Engineers

    • Tomonobu Okada
      Cisco Principal Engineer

    Was this Document Helpful?

    FeedbackFeedback

    Contact Cisco

    This Document Applies to These Products