Troubleshoot High Swap Memory Utilization Issues with CPS

Introduction

This document describes the procedure to troubleshoot High Swap Memory Utilization Issues with Cisco Policy Suit (CPS).

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Linux
• CPS

Note: Cisco recommends that you must have privilege root access to CPS CLI.

Components Used

The information in this document is based on these software and hardware versions:

• CPS 20.2
• Unified Computing System (UCS)-B

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

Swap space in Linux is used when the amount of physical memory (Random Access Memory (RAM)) is full. If the system needs more memory resources and the RAM is full, inactive pages in memory are moved to the swap space. While swap space can help machines with a small amount of RAM, it must not be considered a replacement for more RAM. Swap space is located on hard drives, which have a slower access time than physical memory. Swap space can be a dedicated swap partition (recommended), a swap file, or a combination of swap partitions and swap files.

In years past, the recommended amount of swap space increased linearly with the amount of RAM in the system. However, modern systems often include hundreds of gigabytes of RAM. As a consequence, recommended swap space is considered a function of system memory workload, not system memory.

Memory swap is a memory reclamation method wherein memory contents not currently in use are swapped to a disk to make the memory available for other applications or processes. The exact state or "page" of memory is copied to the disk to make the data contiguous and easy to restore later.

Like virtual memory, swap space is a secondary memory. It is used by the Operating System (OS) when there is no physical memory available for further execution of the processes. If the OS faces a situation when it needs memory, but the RAM is full, it moves the inactive pages from RAM to swap memory.

To check for available free swap space in your machine, you can use the free command.

[root@dc1-qns01 ~]# free -m
total used free shared buff/cache available
Mem: 15876 4918 7750 792 3207 9987
Swap: 4095 0 4095
[root@dc1-qns01 ~]#

Problem

A higher percentage of swap use is normal when provisioned modules make heavy use of the disk. High swap usage must be a sign that the system experiences memory pressure.

This alarm gets generated whenever available swap memory on the Virtual Machine (VM) is lower than the configured threshold value.

Also, high swap memory usage is observed on pcrfclient01 VM at the tome of resiliency event. Diagnostics fails after runs the diagnostic.sh due to low swap memory issue in the pcrfclient VM.

Checking swap space for all VMs...
Checking swap memory usage on pcrfclient01...[FAIL]
Swap usage is 1835 MB. This will likely lead to a slowdown in your system!
Please ensure your memory is provisioned properly.
If systems memory usage is no longer high, you can reset swap with: swap2ram.sh

Condition: Diagnostics fails after runs the diagnostic.sh due to low swap memory issue in the pcrfclient VM.

Procedure to Resolve High Swap Memory Utilization Issues with CPS

Approach 1.

Solution to High swap memory usage is observed on pcrfclient01 VM at the time of resiliency event.

1. Log in to the pcrfclient VM and check the disk memory by the use of the  du -sh *  command.

2. Check /var/lib/carbon/whisper/cisco/quantum directory for memory usage. If an increase in the memory usage is observed, then issue is due to the statistics files.

3. To recover memory, delete the statistics (.wsp) files or increase the VM disk memory.

4. To recover the memory, delete the statistics manually from the folder,   /var/lib/carbon/whisper/cisco/quantum  or delete the statistics by the use of these steps: 

• Run  df –h command and note down the disk spaces.
• Run these commands:

        #du -h --max-depth=0 /var/lib/carbon/whisper/cisco/quantum/qps
        #monit stop grafana-server
        #monit stop carbon-cache
        #vi /etc/carbon/storage-schemas.conf

• Update the retention period (from 90 days to 30 days) which is located at the end of the file. This results in more aggressive retention period.
  Older - retentions: 10s:1d,60s:90d
  Update to - retentions: 10s:1d,60s:30d
  
  
• After the update of retention period, run these commands:
                     

         #monit start carbon-cache
         #monit start grafana-server

Note: Alternatively, you can use  systmctl start/stop/restart  command instead of  monit start/stop/restart  command.

• Create a file named resize.sh and add the these.

          #monit stop carbon-cache
          #cd /var/lib/carbon/whisper/cisco/quantum/qps
          #find ./ -type f -name '*.wsp' -exec whisper-resize --nobackup {} $1 \;
          #chown -R carbon:carbon *
          #monit start carbon-cache
          #monit restart grafana-server

• Add the necessary permissions by the use of the  chmod 777 resize.sh  command.
• Run  ./resize.sh "updated_retention_value"

Sample Command: #./resize.sh "10s:1d 60s:30d"

• Once the script is completed, run the these commands to confirm the amount of data has reduced by the comparison of previous and current result.

           #run df –h: disk space should reduce or less than previous result
           #du -h --max-depth=0 /var/lib/carbon/whisper/cisco/quantum/qps

• Verify that the Grafana displays the last 30 days statistics and not more than 30 days. (updated retention period is 30 days).

Approach 2.

General recommendation to resolve high swap memory utilization.

To clear the swap memory on your system, you simply need to cycle off the swap. This moves all data from swap memory back into RAM. It also means that you need to be sure you have the RAM to support this operation. An easy way to do this is to run  free -m  to see what is used in swap and in RAM. Once you power it off, you can wait an arbitrary amount of time (30 sec or so) to give the operation time to complete, then power the swap back on. This clears the swap memory cache and re-enables it. Here are all of the commands you need.

1. Check swap space usage: # free -m

2. Disable swap:  # swapoff -a
Wait approx 30 sec. Run free -m to see the amount of swap used/available decrease over time.

3. Enable swap:# swapon -a