Prime Infrastructure - Troubleshooting Low Disk Space Issues

Introduction

This document describes how to analyze commonly seen low disk space related issues in the Cisco Prime Infrastructure and their troubleshooting methodology.

The Cisco Prime Infrastructure displays a warning/critical popup message in the UI when you login to Cisco Prime Infrastructure server. Threshold crossings for these alarms are calculated based on the usage of the Prime Infrastructure optvol and localdiskvol partitions only.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Cisco Prime Infrastructure software
• Linux OS CLI

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from devices in a specific lab environment where symptoms were observed. If your network is live, make sure that you understand the potential impact of any command.

Background Information

Prime Infrastructure enables you to configure and monitor one or more controllers, switches and associated access points. Prime Infrastructure includes the same configuration, performance monitoring, security, fault management, and accounting options used at the controller level and adds a graphical view of multiple controllers and managed access points.

On Linux, Prime Infrastructure runs as a service, which runs continuously and resumes running after a reboot.

Troubleshoot

Step 1. Network Control System (NCS) cleanup utility is an inbulit utility which is used to reclaim the disk space in the Cisco Prime Infrastructure. Refer to Command Reference Guide for Cisco Prime Infrastructure 3.1 on how to use the utility and command reference.

Step 2. If the NCS cleanup utility doesn't reclaim the siginificant disk space, find out which directories consumes the disk space in the Cisco Prime Infrastructure server and causes it to run on low disk space.

You may need the access to root (ade) shell of Cisco Prime Infrastructure in order to run the next steps.

Caution: Root access should be handled by the authorised system administrator who has decent knowledge on Linux OS CLI.

1. Login in the Cisco Prime Infrastructure CLI via admin user.
   
   
2. Navigate to the root (ade) shell of the Cisco Prime Infrastructure.
   
   
3. Enter these commands:
   
   
   • Checking for /opt directory usage with df -h /opt
     
     For example:

   ade# df -h /opt
   Filesystem Size Used Avail Use% Mounted on
   /dev/mapper/smosvg-optvol 120G 54G 61G 47% /opt

   • Find which directories in /opt consuming major chunk of disk space with du -h --max-depth=6 /opt | grep [0-9]G | sort -k2

   ade# du -h --max-depth=6 /opt | grep [0-9]G | sort -k2
   54G /opt
   11G /opt/CSCOlumos
   3.4G /opt/CSCOlumos/da
   3.3G /opt/CSCOlumos/da/cdb
   3.6G /opt/CSCOlumos/staging
   2.0G /opt/CSCOlumos/staging/appdf
   1.2G /opt/CSCOlumos/staging/pf
   33G /opt/oracle
   33G /opt/oracle/base
   3.1G /opt/oracle/base/fast_recovery_area
   3.1G /opt/oracle/base/fast_recovery_area/WCS
   3.1G /opt/oracle/base/fast_recovery_area/WCS/onlinelog
   25G /opt/oracle/base/oradata
   25G /opt/oracle/base/oradata/WCS
   22G /opt/oracle/base/oradata/WCS/datafile
   3.1G /opt/oracle/base/oradata/WCS/onlinelog
   4.3G /opt/oracle/base/product
   4.3G /opt/oracle/base/product/11.2.0
   4.3G /opt/oracle/base/product/11.2.0/dbhome_1
   

   
   
4. Based on the above output, we would be able to decide if the disk space is consumed by Oracle database or the other directories.
   
   
5. Compare the /opt directory size and  /opt/oracle size, if the /opt/oracle directory's size is more then 60-70% of the /opt directory, consider adding the more disk space to the VM (if the virtual appliance) or hard-drive upgrade in case of physical appliance. 

Step 3. If you observe the /opt/oracle directory size increasing suspiciously and need further attention, please open a TAC for this and have this checked with the cisco TAC engineer.

Preventive Measures 

Low disk space can affect the Cisco Prime Infrastucture performance in various ways, ranging from slow performance to even a server crash.

You may run into other problems such as failing application backup becasue of insufficient disk space or even failure while upgrading the Cisco Prime Infrastucture. 

Taking some simple preventive measure keeps the life easy for the system administrator responsible for Cisco Prime Infrastucture and avoids the issue which may appear due to low disk in the server.

Refer to Managing Disk Space Issues  for the best practices you can follow in order keep the Cisco Prime Infrastucture disk utilization under control.