Troubleshoot Excessive Disk Utilization on Sourcefire Appliances

Introduction

A FireSIGHT Management Center or a FirePOWER appliance can run out of disk space for various reasons. When happens, the high disk utilization triggers health alert or may fail a software update attempt. This article describes the root causes of excessive disk utilization and some troubleshooting steps.

Verification Steps

Determine the partition that is highly used. The following command shows the disk utilization:

On a FireSIGHT Management Center,

admin@3DSystem:~# df -TH

On 7000 and 8000 Series appliances and on NGIPS virtual devices,

> show disk

Both commands show an output like below:

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5             2.9G  566M  2.2G  21% /
/dev/sda1              99M   16M   79M  17% /boot
/dev/sda7              52G  8.5G   41G  18% /Volume
none                   11G   20K   11G   1% /dev/shm
/dev/sdb1             418G  210M  395G   1% /var/storage 

 Note: Disk size and utilization can vary on various appliance models. If this is an NGIPS virtual device, verify that the size of the partitions comply with the minimum disk space requirements.

Caution: Any additional partition that is not shown above is unsupported.

On 7000 and 8000 Series appliances and on NGIPS virtual devices, you can run the following command to display detailed disk usage statistics:

> show disk-manager

An example output:

> show disk-manager
Silo                                    Used        Minimum     Maximum
Temporary Files                         143.702 MB  402.541 MB  1.572 GB
Action Queue Results                    0 KB        402.541 MB  1.572 GB
Connection Events                       17.225 GB   3.931 GB    23.586 GB
User Identity Events                    0 KB        402.541 MB  1.572 GB
UI Caches                               587 KB      1.179 GB    2.359 GB
Backups                                 0 KB        3.145 GB    7.862 GB
Updates                                 13 KB       4.717 GB    11.793 GB
Other Detection Engine                  0 KB        2.359 GB    4.717 GB
Performance Statistics                  72.442 MB   805.082 MB  9.435 GB
Other Events                            669.819 MB  1.572 GB    3.145 GB
IP Reputation & URL Filtering           0 KB        1.966 GB    3.931 GB
Archives & Cores & File Logs            1.381 GB    3.145 GB    15.724 GB
RNA Events                              0 KB        3.145 GB    12.579 GB
File Capture                            12.089 MB   4.717 GB    14.152 GB
IPS Events                              3.389 GB    7.076 GB    15.724 GB

If the /Volume Partition is Full

Old Backup Files

• If you store large volume of old backup files on the system, it can take excessive space on your disk.

Troubleshooting Steps

• Delete the old backup files using the web user interface. In order to remove backup files, navigate to System > Tools > Backup/Restore.

Tip: On a FireSIGHT System, you can configure remote storage to store the large backup files.

Older Software Update and Patch Files

• If you always keep the previous software update, upgrade, and patch files (such as, 5.0 or 5.1), the system can run out disk space.

Troubleshooting Steps

• Delete the older update and patch files that are no longer necessary. In order to delete them, please navigate to System > Updates.

Excessive Event Files Are Stored

• Managed device or sensor might have stopped sending events to the FireSIGHT Management Center.
• A device may be generating more events than a Management Center is designed to receive (per second).
• There might be a communication issue between the managed device and the management center.

Troubleshooting Steps

• Reapply the policy that are related to the event. For example, if you are not seeing connection events, reapply the Access Controil policy and see if any new events are now being received by the Management Center.
• If a FireSIGHT Management Center is unable to receive new IPS events, please check if there is any communication issues between the managed device and the management center.

Excessive Unknown Files

• The FireSIGHT System stores the unknown Network Discovery data (OS, host and service information).

Troubleshooting Steps

• If the system cannot determine the operating system on a host on your network, you can use Nmap to actively scan the host. Nmap uses the information it obtains from the scan to rate the possible operating systems. It then uses the operating system that has the highest rating as the host operating system identification.
• Create a correlation rule that triggers when the system detects a host with an unknown operating system.
  The rule should trigger when an discovery event occurs and the OS information for a host has changed and it meets the following conditions: OS Name is unknown.

Large Database to Store Events

• If you increase the database event limit beyond the guideline or best practice, the FireSIGHT Management Center can run out of disk space.

Troubleshooting Steps

• Check the values of the database limit. To improve disk utilization and performance, you should tailor event limits to the number of events you regularly work with. For some event types, you can disable storage.
• In order to change the database limit, please navigate to the System Policy page, click Edit next to the name of the system policy, and then click Database on the left section. To access the System Policy page, please navigate to System > Local > System Policy.

Receive Health Alerts For Over 85% Disk Utilization

Possible Reasons

• The event rate may be very high. Therefore the device is generating and storing lots of events.
• Communication problems between the managed device and FireSIGHT Management Center.

Troubleshooting Steps

• Changing the alert threshold level to 87% (Warning) and 92% (Critical) can be a simple solution to frequent health alerts.
• Read the Release Notes to see if there was a known issue with the pruning system. When a solution is available, please update the software version to the latest release to address this issue.

The /var/log/messages files contain data older than 24 hours, or larger than 25MB

Possible Reasons

• Logrotate daemon may not be working properly.

Troubleshooting Steps

• If you encounter this issue, please update the software version of your FireSIGHT Systems to the latest release. If you are running the latest version, but still experiencing this issue, please contact Cisco Technical Assistance Center (TAC).

If the Root ( / ) Partition is Full

User Files are Saved on the Root ( / ) Partition

Possible Reasons

• The root ( / ) partition is a fixed size and is not intended for personal storage.
• The /var/tmp drectory is used manually for temporary storage, instead of the /var/common directory.

Troubleshooting Steps

• Check for unnecessary files on the /root, /home, and /tmp folder. Since these folders are not created for personal storage, you can delete any personal file with rm command.

Unsupported Processes are Writing to Root ( / ) Partition

Possible Reasons

• If you install third party software which creates files on root ( / ) partition, you can experience health alert for high disk usage.

Troubleshooting Steps

• Check if any unsupported packages are installed. Run the following command to find the installed packages:
  
  

  admin@3DSystem:~$ rpm -qa --last

• Check pstree and top to see if unsupported processes are running. Run the following commands:
  
  

  admin@3DSystem:~$ pstree -ap 

  admin@3DSystem:~$ top