FireSIGHT System Returns "Input/Output Error" Message

Introduction

When you are working on a FireSIGHT System, you may receive a message for I/O Error or Input/Output Error. This document describes how to investigate this issue, and how to troubleshoot it.

Symptoms

• Unable to apply intrusion policy. The Task Status may display the following error message:
  
  

  Could not create directory /var/tmp/PolicyExport_XXXX:
   Input/output error

• A query for intrusion events fails. The search result may show the following error:
  
  
  
  
• Unable to load the health monitor on the web user interface.
• Unable to view the managed devices.

Verification

In order to verify the issue, follow the steps below:

Step 1: Connect to your FireSIGHT System via Secure Shell (SSH).
 
Step 2: Elevate your privilege to root user:

• On FireSIGHT Management Center and FirePOWER Appliance, run:
  
  

  admin@FireSIGHT:~$ sudo su -root@FireSIGHT:~#

• On FirePOWER Appliance, run:
  
  

  > expert
  admin@FirePOWER:~$ sudo su -
  root@FirePOWER:~# 

Step 3:  Run the following commands to investigate this issue:

• The dmesg command output shows Input/Output Error. For example:
  
  

  root@FireSIGHT:~# dmesg
  
  -sh: /bin/dmesg: Input/output error

• The ls command returns Input/Output Error. For example:
  
  

  admin@FireSIGHT:~$ ls
  
  ls: reading directory .: Input/output error

• An attempt to generate troubleshoot file generates Input/Output Error. For example:

  admin@FireSIGHT:~$ sudo sf_troubleshoot.pl
  
  /usr/local/sf/bin/sf_troubleshoot.pl: Input/output error

• I/O error messages are found on the /var/log/messages. For example:
  
  

  admin@FireSIGHT:~$ grep -i error /var/log/messages
  
  Sourcefire3D kernel: sd 2:2:0:0: scsi: Device offlined - not ready after error recovery
  Sourcefire3D kernel: end_request: I/O error, dev sda, sector 1109804126
  Sourcefire3D kernel: Buffer I/O error on device sda7, logical block 0
  Sourcefire3D kernel: lost page write due to I/O error on sda7
  Sourcefire3D kernel: Buffer I/O error on device sda7, logical block 137396224
  Sourcefire3D kernel: lost page write due to I/O error on sda7
  Sourcefire3D kernel: EXT2-fs error (device sda7): read_block_bitmap: Cannot read block 
  bitmap - block_group = 4208, block_bitmap = 13

• The Input/Output Error is found on the /var/log/action_queue.log:

  
  

  Error in tempdir() using /var/tmp/PolicyExport_XXXXX: Could not create directory
  /var/tmp/PolicyExport_XXXX: Input/output error

Solution

Gracefully reboot your appliance to perform a file system check:

root@FireSIGHT:~# reboot

If this does not resolve the issue, perform a forced reboot on the appliance:

root@FireSIGHT:~# reboot -f

After you run the reboot -f command, the FireSIGHT System restarts and performs a file system check. For example:

After a forced reboot, if you are still encountering this issue, please contact Cisco Technical Support for assistance.