I got a I/O error on one of my disks that caused the journal to abort
(error below) and be mounted in read only mode. I rebooted and the file
system check ran automatically and resolved the error. I did some searching
around and couldn't find a likely cause (the disk is a virtual disk, shared
from a SAN for VMWare ESXi5.0).
An hour after bringing the system back up, we got another one. Same sort of
error, different inode. Went to run level 1, and ran fsck manually (running
now).
I don't see any obvious errors in the vmware-log file for the VM. Or on the
SAN. All the other VMs are running fine and I don't see any issues there.
Is there a good process to follow when trying to debug an I/O error like
this? There are so many variables in this case. Wasn't sure if there were
any ext3 utilities that I don't know about but should be using
Error:
Oct 18 02:52:38 shiva clamd[3978]: SelfCheck: Database status OK.
Oct 18 03:00:03 shiva kernel: end_request: I/O error, dev sdb, sector
51083840
Oct 18 03:00:03 shiva kernel: Aborting journal on device sdb10.
Oct 18 03:00:03 shiva kernel: ext3_abort called.
Oct 18 03:00:03 shiva kernel: EXT3-fs error (device sdb10):
ext3_journal_start_sb: Detected aborted journal
Oct 18 03:00:03 shiva kernel: Remounting filesystem read-only
Oct 18 03:00:03 shiva kernel: EXT3-fs error (device sdb10) in
ext3_reserve_inode_write: Journal has aborted
Oct 18 03:00:03 shiva kernel: EXT3-fs error (device sdb10) in
ext3_reserve_inode_write: Journal has aborted
Oct 18 03:02:38 shiva clamd[3978]: SelfCheck: Database status OK.
