CentOS - Feb 2011 - Journal Aborts in VMware ESX (Filesystem Corruption)

I have several CentOS5 hosts in a VMware ESX 3.5.0 226117 environment
using iSCSI storage.  Recently we've begun to experience journal aborts
resulting in remounted-read-only filesystems as well as other filesystem
issues - I can unmount a filesystem and force a check with "fsck -f"
and
occasionally find errors.

I've found -
<https://bugzilla.redhat.com/show_bug.cgi?id=228108>
<http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=51306>
- which seem related but I believe I am running a kernel that contains
these fixes.

My kernel is 2.6.18-194.32.1.el5 on one of the most effected hosts.

Does anyone else have experience with similar issues or know of the
status of this Bug/KB?

I can install, boot, run, and then at some seemingly random moment -

init_special_inode: bogus i_mode (50632)
init_special_inode: bogus i_mode (137147)
init_special_inode: bogus i_mode (172036)
init_special_inode: bogus i_mode (175720)
init_special_inode: bogus i_mode (72350)
init_special_inode: bogus i_mode (174751)
EXT3-fs error (device sdb2): ext3_lookup: unlinked inode 19698169 in dir
#19696695
Aborting journal on device sdb2.
init_special_inode: bogus i_mode (165661)
EXT3-fs error (device sdb2): ext3_lookup: unlinked inode 19698131 in dir
#19696695
init_special_inode: bogus i_mode (76763)
init_special_inode: bogus i_mode (3116)
init_special_inode: bogus i_mode (75363)
init_special_inode: bogus i_mode (77034)
init_special_inode: bogus i_mode (132237)
EXT3-fs error (device sdb2): ext3_lookup: unlinked inode 19698139 in dir
#19696695
init_special_inode: bogus i_mode (53031)
init_special_inode: bogus i_mode (33361)
init_special_inode: bogus i_mode (77546)
init_special_inode: bogus i_mode (6516)
EXT3-fs error (device sdb2): ext3_lookup: unlinked inode 19698143 in dir
#19696695
init_special_inode: bogus i_mode (6442)
init_special_inode: bogus i_mode (72554)
EXT3-fs error (device sdb2): ext3_lookup: unlinked inode 19698142 in dir
#19696695
EXT3-fs error (device sdb2): ext3_lookup: unlinked inode 19698164 in dir
#19696695
init_special_inode: bogus i_mode (73171)
init_special_inode: bogus i_mode (154432)
init_special_inode: bogus i_mode (34302)
init_special_inode: bogus i_mode (131733)
init_special_inode: bogus i_mode (30773)
ext3_abort called.
EXT3-fs error (device sdb2): ext3_journal_start_sb: Detected aborted
journal
Remounting filesystem read-only

On Sun, Feb 13, 2011 at 9:09 AM, Adam Tauno Williams
<awilliam at whitemice.org> wrote:
> I have several CentOS5 hosts in a VMware ESX 3.5.0 226117 environment
> using iSCSI storage. ?Recently we've begun to experience journal aborts
> resulting in remounted-read-only filesystems as well as other filesystem
> issues - I can unmount a filesystem and force a check with "fsck
-f" and
> occasionally find errors.
>
> I've found -
> <https://bugzilla.redhat.com/show_bug.cgi?id=228108>
>
<http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=51306>
> - which seem related but I believe I am running a kernel that contains
> these fixes.
>
I ran into a similar problem, but it was not specifically iSCSI.  We
ended up setting a sysctl.conf file.  Give me a few and I will find
the setting..

On Sun, Feb 13, 2011 at 9:09 AM, Adam Tauno Williams
<awilliam at whitemice.org> wrote:
> I have several CentOS5 hosts in a VMware ESX 3.5.0 226117 environment
> using iSCSI storage. ?Recently we've begun to experience journal aborts
> resulting in remounted-read-only filesystems as well as other filesystem
> issues - I can unmount a filesystem and force a check with "fsck
-f" and
> occasionally find errors.
http://communities.vmware.com/message/245983

The setting we used to resolve was vm.min_free_kbytes = 8192

Previous to this we were seeing the error pop up every week or so.