I've got a really weird situation here. I'm using RHEL 4 and connecting to an EMC storage device using fibre and qla2300. The luns are put into LVM and we have a number of 400G partitions coming off that, I made a snapshot and ran fsck -yn on it with the following output: fsck 1.35 (28-Feb-2004) e2fsck 1.35 (28-Feb-2004) Pass 1: Checking inodes, blocks, and sizes Inode 2392655 has illegal block(s). Clear? no Illegal block #9 (4101620032) in inode 2392655. IGNORED. Inode 2392655, i_blocks is 672, should be 664. Fix? no Inode 29507663 has illegal block(s). Clear? no Illegal block #9 (4101620032) in inode 29507663. IGNORED. Inode 32918223 has illegal block(s). Clear? no Illegal block #9 (4101620032) in inode 32918223. IGNORED. Inode 33768943 has illegal block(s). Clear? no Illegal block #9 (4101620032) in inode 33768943. IGNORED. Inode 33771951 has illegal block(s). Clear? no Illegal block #9 (4101620032) in inode 33771951. IGNORED. Duplicate blocks found... invoking duplicate block passes. Pass 1B: Rescan for duplicate/bad blocks Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Duplicate/bad block(s) in inode 37063007: 74791522 Duplicate/bad block(s) in inode 37063010: 74791523 74791524 74791525 Duplicate/bad block(s) in inode 37389296: 74791530 Duplicate/bad block(s) in inode 37389297: 74791531 Duplicate/bad block(s) in inode 37389298: 74791532 74791533 74791537 74791538 74791543 Duplicate/bad block(s) in inode 37389300: 74791544 Duplicate/bad block(s) in inode 37389304: 74791521 Duplicate/bad block(s) in inode 37389305: 74791527 Duplicate/bad block(s) in inode 37389307: 74791520 Duplicate/bad block(s) in inode 37389325: 74791529 74791545 74791547 Duplicate/bad block(s) in inode 37389367: 74791520 74791521 74791522 74791523 74791524 74791525 74791527 74791529 74791530 74791531 74791532 74791533 74791537 74791538 74791543 74791544 74791545 74791547 Pass 1C: Scan directories for inodes with dup blocks. Pass 1D: Reconciling duplicate blocks (There are 11 inodes containing duplicate/bad blocks.) File /8/004/006/176/558/pages/html-4523408551 (inode #37063007, mod time Wed Apr 8 19:25:11 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/921/648/pages/html-4532555619 (inode #37063010, mod time Wed Apr 8 18:48:15 2009) has 3 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM (inode #37389296, mod time Mon Apr 6 09:11:33 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM (inode #37389297, mod time Mon Apr 6 09:12:00 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM (inode #37389298, mod time Mon Apr 6 09:12:20 2009) has 5 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM (inode #37389300, mod time Mon Apr 6 09:15:33 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM (inode #37389304, mod time Mon Apr 6 21:12:07 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM (inode #37389305, mod time Tue Apr 7 11:26:00 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod time Tue Apr 7 11:29:54 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM (inode #37389325, mod time Tue Apr 7 13:07:38 2009) has 3 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) has 18 duplicate block(s), shared with 10 file(s): /8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM (inode #37389325, mod time Tue Apr 7 13:07:38 2009) /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM (inode #37389300, mod time Mon Apr 6 09:15:33 2009) /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM (inode #37389298, mod time Mon Apr 6 09:12:20 2009) /8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM (inode #37389297, mod time Mon Apr 6 09:12:00 2009) /8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM (inode #37389296, mod time Mon Apr 6 09:11:33 2009) /8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM (inode #37389305, mod time Tue Apr 7 11:26:00 2009) /8/004/006/921/648/pages/html-4532555619 (inode #37063010, mod time Wed Apr 8 18:48:15 2009) /8/004/006/176/558/pages/html-4523408551 (inode #37063007, mod time Wed Apr 8 19:25:11 2009) /8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM (inode #37389304, mod time Mon Apr 6 21:12:07 2009) /8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod time Tue Apr 7 11:29:54 2009) Clone duplicate/bad blocks? no Delete file? no Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Block bitmap differences: -4806293 Fix? no Free blocks count wrong for group #2282 (0, counted=18). Fix? no Free blocks count wrong (22802537, counted=22802555). Fix? no Inode bitmap differences: -10549863 -10549865 -(10549871--10549873) -(10549876--10549879) -10549883 -(10549885--10549888) Fix? no Free inodes count wrong for group #643 (15860, counted=15846). Fix? no Free inodes count wrong (47980634, counted=47980620). Fix? no webspace2: ********** WARNING: Filesystem still has errors ********** webspace2: 4448166/52428800 files (3.4% non-contiguous), 82055063/104857600 blocks So, I took the partition offline and ran fsck -yf on it with the following output: fsck 1.35 (28-Feb-2004) e2fsck 1.35 (28-Feb-2004) Pass 1: Checking inodes, blocks, and sizes Inode 2392655 has illegal block(s). Clear? yes Illegal block #9 (4101620032) in inode 2392655. CLEARED. Inode 2392655, i_blocks is 672, should be 664. Fix? yes Inode 29507663 has illegal block(s). Clear? yes Illegal block #9 (4101620032) in inode 29507663. CLEARED. Inode 32918223 has illegal block(s). Clear? yes Illegal block #9 (4101620032) in inode 32918223. CLEARED. Inode 33768943 has illegal block(s). Clear? yes Illegal block #9 (4101620032) in inode 33768943. CLEARED. Inode 33771951 has illegal block(s). Clear? yes Illegal block #9 (4101620032) in inode 33771951. CLEARED. Duplicate blocks found... invoking duplicate block passes. Pass 1B: Rescan for duplicate/bad blocks Duplicate/bad block(s) in inode 37062984: 74791523 Duplicate/bad block(s) in inode 37063012: 74791524 74791525 Duplicate/bad block(s) in inode 37290160: 74791522 Duplicate/bad block(s) in inode 37389296: 74791530 Duplicate/bad block(s) in inode 37389297: 74791531 Duplicate/bad block(s) in inode 37389298: 74791532 74791533 74791537 74791538 74791543 Duplicate/bad block(s) in inode 37389300: 74791544 Duplicate/bad block(s) in inode 37389304: 74791521 Duplicate/bad block(s) in inode 37389305: 74791527 Duplicate/bad block(s) in inode 37389307: 74791520 Duplicate/bad block(s) in inode 37389325: 74791529 74791545 74791547 Duplicate/bad block(s) in inode 37389367: 74791520 74791521 74791522 74791523 74791524 74791525 74791527 74791529 74791530 74791531 74791532 74791533 74791537 74791538 74791543 74791544 74791545 74791547 Pass 1C: Scan directories for inodes with dup blocks. Pass 1D: Reconciling duplicate blocks (There are 12 inodes containing duplicate/bad blocks.) File /8/004/006/730/448/pages/html-4530801363 (inode #37062984, mod time Thu Apr 9 15:12:45 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/176/558/pages/4523428551-Apr__9_2009_10:22:19:750AM (inode #37063012, mod time Thu Apr 9 11:25:50 2009) has 2 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/389/058/pages/html-4527727459 (inode #37290160, mod time Thu Apr 9 14:37:51 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM (inode #37389296, mod time Mon Apr 6 09:11:33 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM (inode #37389297, mod time Mon Apr 6 09:12:00 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM (inode #37389298, mod time Mon Apr 6 09:12:20 2009) has 5 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM (inode #37389300, mod time Mon Apr 6 09:15:33 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM (inode #37389304, mod time Mon Apr 6 21:12:07 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM (inode #37389305, mod time Tue Apr 7 11:26:00 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod time Tue Apr 7 11:29:54 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM (inode #37389325, mod time Tue Apr 7 13:07:38 2009) has 3 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) has 18 duplicate block(s), shared with 11 file(s): /8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM (inode #37389325, mod time Tue Apr 7 13:07:38 2009) /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM (inode #37389300, mod time Mon Apr 6 09:15:33 2009) /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM (inode #37389298, mod time Mon Apr 6 09:12:20 2009) /8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM (inode #37389297, mod time Mon Apr 6 09:12:00 2009) /8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM (inode #37389296, mod time Mon Apr 6 09:11:33 2009) /8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM (inode #37389305, mod time Tue Apr 7 11:26:00 2009) /8/004/006/176/558/pages/4523428551-Apr__9_2009_10:22:19:750AM (inode #37063012, mod time Thu Apr 9 11:25:50 2009) /8/004/006/730/448/pages/html-4530801363 (inode #37062984, mod time Thu Apr 9 15:12:45 2009) /8/004/006/389/058/pages/html-4527727459 (inode #37290160, mod time Thu Apr 9 14:37:51 2009) /8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM (inode #37389304, mod time Mon Apr 6 21:12:07 2009) /8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod time Tue Apr 7 11:29:54 2009) Duplicated blocks already reassigned or cloned. Pass 2: Checking directory structure Directory inode 29507663 has an unallocated block #9. Allocate? yes Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Block bitmap differences: -4806293 Fix? yes Free blocks count wrong for group #2 (16059, counted=16040). Fix? yes Free blocks count wrong for group #146 (7244, counted=7245). Fix? yes Free blocks count wrong for group #2282 (0, counted=18). Fix? yes Inode bitmap differences: -10549863 -10549865 -(10549871--10549873) -(10549876--10549879) -10549883 -(10549885--10549888) Fix? yes webspace2: ***** FILE SYSTEM WAS MODIFIED ***** webspace2: 4451892/52428800 files (3.5% non-contiguous), 82037602/104857600 blocks Then I saw this in /var/log/messages: Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_readdir: directory #29507663 contains a hole at offset 4096 Apr 9 21:53:56 kernel: Aborting journal on device dm-4. Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_readdir: directory #29507663 contains a hole at offset 8192 Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_readdir: directory #29507663 contains a hole at offset 12288 Apr 9 21:53:56 kernel: ext3_abort called. Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_journal_start_sb: Detected aborted journal Apr 9 21:53:56 kernel: Remounting filesystem read-only So I took it off-line and ran fsck -yf again with the following output: fsck 1.35 (28-Feb-2004) e2fsck 1.35 (28-Feb-2004) webspace2: recovering journal Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Directory inode 29507663 has an unallocated block #1. Allocate? yes Directory inode 29507663 has an unallocated block #2. Allocate? yes Directory inode 29507663 has an unallocated block #3. Allocate? yes Directory inode 29507663 has an unallocated block #4. Allocate? yes Directory inode 29507663 has an unallocated block #5. Allocate? yes Directory inode 29507663 has an unallocated block #6. Allocate? yes Directory inode 29507663 has an unallocated block #7. Allocate? yes Directory inode 29507663 has an unallocated block #8. Allocate? yes Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Free blocks count wrong for group #2 (16029, counted=16021). Fix? yes Free blocks count wrong (22763960, counted=22763952). Fix? yes webspace2: ***** FILE SYSTEM WAS MODIFIED ***** webspace2: 4454345/52428800 files (3.5% non-contiguous), 82093648/104857600 blocks What on earth is going on?! Can I expect the partition to go read-only again? Also, over the last week a number of other partitions from the same lvm and EMC setup and they all came out with: Illegal block #9 (4101620032) in inode 803439. CLEARED. In the fsck......
I've got a really weird situation here. I'm using RHEL 4 and connecting to an EMC storage device using fibre and qla2300. The luns are put into LVM and we have a number of 400G partitions coming off that, I made a snapshot and ran fsck -yn on it with the following output: fsck 1.35 (28-Feb-2004) e2fsck 1.35 (28-Feb-2004) Pass 1: Checking inodes, blocks, and sizes Inode 2392655 has illegal block(s). Clear? no Illegal block #9 (4101620032) in inode 2392655. IGNORED. Inode 2392655, i_blocks is 672, should be 664. Fix? no Inode 29507663 has illegal block(s). Clear? no Illegal block #9 (4101620032) in inode 29507663. IGNORED. Inode 32918223 has illegal block(s). Clear? no Illegal block #9 (4101620032) in inode 32918223. IGNORED. Inode 33768943 has illegal block(s). Clear? no Illegal block #9 (4101620032) in inode 33768943. IGNORED. Inode 33771951 has illegal block(s). Clear? no Illegal block #9 (4101620032) in inode 33771951. IGNORED. Duplicate blocks found... invoking duplicate block passes. Pass 1B: Rescan for duplicate/bad blocks Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Duplicate/bad block(s) in inode 37063007: 74791522 Duplicate/bad block(s) in inode 37063010: 74791523 74791524 74791525 Duplicate/bad block(s) in inode 37389296: 74791530 Duplicate/bad block(s) in inode 37389297: 74791531 Duplicate/bad block(s) in inode 37389298: 74791532 74791533 74791537 74791538 74791543 Duplicate/bad block(s) in inode 37389300: 74791544 Duplicate/bad block(s) in inode 37389304: 74791521 Duplicate/bad block(s) in inode 37389305: 74791527 Duplicate/bad block(s) in inode 37389307: 74791520 Duplicate/bad block(s) in inode 37389325: 74791529 74791545 74791547 Duplicate/bad block(s) in inode 37389367: 74791520 74791521 74791522 74791523 74791524 74791525 74791527 74791529 74791530 74791531 74791532 74791533 74791537 74791538 74791543 74791544 74791545 74791547 Pass 1C: Scan directories for inodes with dup blocks. Pass 1D: Reconciling duplicate blocks (There are 11 inodes containing duplicate/bad blocks.) File /8/004/006/176/558/pages/html-4523408551 (inode #37063007, mod time Wed Apr 8 19:25:11 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/921/648/pages/html-4532555619 (inode #37063010, mod time Wed Apr 8 18:48:15 2009) has 3 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM (inode #37389296, mod time Mon Apr 6 09:11:33 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM (inode #37389297, mod time Mon Apr 6 09:12:00 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM (inode #37389298, mod time Mon Apr 6 09:12:20 2009) has 5 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM (inode #37389300, mod time Mon Apr 6 09:15:33 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM (inode #37389304, mod time Mon Apr 6 21:12:07 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM (inode #37389305, mod time Tue Apr 7 11:26:00 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod time Tue Apr 7 11:29:54 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM (inode #37389325, mod time Tue Apr 7 13:07:38 2009) has 3 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) has 18 duplicate block(s), shared with 10 file(s): /8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM (inode #37389325, mod time Tue Apr 7 13:07:38 2009) /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM (inode #37389300, mod time Mon Apr 6 09:15:33 2009) /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM (inode #37389298, mod time Mon Apr 6 09:12:20 2009) /8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM (inode #37389297, mod time Mon Apr 6 09:12:00 2009) /8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM (inode #37389296, mod time Mon Apr 6 09:11:33 2009) /8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM (inode #37389305, mod time Tue Apr 7 11:26:00 2009) /8/004/006/921/648/pages/html-4532555619 (inode #37063010, mod time Wed Apr 8 18:48:15 2009) /8/004/006/176/558/pages/html-4523408551 (inode #37063007, mod time Wed Apr 8 19:25:11 2009) /8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM (inode #37389304, mod time Mon Apr 6 21:12:07 2009) /8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod time Tue Apr 7 11:29:54 2009) Clone duplicate/bad blocks? no Delete file? no Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Block bitmap differences: -4806293 Fix? no Free blocks count wrong for group #2282 (0, counted=18). Fix? no Free blocks count wrong (22802537, counted=22802555). Fix? no Inode bitmap differences: -10549863 -10549865 -(10549871--10549873) -(10549876--10549879) -10549883 -(10549885--10549888) Fix? no Free inodes count wrong for group #643 (15860, counted=15846). Fix? no Free inodes count wrong (47980634, counted=47980620). Fix? no webspace2: ********** WARNING: Filesystem still has errors ********** webspace2: 4448166/52428800 files (3.4% non-contiguous), 82055063/104857600 blocks So, I took the partition offline and ran fsck -yf on it with the following output: fsck 1.35 (28-Feb-2004) e2fsck 1.35 (28-Feb-2004) Pass 1: Checking inodes, blocks, and sizes Inode 2392655 has illegal block(s). Clear? yes Illegal block #9 (4101620032) in inode 2392655. CLEARED. Inode 2392655, i_blocks is 672, should be 664. Fix? yes Inode 29507663 has illegal block(s). Clear? yes Illegal block #9 (4101620032) in inode 29507663. CLEARED. Inode 32918223 has illegal block(s). Clear? yes Illegal block #9 (4101620032) in inode 32918223. CLEARED. Inode 33768943 has illegal block(s). Clear? yes Illegal block #9 (4101620032) in inode 33768943. CLEARED. Inode 33771951 has illegal block(s). Clear? yes Illegal block #9 (4101620032) in inode 33771951. CLEARED. Duplicate blocks found... invoking duplicate block passes. Pass 1B: Rescan for duplicate/bad blocks Duplicate/bad block(s) in inode 37062984: 74791523 Duplicate/bad block(s) in inode 37063012: 74791524 74791525 Duplicate/bad block(s) in inode 37290160: 74791522 Duplicate/bad block(s) in inode 37389296: 74791530 Duplicate/bad block(s) in inode 37389297: 74791531 Duplicate/bad block(s) in inode 37389298: 74791532 74791533 74791537 74791538 74791543 Duplicate/bad block(s) in inode 37389300: 74791544 Duplicate/bad block(s) in inode 37389304: 74791521 Duplicate/bad block(s) in inode 37389305: 74791527 Duplicate/bad block(s) in inode 37389307: 74791520 Duplicate/bad block(s) in inode 37389325: 74791529 74791545 74791547 Duplicate/bad block(s) in inode 37389367: 74791520 74791521 74791522 74791523 74791524 74791525 74791527 74791529 74791530 74791531 74791532 74791533 74791537 74791538 74791543 74791544 74791545 74791547 Pass 1C: Scan directories for inodes with dup blocks. Pass 1D: Reconciling duplicate blocks (There are 12 inodes containing duplicate/bad blocks.) File /8/004/006/730/448/pages/html-4530801363 (inode #37062984, mod time Thu Apr 9 15:12:45 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/176/558/pages/4523428551-Apr__9_2009_10:22:19:750AM (inode #37063012, mod time Thu Apr 9 11:25:50 2009) has 2 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/389/058/pages/html-4527727459 (inode #37290160, mod time Thu Apr 9 14:37:51 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM (inode #37389296, mod time Mon Apr 6 09:11:33 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM (inode #37389297, mod time Mon Apr 6 09:12:00 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM (inode #37389298, mod time Mon Apr 6 09:12:20 2009) has 5 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM (inode #37389300, mod time Mon Apr 6 09:15:33 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM (inode #37389304, mod time Mon Apr 6 21:12:07 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM (inode #37389305, mod time Tue Apr 7 11:26:00 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod time Tue Apr 7 11:29:54 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM (inode #37389325, mod time Tue Apr 7 13:07:38 2009) has 3 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) has 18 duplicate block(s), shared with 11 file(s): /8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM (inode #37389325, mod time Tue Apr 7 13:07:38 2009) /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM (inode #37389300, mod time Mon Apr 6 09:15:33 2009) /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM (inode #37389298, mod time Mon Apr 6 09:12:20 2009) /8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM (inode #37389297, mod time Mon Apr 6 09:12:00 2009) /8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM (inode #37389296, mod time Mon Apr 6 09:11:33 2009) /8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM (inode #37389305, mod time Tue Apr 7 11:26:00 2009) /8/004/006/176/558/pages/4523428551-Apr__9_2009_10:22:19:750AM (inode #37063012, mod time Thu Apr 9 11:25:50 2009) /8/004/006/730/448/pages/html-4530801363 (inode #37062984, mod time Thu Apr 9 15:12:45 2009) /8/004/006/389/058/pages/html-4527727459 (inode #37290160, mod time Thu Apr 9 14:37:51 2009) /8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM (inode #37389304, mod time Mon Apr 6 21:12:07 2009) /8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod time Tue Apr 7 11:29:54 2009) Duplicated blocks already reassigned or cloned. Pass 2: Checking directory structure Directory inode 29507663 has an unallocated block #9. Allocate? yes Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Block bitmap differences: -4806293 Fix? yes Free blocks count wrong for group #2 (16059, counted=16040). Fix? yes Free blocks count wrong for group #146 (7244, counted=7245). Fix? yes Free blocks count wrong for group #2282 (0, counted=18). Fix? yes Inode bitmap differences: -10549863 -10549865 -(10549871--10549873) -(10549876--10549879) -10549883 -(10549885--10549888) Fix? yes webspace2: ***** FILE SYSTEM WAS MODIFIED ***** webspace2: 4451892/52428800 files (3.5% non-contiguous), 82037602/104857600 blocks Then I saw this in /var/log/messages: Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_readdir: directory #29507663 contains a hole at offset 4096 Apr 9 21:53:56 kernel: Aborting journal on device dm-4. Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_readdir: directory #29507663 contains a hole at offset 8192 Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_readdir: directory #29507663 contains a hole at offset 12288 Apr 9 21:53:56 kernel: ext3_abort called. Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_journal_start_sb: Detected aborted journal Apr 9 21:53:56 kernel: Remounting filesystem read-only So I took it off-line and ran fsck -yf again with the following output: fsck 1.35 (28-Feb-2004) e2fsck 1.35 (28-Feb-2004) webspace2: recovering journal Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Directory inode 29507663 has an unallocated block #1. Allocate? yes Directory inode 29507663 has an unallocated block #2. Allocate? yes Directory inode 29507663 has an unallocated block #3. Allocate? yes Directory inode 29507663 has an unallocated block #4. Allocate? yes Directory inode 29507663 has an unallocated block #5. Allocate? yes Directory inode 29507663 has an unallocated block #6. Allocate? yes Directory inode 29507663 has an unallocated block #7. Allocate? yes Directory inode 29507663 has an unallocated block #8. Allocate? yes Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Free blocks count wrong for group #2 (16029, counted=16021). Fix? yes Free blocks count wrong (22763960, counted=22763952). Fix? yes webspace2: ***** FILE SYSTEM WAS MODIFIED ***** webspace2: 4454345/52428800 files (3.5% non-contiguous), 82093648/104857600 blocks What on earth is going on?! Can I expect the partition to go read-only again? Also, over the last week a number of other partitions from the same lvm and EMC setup and they all came out with: Illegal block #9 (4101620032) in inode 803439. CLEARED. In the fsck......
On Sat, 11 Apr 2009, Hiren Joshi wrote:> The luns are put into LVM and we have a number of 400G partitions coming off > that, I made a snapshot and ran fsck -yn on it with the following output:Why did you run fsck? Did you suspect filesystem errors, i.e. were there other reasons to run fsck in the first place? (unclean shutdown, power outages, etc.)> fsck 1.35 (28-Feb-2004) > e2fsck 1.35 (28-Feb-2004)e2fsprogs-1.41.5 has been released[0] today, you may want to upgrade these before doing anything to the filesystem. Also, was there anything devices related in the kernel logs? Can you read off the raw LVM device w/o errors? When did the errors start / when was your last fsck with no errors reported? It's strange that the filesystem starts to show errors "out of the blue", so I'm fishing for hardware related issues, because: if it's software related, you'd have to upgrade the ext3 "driver", i.e. the kernel - probably not possible for w/o breaking ot of RHEL4. Christian. [0] http://e2fsprogs.sf.net/ -- Bruce Schneier isn't fooled by decoy states.
Hiren Joshi wrote:> I've got a really weird situation here. I'm using RHEL 4 and connecting > to an EMC storage device using fibre and qla2300. > > The luns are put into LVM and we have a number of 400G partitions coming > off that, I made a snapshot and ran fsck -yn on it with the following > output: > > fsck 1.35 (28-Feb-2004) > e2fsck 1.35 (28-Feb-2004) > Pass 1: Checking inodes, blocks, and sizes > Inode 2392655 has illegal block(s). Clear? no > > Illegal block #9 (4101620032) in inode 2392655. IGNORED. > Inode 2392655, i_blocks is 672, should be 664. Fix? noIs there any chance that some other node on the san has this lun mounted, or is writing to it? -Eric