Ext3 users - May 2014 - [long] major problems on fs; e2fsck running out of memory

Hello ext3 list,

I am having an odd issue with one of my filesystems, and I am hoping
someone here can help out.  Yes, I do have backups.  :)  But as is often
the case, it's nice to avoid restoring from backup if possible.  If
there is a more appropriate place for this question please let me know.

After quite a while between reboots, I saw a report on the console that
the filesystem was inconsistent and could not be automatically repaired.
After some aborted tests (which I did not log, unfortunately, I was able
to get this far:

# head fsck.out
fsck from util-linux-ng 2.17.2
/dev/mapper/vg1--sdb-lv_vz contains a file system with errors, check forced.

# time passes, progress bar gets to 51.8% with no problems, then

Pass 1: Checking inodes, blocks, and sizes
Inode 266338321 has imagic flag set.  Clear? yes

Inode 266338321 has a extra size (34120) which is invalid
Fix? yes

Inode 266338321 has compression flag set on filesystem without
compression support.  Clear? yes

# some 150k messages later

Inode 266349409, i_blocks is 94855766560840, should be 0.  Fix? yes

Inode 266349363 has a bad extended attribute block 1262962006.  Clear?
yes

Inode 266349363 has illegal block(s).  Clear? yes

Illegal block #6 (1447645766) in inode 266349363.  CLEARED.
Illegal indirect block (1447642454) in inode 266349363.  CLEARED.
Illegal block #270533644 (1702521203) in inode 266349363.  CLEARED.
Warning... fsck.ext4 for device /dev/mapper/vg1--sdb-lv_vz exited with signal
11.

I wasn't sure what that meant, and somewhat without thinking, I made
more attempts to repair the fs (including, IIRC, some attempts to mount
the filesystem ro).  Here's the next fsck attempt:

fsck from util-linux-ng 2.17.2
fsck.ext4: Group descriptors look bad... trying backup blocks...
One or more block group descriptor checksums are invalid.  Fix? yes

Group descriptor 0 checksum is invalid.  FIXED.
Group descriptor 1 checksum is invalid.  FIXED.

# many group descriptors fixed

Group descriptor 40834 checksum is invalid.  FIXED.
Group descriptor 40836 checksum is invalid.  FIXED.
/dev/mapper/vg1--sdb-lv_vz contains a file system with errors, check forced.
This doesn't bode well, but we'll try to go on...
Pass 1: Checking inodes, blocks, and sizes

# again gets to 51.8% with no problems
# again over 100k lines of errors

Inode 266356018 is too big.  Truncate? yes

Block #537922572 (62467) causes directory to be too big.  CLEARED.
Warning... fsck.ext4 for device /dev/mapper/vg1--sdb-lv_vz exited with signal
11.

I tried once more with e2fsck 1.41.12 (stock from CentOS 6), then on a
search, tried e2fsck 1.42.10 from source.

ext2fs_check_desc: Corrupt group descriptor: bad block for block bitmap
./e2fsprogs-1.42.10/e2fsck/e2fsck: Group descriptors look bad... trying backup
blocks...
/dev/mapper/vg1--sdb-lv_vz contains a file system with errors, check forced.
./e2fsprogs-1.42.10/e2fsck/e2fsck: A block group is missing an inode table while
reading bad blocks inode
This doesn't bode well, but we'll try to go on...
Pass 1: Checking inodes, blocks, and sizes

# again gets to 51.8% with no problems
# again over 100k lines of errors

Illegal block #6 (1498565709) in inode 266374005.  CLEARED.
Inode 266374005 is too big.  Truncate? yes

Block #73401356 (909652270) causes directory to be too big.  CLEARED.
Error storing directory block information (inode=266374005, block=0,
num=22224176): Memory allocation failed

/dev/mapper/vg1--sdb-lv_vz: ***** FILE SYSTEM WAS MODIFIED *****

/dev/mapper/vg1--sdb-lv_vz: ***** FILE SYSTEM WAS MODIFIED *****

Repeated attempts seem to get farther into repairs, but there's still a
large number of repairs reported, which seems scary, and it still runs
out of memory on a 128GB-memory server.  I don't currently have a
filesystem with more than 128GB of free space if I wanted to use the
scratch_files option (though if that's really the solution, I'll make a
way).

The 51.8% seems very suspicious to me.  A few weeks ago, I did an online
resize2fs, and the original filesystem was about 52% the size of the new
one (from 2.7TB to 5.3TB).  The resize2fs didn't report any errors, and
I haven't seen any physical errors in the logs, so this is the first
indication I've had of a problem.

My tune2fs output and other possible information is below.

Is there any hope for this filesytem?  The "doesn't bode well"
message
doesn't give me hope, but perhaps there's some last-ditch efforts I can
make to try to recover.  If you need any other information about the
filesystem please let me know.

--keith

# uname -a
Linux XXX.XXX 2.6.32-042stab090.2 #1 SMP Wed May 21 19:25:03 MSK 2014 x86_64
x86_64 x86_64 GNU/Linux
# free -g
             total       used       free     shared    buffers     cached
Mem:           125          0        125          0          0          0
-/+ buffers/cache:          0        125
Swap:            0          0          0
# lvs
  LV       VG      Attr   LSize  Origin Snap%  Move Log Copy%  Convert
  lv_local vg0-sda -wi-ao 19.53g
  lv_swap  vg0-sda -wi-a-  2.00g
  lv_tmp   vg0-sda -wi-ao 19.53g
  lv_usr   vg0-sda -wi-ao 19.53g
  lv_var   vg0-sda -wi-ao 19.53g
  lv_vz    vg1-sdb -wi-a-  5.36t
# vgs
  VG      #PV #LV #SN Attr   VSize  VFree
  vg0-sda   1   5   0 wz--n- 96.09g 15.96g
  vg1-sdb   1   1   0 wz--n-  5.36t     0
# pvs
  PV         VG      Fmt  Attr PSize  PFree
  /dev/sda3  vg0-sda lvm2 a--  96.09g 15.96g
  /dev/sdb1  vg1-sdb lvm2 a--   5.36t     0


tune2fs 1.41.12 (17-May-2010)
Filesystem volume name:   <none>
Last mounted on:          /vz
Filesystem UUID:          74a4ea8b-03ed-4e9c-ab01-8574517cd5af
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      ext_attr resize_inode dir_index filetype
extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink
extra_isize
Filesystem flags:         signed_directory_hash
Default mount options:    (none)
Filesystem state:         not clean with errors
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              359661568
Block count:              1438622720
Reserved block count:     60203550
Free blocks:              1030108897
Free inodes:              357427346
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      681
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Thu May 31 14:47:29 2012
Last mount time:          Sun Oct 27 21:48:21 2013
Last write time:          Fri May 30 23:22:31 2014
Mount count:              1
Maximum mount count:      21
Last checked:             Sun Oct 27 21:38:53 2013
Check interval:           15552000 (6 months)
Next check after:         Fri Apr 25 21:38:53 2014
Lifetime writes:          14 TB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     28
Desired extra isize:      28
Default directory hash:   half_md4
Directory Hash Seed:      37c55228-9d7b-4a34-bd88-8322f435b9cb
Journal backup:           inode blocks




-- 
kkeller@wombat.san-francisco.ca.us

* Keith Keller <kkeller at wombat.san-francisco.ca.us> hat geschrieben:

Hello Keith
> Yes, I do have backups.  :)
> # some 150k messages later
> # again over 100k lines of errors
> # again over 100k lines of errors
> Repeated attempts seem to get farther into repairs, but there's still a
> large number of repairs reported, which seems scary,
The number of errors and the kind of errors suggest, that there is just
bogus data on the file system, either due to hardware errors (i.e.
misdirected writes or writes of bogus data) or due to lower level software
problems (e.g. in the lvm code). Either way. I suggest, you check whether
you can manually save files modified since your last backup and then just
restore your backups. After that, check your manually saved files to
be clean before incorporating them back into the new live file system.

With a backup and the current state of your file system it's hardly worth
the efford to even try to get the errors fixed. You will invest more time
in fixing the errors than in just restoring the backup. And even after
fixing all the errors, many files may be corrupted anyways. And in my
experience, most of the time, no data is better than bad data.

Regards, Bodo

Unfortunately, there has been a huge number of bug fixes for ext4's
online resize since 2.6.32 and 1.42.11.  It's quite possible that you
hit one of them.
> The 51.8% seems very suspicious to me.  A few weeks ago, I did an online
> resize2fs, and the original filesystem was about 52% the size of the new
> one (from 2.7TB to 5.3TB).  The resize2fs didn't report any errors, and
> I haven't seen any physical errors in the logs, so this is the first
> indication I've had of a problem.
Well, actually it's not quite that simple.  There are multiple passes
to e2fsck, and the first pass is estimated to be 70% of the total
e2fsck run.  So 51.8% reported by the progress means e2fsck had gotten
74% of the way through pass 1.  So that would mean that it had got
through about inodes associated to about 3.9TB into the file system.

That being said, it's pretty clear that portions of the inode table
and block group descriptor was badly corrupted.  So I suspect there
isn't going to be much that can be done to try to repair the file
system completely.  If there are specific files you need to recover,
I'd suggest trying to recover them first before trying to do anything
else.  The good news is that probably around 75% of your files can
probably be recovered.

						- Ted

Hi Bodo and Ted,

Thank you both for your responses; they confirm what I thought might be
the case.  Knowing that I can try to proceed with your suggestions.  I
do have some followup questions for you:


On Sun, Jun 01, 2014 at 09:05:09PM -0400, Theodore Ts'o
wrote:
> Unfortunately, there has been a huge number of bug fixes for ext4's
> online resize since 2.6.32 and 1.42.11.  It's quite possible that you
> hit one of them.
Would this scenario be explained by these bugs?  I'd expect that if a
resize2fs failed, it would report a problem pretty quickly.  (But
perhaps that's the nature of some of these bugs.)
> Well, actually it's not quite that simple.  There are multiple passes
> to e2fsck, and the first pass is estimated to be 70% of the total
> e2fsck run.  So 51.8% reported by the progress means e2fsck had gotten
> 74% of the way through pass 1.  So that would mean that it had got
> through about inodes associated to about 3.9TB into the file system.
Aha!  Thanks for the clarification.  That's certainly well more than the
original fs size.
> That being said, it's pretty clear that portions of the inode table
> and block group descriptor was badly corrupted.  So I suspect there
> isn't going to be much that can be done to try to repair the file
> system completely.  If there are specific files you need to recover,
> I'd suggest trying to recover them first before trying to do anything
> else.  The good news is that probably around 75% of your files can
> probably be recovered.
So, now when I try to mount, I get an error:

# mount -o ro -t ext4 /dev/mapper/vg1--sdb-lv_vz /vz/
mount: Stale NFS file handle

That's clearly a spurious error, so I checked dmesg:

# dmesg|tail
[159891.219387] EXT4-fs (dm-0): ext4_check_descriptors: Checksum for group 42252
failed (36703!=0)
[159891.219586] EXT4-fs (dm-0): ext4_check_descriptors: Checksum for group 42253
failed (51517!=0)
[159891.219786] EXT4-fs (dm-0): ext4_check_descriptors: Checksum for group 42254
failed (51954!=0)
[159891.220025] EXT4-fs (dm-0): ext4_check_descriptors: Checksum for group 42496
failed (37296!=0)
[159891.220225] EXT4-fs (dm-0): ext4_check_descriptors: Checksum for group 42497
failed (31921!=0)
[159891.220451] EXT4-fs (dm-0): ext4_check_descriptors: Checksum for group 42498
failed (2993!=0)
[159891.220650] EXT4-fs (dm-0): ext4_check_descriptors: Checksum for group 42499
failed (59056!=0)
[159891.220850] EXT4-fs (dm-0): ext4_check_descriptors: Checksum for group 42500
failed (28571!=22299)
[159891.225762] EXT4-fs (dm-0): get root inode failed
[159891.227436] EXT4-fs (dm-0): mount failed

and before that there are many other checksum failed errors.  When I
try a rw mount I get these messages instead:

[160052.031554] EXT4-fs (dm-0): ext4_check_descriptors: Checksum for group 0
failed (43864!=0)
[160052.031782] EXT4-fs (dm-0): group descriptors corrupted!

Are there any other options I can try to force the mount so I can try to
get to the changed files?  If that'll be challenging, I'll just
sacrifice
those files, but if it'd be relatively straightforward I'd like to make
the attempt.

Thanks again!

--keith

-- 
kkeller@wombat.san-francisco.ca.us

* "Theodore Ts'o" <tytso@mit.edu> hat geschrieben:

Hi Theodore.
> That being said, it's pretty clear that portions of the inode table
> and block group descriptor was badly corrupted. [...]
Keith is not the first one with problems of this class and he will
probably not be the last one. He later told us, that at first, mounting
the file system still worked. And that acually means (taking low level
software errors or hardware errors out of the equation), that actually
e2fsck created the current situation. In my opinion, e2fsck has one major
flaw actually causing this sort of troubles:

e2fsck tries to fix the errors as it actually finds them. That's bad,
because at that point it's still unclear, whether the problem can be
safely fixed yet. So, the thing e2fsck SHOULD do is:

1. Scan the file system for all errors, remember the errors BUT DON'T
   TOUCH ANYTHING.
2. Once all errors (including differences in allocation bitmaps) have been
   collected, it should then first summarize the bugs (like: 100 times
   checksum errors, 56000 times illegal pointers etc) and then ask, what
   to do.
3. Actually fix the errors one-by-one taking into account calculated
   allocation bitmaps (instead of the ones stored in the file system).
   Some errors have to be fixed before other ones, resolving multiple used
   clusters being the first kind of errors to be fixed).

This would not only allow the user to cancel at this point with no changes
to the file system being done yet, it would also allow e2fsck to make
sure, that newly allocated clusters will always go to clusters, which are
actually not in use.

What do you think about this?

Regards, Bodo