Ext3 users - Jun 2014 - Re: [long] major problems on fs; e2fsck running out of memory

On Sun, Jun 01, 2014 at 11:24:51PM -0400, Theodore Ts'o
wrote:
> 
> The "get root inode failed" is rather unfortunate.
Heh, I like your understatement.  :)  I think this helps answer part of
my questions in my second email: I should probably try to preserve
changes from last backup before getting too deep into a tricky e2fsck.
At one point the fs was still mountable, so I could have tried to copy
files off first.  (In a physical failure scenario it's exactly what I'd
have done, but I wasn't thinking of that in this case.)
> Try running "debugfs /dev/dm0"
> 
> and then use the "stat /" command.
No happiness:

# ./e2fsprogs-1.42.10/debugfs/debugfs /dev/dm-0
debugfs 1.42.10 (18-May-2014)
debugfs:  stat /
stat: A block group is missing an inode table while reading inode 2

My hunch is that it would take a large and lucky effort to try to get
anything useful off this fs.  Does that seem like a reasonable guess?

--keith

-- 
kkeller@wombat.san-francisco.ca.us

On Sun, Jun 01, 2014 at 08:54:24PM -0700, Keith Keller
wrote:
> 
> Heh, I like your understatement.  :)  I think this helps answer part of
> my questions in my second email: I should probably try to preserve
> changes from last backup before getting too deep into a tricky e2fsck.
> At one point the fs was still mountable, so I could have tried to copy
> files off first.  (In a physical failure scenario it's exactly what
I'd
> have done, but I wasn't thinking of that in this case.)
Yeah, it would have been nice to have preserved the outputs from
earlier e2fsck runs just so we could see what e2fsck did that
apparently ended up overwriting parts of the block group descriptors.

It might be possible to read the block group descriptors associated
with one of the backup superblocks to find the portion of the inode
table associated with inode #2.  (i.e., try using "debugfs -s 32768
/dev/dm0").

One of the things that might have detected the problem sooner, and
perhaps allowed you to recover more smoothly, would be to try running
e2fsck immediately after running resize2fs.  With the vintage kernel
and e2fsprogs shipping with the version of CentOS you are apparently
using, online resizing is probably safer than off-line --- although if
you are using the 1.42.10 version of resize2fs and the 2.6.32 based
kernel, I'd probably sugges off-line resizes as being more safe.  And
either way, running e2fsck on the file system after the resize is
probably a good idea.
> My hunch is that it would take a large and lucky effort to try to get
> anything useful off this fs.  Does that seem like a reasonable guess?
I'd try using the backup superblock approach, but if that doesn't
work, yes, that's probably a reasonable conclusion.

Regards,

					- Ted

On Mon, Jun 02, 2014 at 07:30:25AM -0400, Theodore Ts'o
wrote:
> Yeah, it would have been nice to have preserved the outputs from
> earlier e2fsck runs just so we could see what e2fsck did that
> apparently ended up overwriting parts of the block group descriptors.
I think I still have most of my e2fsck outputs.  The only ones I don't
have, which might be the most helpful, are the initial one or two, when
I thought the recovery would be easy.  I can post them somewhere if you
think it'd be helpful in tracking down a bug (they are probably too long
to post to the list).
> One of the things that might have detected the problem sooner, and
> perhaps allowed you to recover more smoothly, would be to try running
> e2fsck immediately after running resize2fs.  With the vintage kernel
> and e2fsprogs shipping with the version of CentOS you are apparently
> using, online resizing is probably safer than off-line --- although if
> you are using the 1.42.10 version of resize2fs and the 2.6.32 based
> kernel, I'd probably sugges off-line resizes as being more safe.  And
> either way, running e2fsck on the file system after the resize is
> probably a good idea.
Well, if you're going to run e2fsck, you may as well do an offline
resize, since you have to umount the filesystem anyway.

Just to clarify, the kernel I'm running is actually an OpenVZ kernel, so
I wouldn't rely totally on the version number.  I want to try to find
their changelogs to see what sort of bug fixes they've included in their
kernel.  The resize2fs was done under an older OpenVZ kernel; the
current kernel is the latest OpenVZ kernel.

Ah, here is the latest changelog:

https://openvz.org/Download/kernel/rhel6/042stab090.2

The kernel which was running for the resize was this one (which is
definitely old and crufty):

https://openvz.org/Download/kernel/rhel6/042stab055.10
> I'd try using the backup superblock approach, but if that doesn't
> work, yes, that's probably a reasonable conclusion.
Great!  Again, thank you so much for all your help.

--keith

-- 
kkeller@wombat.san-francisco.ca.us