Ext3 users - May 2008 - needs help, root inode gone after usb bus reset on sata disks

Hello everybody,

I am new to this list, so welcome everybody.

Last 2 week I had two harddisk crashes with my ext2 file system.

This is what sort of happed with both of the disk:

I pluged in my USB to SATA converter in my harddisk that has an ext2
filesystem. I mounted the partition, went to a directory that had a DVD
image. I mounted the dvd image in the same directory and started
watching the movie. After 40 minutes the movie stops.

After some investigation I saw the ISO image was not mounted anymore.
And dmesg was showing 3 USB bus reset log entries.

I rebooted the computer and tried to mount the usb ext2 disk partition
again. But it failed.

dmesg and a fsck showed messages about the root inode being gone!

fsck.ext2 -p /dev/sdd1 did not work manual run is needed.

On the first 500GB disk I did an fsck.ext2 -y /dev/sdd1 did did not
fixed my disk it had still errors, I lost 35% of my data, but the
partition was mountable again, and the files where in the lost+found
directory.

I don't want this to happen with the second 750GB harddisk, I would like
all my data back.

fsck.ext3 -n /dev/sdd1 > fsck-crash-info.txt 2>&1
http://filebin.ca/mczmks/fsck-crash-info.zip

What should I do? What commands do you want me to run to provide more
info? How can i restore my root inode?

Thanks in advance,

Jelle de Jong

On Tue, May 27, 2008 10:56, Jelle de Jong wrote:
> image. I mounted the dvd image in the same directory and started watching
> the movie. After 40 minutes the movie stops.
Maybe someone on the list can tell you how the movie ends, if you tell us
the title :-)
> dmesg and a fsck showed messages about the root inode being gone!
> fsck.ext2 -p /dev/sdd1 did not work manual run is needed.
> On the first 500GB disk I did an fsck.ext2 -y /dev/sdd1 did did not
Do you still have the logs from these fsck runs? Might be interesting what
the exact errors were...
> fsck.ext3 -n /dev/sdd1 > fsck-crash-info.txt 2>&1
> http://filebin.ca/mczmks/fsck-crash-info.zip
Hm, filebin.ca times out, any chance to put this fsck-crash-info.txt
somewhere else? (no need to zip, gzip will do fine...)
> What should I do? What commands do you want me to run to provide more
> info? How can i restore my root inode?
500GB, 750GB...and no backups? Ouch :-\

Christian.
-- 
make bzImage, not war

On Tue, May 27, 2008 at 10:56:32AM +0200, Jelle de Jong
wrote:
>
> I pluged in my USB to SATA converter in my harddisk that has an ext2
> filesystem. I mounted the partition, went to a directory that had a DVD
> image. I mounted the dvd image in the same directory and started
> watching the movie. After 40 minutes the movie stops.
Were you doing anything else on the computer; where there any write
operations taking place?  If you were just reading from the
filesystem, the fact that your filesystem was that badly damaged makes
me deeply suspicious about your USB to SATA converter.
> fsck.ext2 -p /dev/sdd1 did not work manual run is needed.
>
> On the first 500GB disk I did an fsck.ext2 -y /dev/sdd1 did did not
> fixed my disk it had still errors, I lost 35% of my data, but the
> partition was mountable again, and the files where in the lost+found
> directory.
It looks like garbage was written into your block group descriptors,
but since the superblock looked OK, e2fsck -y tried its best, but in
this case it may have done more harm than good.  (In general, if you
see e2fsck asking permission to relocate an inode table; there's
something very wrong, and you probably want to say 'n' and do an image
level backup of the filesystem before proceeding.)
> I don't want this to happen with the second 750GB harddisk, I would
like
> all my data back.
Well, there's no guarantee the same corruption will have taken place
on your other hard drive.  Running e2fsck -n on that second hard drive
and letting an expert examine it would be a good first step, *before*
blindly running e2fsck -y.

In the next version of e2fsprogs (in development in the git
repository), e2fsck will have the ability to create an "undo" log
which will make e2fsck -y safer, but personally I've always liked to
individually hit return to say 'yes' to each >question.
> fsck.ext3 -n /dev/sdd1 > fsck-crash-info.txt 2>&1
> http://filebin.ca/mczmks/fsck-crash-info.zip
>
> What should I do? What commands do you want me to run to provide more
> info? How can i restore my root inode?
So this is from your 500GB disk, as I understand it, right?  I'd
really need to see the results of "e2fsck -n" *before* you ran
"e2fsck
-y" but seeing what I see there, taking an image-level backup before
you had begun would have been really good idea.

I'm not sure there's anythign you'll be able to do about restoring
your root inode.  But if it was just the root inode that was
destroyed, that's actually not a big deal; you'll just have files in
lost+found, and you can usually piece together the root directory
fairly easily.

The bigger problem is the other parts of the filesystem that were
corrupted, due to what was apparently a hardware failure.  I'm
actually really not a fan of USB as an interconnect for disks, because
the cables can be flakey; it's not that hard for them to come lose,
which may have been what caused your USB<->SATA converter to flake
out, but it apparently did so in a very spectacular fashion.

When I have time I'll have to add a better automated hueristic to
e2fsck try to do this automatically (although even when I make e2fsck
-y smarter, there *still* will be cases where a human with experience
and intelligence and common sense will do better than a program), but
for now, if you see a message about wanting to relocate an inode
table, you'll want to look at the output of "dumpe2fs /dev/sdXX",
"dumpe2fs -o superblock=32768 /dev/sdXX", and "dumpe2fs -o
superblock=98304 /dev/sdXX" (these numbers are assuming a 4k
blocksize, which is the common default).  If the location of the inode
table blocks makes more sense when dumpe2fs is told to look at the
backup superblock at 32768, it may be that e2fsck -b 32768 /dev/sdXX
will do a better job of recovering the filesystem.

     	  	     		       - Ted