Ext3 users - Dec 2010 - Overwritten beginning of ext3 filesystem. Recovery?

Hello list

I accidentally trashed the first ~10-20GB of a 1TB ext3 filesystem with
a heedless RAID1 rebuild (excruciating detail below). I'm now looking for
options to get as much as possible of the remaining data back.

I've been searching the web for over a day now but all my results are  
either not
what I need (MBR, partition table and superblock are OK) or too lowlevel
(revocering many thousands of nameless and structureless mails/jpgs/docs just
doesn't cut it here, IMVHO).
My main problem is not that I accidentally deleted files, but that  
basically my
/ directory just went "poof" and left the rest sitting around.

Since the damaged filesystem was clean before my accident, I'm figuring I
just
might get most of the data back: even much of the directory structure should
still be there if I only knew how to get at it.

I'd be most grateful for any tips, tools, or even documentation to aid in
writing my own tool.

Thans in advance for your time,
Florian Weber

PS: that _was_ my backup :-( Thanks for not mentioning it.

-------------

Details:

Starting point:
---------------
I've been running the following setup on my machine:
* Two same-size harddisks, currently 1TB, one big partition each -->  
sda[1], sdb[1]
* Linux software RAID1 consisting of these partitions --> md0
* A single ext3 filesystem, default parameters, reserved blocks lowered to 1%
* All system and data inside this single partition, ca. 350-400GB
* (Much too) infrequent backups ... yes, yes, I know, I know ...

Intentions:
-----------
After many years, I wanted to move from Gentoo to KUbuntu. No big deal:
* Shutdown PC, pull disk sdb from the RAID
* Install Ubuntu on sda as if working on a blank disk (setup as above, with
one of the RAID1 disks physically missing during the install)
* Boot the new system from sda, still in degraded mode
* Treating sdb like a standalone ext3 disk: mount, copy configs and /home,
umount
* Get the system into working order (config files reconciled, all applications
running)
* Determine that the "old" stuff is not needed anymore
* Put sdb back into the RAID1 and rebuild

What went wrong:
----------------
Before the initial shutdown, I did not change the partition type on sdb from
0xFD to 0x83 to prevent RAID autodetection. Booting with sdb  
reattached (to get
at my personal data) would therefore (correctly) have resulted in a  
RAID rebuild --> very bad.

So I figured: I'll attach the disk, boot with "raid=noautodetect"
in the
kernel commandline, and I'll be fine. But: unlike my previous setup,  
Ubuntu has
a silent bootloader and I missed my chance to enter the commandline.

And the RAID instantly started rebuilding itself onto my backup disk :-O

I quickly realised what was happening and cleanly shut down my
system (incurring some additional damage from the running rebuild, but  
the worst
was already done). Total running time was about 3 minutes, in parallel to the
system booting up and shutting down.

What I have now:
----------------
* A working, new Ubuntu installation on a degraded RAID1 array, without
personal data. I'm currently typing on this system.

* A harddisk (sdb) that previously contained a working system with a total of
350-400GB data, but was subject to a RAID1 rebuild for <3-4 minutes
at <=100MB/sec. The disc is not connected at the moment.

* The MBR on sdb is the new one. That's OK.
* The partition table on sdb is the new one. It looks identical to the  
old one.
* The ext3 superblock on sdb1 is the new one. It's basically the same  
as the old
one. I compared it against one of the (old) backup superblocks at the  
end of the
partition.
* I have a dd image of partition sdb1
* I can mount the image of sdb1 and do an ls. I see data from the new  
system. Much content is missing, obviously, since it was not synced  
over yet
* I can "fsck -n" the image of sdb1. Many errors of course
("inode contains
invalid block", "too many illegal blocks", "i_size
wrong", "i_Blocks wrong"),
since much stuff was not synced over yet
* At some point, "fsck -n" stops with "illegal indirect
block"
* I have not yet tried to "fsck -y". That would be my next step.

* I have 1TB of free space available and can organise more

I do realise this is not for the faint of heart, but I'm done with my  
fainting for this instance ;-)

Still with hope,
Florian Weber

-------------------------------------------------------
Buergernetz Pfaffenhofen Webmail - http://www.bn-paf.de

Hello list

For the benefit of those searching the archives, here's how I got out of the
mess described in my first mail on27dec2010.

0a. Be extra careful and use your brain before even touching the keyboard. Go 
to extreme measures to prevent typos.
0b. Keep backups: make a dd copy of your disk/partition and put the original 
hardware into a safe. Do not work on the master image. Make it read-only, 
create _another_ working copy and use _that_ for recovery. You will make 
mistakes and you do not want to touch the hardware.
0c. Be prepared to learn a lot of things about filesystems you never wanted to 
know.
0c. Whatevery you do, check the units your tools are using. Each time. They 
might be filesystem blocks, disk blocks, inodes, bytes, kB, kiB, ....
0d. Have a look at the thread "recovery recommendations" started by
"m.p." on
21jan2011

1. I overwrote the stuff I definitely knew to be faulty with zeroes, i.e. the 
first 10GB. I erred on the safe side, rather keeping bogus stuff than deleting 
good data.
2. This action killed my partition table. I had to restore it manually but was 
prepared for that.
3. I ran e2fsck on the broken partition. After the first run, it prompted me to 
be run again, which I did.
4. The filesystem now contained lost+found/ as it's only toplevel directory.
Below that were many files and directories
5. I sorted those files/dirs and gave them meaningful names instead of block 
numbers. About 70% were recognizable from their content, among it *all* my 
personal data! The rest was unrecognizable binary and ASCII fragments which I 
discarded, there's a high likelyhood that most of it was actually deleted in
the old filesystem, and it couldn't be restored anyway.
6. I compared against very old backups which showed no data loss.
7. I'm still doing lots of random samples to check for damaged files and
loss
of newer files, but found none so far.

Conclusions:
I got all my data back, but that was pure luck.
To account not only for  hardware but also for software and human failures, I 
have bought a USB harddisk which I use for weekly backups. I'm still 
evaluating which backup tools best fit my needs.

Hope this helps someone.

With best regards,
Florian Weber