I have a 1 TB ext3 filesystem mounted via iscsi on a redhat 9 system w/ kernel version - 2.4.20-30.9. I'm not sure when it happened, but today there appears to be about 7,000 files (600GB) missing. The output from df implies that the files are still there. It shows 861 GB utilized. But du shows only 300 GB of data. I'm sure that there are no processes holding onto deleted files because I have unmounted/mounted the filesystem several times, synced, etc. Here's an excerpt from e2fsck -nf /dev/sdf ... e2fsck 1.32 (09-Nov-2002) Pass 1: Checking inodes, blocks, and sizes Inode 674 has illegal block(s). Clear? no Illegal block #32780 (2552928151) in inode 674. IGNORED. Illegal block #32781 (443979519) in inode 674. IGNORED. Illegal block #32782 (2730682564) in inode 674. IGNORED. Illegal block #32783 (1341333000) in inode 674. IGNORED. Illegal block #32784 (864228082) in inode 674. IGNORED. Illegal block #32785 (1637408843) in inode 674. IGNORED. Illegal block #32786 (2702337062) in inode 674. IGNORED. Illegal block #32787 (399755839) in inode 674. IGNORED. Illegal block #32788 (2350927161) in inode 674. IGNORED. Illegal block #32789 (972130738) in inode 674. IGNORED. Illegal block #32790 (726004907) in inode 674. IGNORED. Too many illegal blocks in inode 674. Clear inode? no [ This message basically repeats itself and eventually e2fsck errors out with... ] Error while iterating over blocks in inode 675: Illegal indirect block found e2fsck: aborted /dev/sdf: ********** WARNING: Filesystem still has errors ********** Things look pretty grim right now. As soon as I run e2fsck -p, these inodes will be deleted and I will lose data, correct? I suspect the cause of this issue was multiple hosts mounting this filesystem r/w. Does anyone know of any method that could be used to recover this data? Any help would be greatly appreciated. - Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/ext3-users/attachments/20041122/d99719bb/attachment.htm>
On Mon, Nov 22, 2004 at 06:05:56PM -0800, Jeff Dinisco wrote:> I have a 1 TB ext3 filesystem mounted via iscsi on a redhat 9 system w/ > kernel version - 2.4.20-30.9. I'm not sure when it happened, but today > there appears to be about 7,000 files (600GB) missing. The output from > df implies that the files are still there. It shows 861 GB utilized. > But du shows only 300 GB of data. I'm sure that there are no processes > holding onto deleted files because I have unmounted/mounted the > filesystem several times, synced, etc. Here's an excerpt from e2fsck > -nf /dev/sdf ...The output from df is calculated by using the free blocks field in the superblock, and is only as accurate as the superblock statistics will be. It does look like garbage has been written into your filesystem, and the situation is probably pretty grim; there will almost certainly be some data loss. It may not be as bad as you think, however, since there may be disconnected inodes (and possibly entire directory hierarchies) which e2fsck can recover and link into the lost+found directory.> Things look pretty grim right now. As soon as I run e2fsck -p, these > inodes will be deleted and I will lose data, correct?Actually, e2fsck -y will be needed to fix the data. E2fsck -p will make "safe" fixes, but anything that might require human judgement will cause e2fsck -p to abort (since it is intended to be used in unattended boot scripts). You will need to answer each of e2fsck's questions manually, or use e2fsck -y to cause e2fsck to automatically assume an answer of "yes" for each question. I'd strongly suggest making a bit-for-bit image backup of the filesystem before proceeding, since it might be possible for an expert to try other means of recoverying data after getting back what data you can with e2fsck -y.> I suspect the > cause of this issue was multiple hosts mounting this filesystem r/w.Ah, yup, that would do it. How long was the filesystem mounted by multiple hosts?> Does anyone know of any method that could be used to recover this data? > Any help would be greatly appreciated.You can try e2fsck -y and hope for the best. Aside from that, you can grep the disk looking for a specific text pattern if there is a few extremely valuable files that have to be recovered at all costs. Failing that, it will be restore from backup times..... (you did keep regular backups, right? :-) - Ted
Thanks for the reply. I'm already in the process of copying the filesystem to an image using "cp /dev/sdf /another_fs/sdf.image" and I plan to fsck -y that image. Is there another method I should use? I'm not sure how long it was mounted r/w on the other host but it's pretty clear that that's the cause. Is there particular service you can recommmend for this type of data recovery (does that answer your backup question)? Thanks. -----Original Message----- From: Theodore Ts'o [mailto:tytso at thunk.org] On Behalf Of Theodore Ts'o Sent: Tuesday, November 23, 2004 12:48 PM To: Jeff Dinisco Cc: ext3-users at redhat.com Subject: Re: files missing On Mon, Nov 22, 2004 at 06:05:56PM -0800, Jeff Dinisco wrote:> I have a 1 TB ext3 filesystem mounted via iscsi on a redhat 9 system > w/ kernel version - 2.4.20-30.9. I'm not sure when it happened, but > today there appears to be about 7,000 files (600GB) missing. The > output from df implies that the files are still there. It shows 861GB utilized.> But du shows only 300 GB of data. I'm sure that there are no > processes holding onto deleted files because I have unmounted/mounted > the filesystem several times, synced, etc. Here's an excerpt from > e2fsck -nf /dev/sdf ...The output from df is calculated by using the free blocks field in the superblock, and is only as accurate as the superblock statistics will be. It does look like garbage has been written into your filesystem, and the situation is probably pretty grim; there will almost certainly be some data loss. It may not be as bad as you think, however, since there may be disconnected inodes (and possibly entire directory hierarchies) which e2fsck can recover and link into the lost+found directory.> Things look pretty grim right now. As soon as I run e2fsck -p, these > inodes will be deleted and I will lose data, correct?Actually, e2fsck -y will be needed to fix the data. E2fsck -p will make "safe" fixes, but anything that might require human judgement will cause e2fsck -p to abort (since it is intended to be used in unattended boot scripts). You will need to answer each of e2fsck's questions manually, or use e2fsck -y to cause e2fsck to automatically assume an answer of "yes" for each question. I'd strongly suggest making a bit-for-bit image backup of the filesystem before proceeding, since it might be possible for an expert to try other means of recoverying data after getting back what data you can with e2fsck -y.> I suspect the > cause of this issue was multiple hosts mounting this filesystem r/w.Ah, yup, that would do it. How long was the filesystem mounted by multiple hosts?> Does anyone know of any method that could be used to recover thisdata?> Any help would be greatly appreciated.You can try e2fsck -y and hope for the best. Aside from that, you can grep the disk looking for a specific text pattern if there is a few extremely valuable files that have to be recovered at all costs. Failing that, it will be restore from backup times..... (you did keep regular backups, right? :-) - Ted
So I copied the image then ran e2fsck -fy on it. Good news is that it restored over 6,000 files. Bad news is it restored them to lost+found w/ no file names. It's my understanding that I should be looking for directory inodes w/ tables mapping file names to inode #'s. My guess is that these inodes have been destroyed at some point which is the reason that I discovered missing files in the 1st place. They don't appear to be in lost+found. Is there a method to recover them? Thanks. -----Original Message----- From: Theodore Ts'o [mailto:tytso at thunk.org] On Behalf Of Theodore Ts'o Sent: Tuesday, November 23, 2004 12:48 PM To: Jeff Dinisco Cc: ext3-users at redhat.com Subject: Re: files missing On Mon, Nov 22, 2004 at 06:05:56PM -0800, Jeff Dinisco wrote:> I have a 1 TB ext3 filesystem mounted via iscsi on a redhat 9 systemw/> kernel version - 2.4.20-30.9. I'm not sure when it happened, buttoday> there appears to be about 7,000 files (600GB) missing. The outputfrom> df implies that the files are still there. It shows 861 GB utilized. > But du shows only 300 GB of data. I'm sure that there are noprocesses> holding onto deleted files because I have unmounted/mounted the > filesystem several times, synced, etc. Here's an excerpt from e2fsck > -nf /dev/sdf ...The output from df is calculated by using the free blocks field in the superblock, and is only as accurate as the superblock statistics will be. It does look like garbage has been written into your filesystem, and the situation is probably pretty grim; there will almost certainly be some data loss. It may not be as bad as you think, however, since there may be disconnected inodes (and possibly entire directory hierarchies) which e2fsck can recover and link into the lost+found directory.> Things look pretty grim right now. As soon as I run e2fsck -p, these > inodes will be deleted and I will lose data, correct?Actually, e2fsck -y will be needed to fix the data. E2fsck -p will make "safe" fixes, but anything that might require human judgement will cause e2fsck -p to abort (since it is intended to be used in unattended boot scripts). You will need to answer each of e2fsck's questions manually, or use e2fsck -y to cause e2fsck to automatically assume an answer of "yes" for each question. I'd strongly suggest making a bit-for-bit image backup of the filesystem before proceeding, since it might be possible for an expert to try other means of recoverying data after getting back what data you can with e2fsck -y.> I suspect the > cause of this issue was multiple hosts mounting this filesystem r/w.Ah, yup, that would do it. How long was the filesystem mounted by multiple hosts?> Does anyone know of any method that could be used to recover thisdata?> Any help would be greatly appreciated.You can try e2fsck -y and hope for the best. Aside from that, you can grep the disk looking for a specific text pattern if there is a few extremely valuable files that have to be recovered at all costs. Failing that, it will be restore from backup times..... (you did keep regular backups, right? :-) - Ted