Hi My sabayon box locked up while doing a repo sync, now I am getting the following errors on one directory on my btrfs partition rm: cannot remove ‘system.2/.repo/projects/packages/wallpapers/HoloSpiral.git/refs’: Directory not empty rm: cannot remove ‘system.2/.repo/projects/packages/wallpapers/HoloSpiral.git/hooks’: Stale NFS file handle rm: cannot remove ‘system.2/.repo/projects/packages/wallpapers/HoloSpiral.git/HEAD’: Stale NFS file handle Doing a btrfsck fails with the following error # btrfsck /dev/sda2 checking extents leaf parent key incorrect 46329503744 bad block 46329503744 owner ref check failed [46329503744 4096] Errors found in extent allocation tree checking fs roots btrfsck: btrfsck.c:394: process_inode_item: Assertion `!(rec->ino !key->objectid || rec->refs > 1)'' failed. Aborted Kernel Linux homeu1 3.5.0-sabayon btrfs Btrfs Btrfs v0.19-dirty Can this be repaired ? The drive is still mountable and I have moved all other files to another drive so I could wipe it but I was wondering if repairing was an option as well.. thanks Nz -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi, On Fri, Aug 17, 2012 at 09:02:07PM -0700, Not Zippy wrote:> My sabayon box locked up while doing a repo sync, now I am getting the > following errors on one directory on my btrfs partitionCan you be more specific how did you get to the state? Did you see any related error messages in syslog (from the lockup or from normal fs use that leads to the following errors)?> rm: cannot remove > ?system.2/.repo/projects/packages/wallpapers/HoloSpiral.git/refs?: > Directory not empty > rm: cannot remove > ?system.2/.repo/projects/packages/wallpapers/HoloSpiral.git/hooks?: > Stale NFS file handle > rm: cannot remove > ?system.2/.repo/projects/packages/wallpapers/HoloSpiral.git/HEAD?: > Stale NFS file handleSo this is a ESTALE error code. If I understand correctly the ?..? marks around the filename, the names of the files can be obtained but subsequent stat fails.> Doing a btrfsck fails with the following error > # btrfsck /dev/sda2 > checking extents > leaf parent key incorrect 46329503744 > bad block 46329503744 > owner ref check failed [46329503744 4096] > Errors found in extent allocation treeOne of the blocks is corrupted, in a way that prevents the file(s) to be removed, from the fsck log there''s bad link between leaf and it parent node plus some refcounting mismatches. As it''s the ''extent allocation tree'', fsck should be able to repair this type of error.> checking fs roots > btrfsck: btrfsck.c:394: process_inode_item: Assertion `!(rec->ino !> key->objectid || rec->refs > 1)'' failed. > Aborted > > Kernel > Linux homeu1 3.5.0-sabayon > btrfs > Btrfs Btrfs v0.19-dirty > > Can this be repaired ? The drive is still mountable and I have moved > all other files to another drive so I could wipe it but I was > wondering if repairing was an option as well..If you still have the image around, please try ''btrfsck --repair'' on it, and use the latest source version from cmason''s repository, unless your distro ships it (can''t tell from the version string). david -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Thanks for the analysis, unfortunately I get the same assert error when I attempt to run the repair from the compiled source. # ./work/builds/btrfs-progs/btrfsck usage: btrfsck dev Btrfs Btrfs v0.19 # ./work/builds/btrfs-progs/btrfsck --repair /dev/sda2 enabling repair mode checking extents leaf parent key incorrect 46329503744 bad block 46329503744 owner ref check failed [46329503744 4096] repair deleting extent record: key 46329503744 168 4096 adding new tree backref on start 46329503744 len 4096 parent 256 root 256 repaired damaged extent references checking fs roots btrfsck: btrfsck.c:397: process_inode_item: Assertion `!(rec->ino !key->objectid || rec->refs > 1)'' failed. Aborted -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Aug 21, 2012 at 09:50:58AM -0700, Not Zippy wrote:> Thanks for the analysis, unfortunately I get the same assert error > when I attempt to run the repair from the compiled source. > > # ./work/builds/btrfs-progs/btrfsck > usage: btrfsck dev > Btrfs Btrfs v0.19''git describe'' would be more helpful, but as the following command succeded, I assume that you''re on the right version.> # ./work/builds/btrfs-progs/btrfsck --repair /dev/sda2 > enabling repair mode > checking extents > leaf parent key incorrect 46329503744 > bad block 46329503744 > owner ref check failed [46329503744 4096] > repair deleting extent record: key 46329503744 168 4096 > adding new tree backref on start 46329503744 len 4096 parent 256 root 256 > repaired damaged extent referencesso it''s able to fix that error, but due to failure in the next phase the change is not permanent -- a quick hack here would be to commit immediatelly after this phase --- a/btrfsck.c +++ b/btrfsck.c @@ -3572,6 +3572,8 @@ int main(int ac, char **av) if (ret) fprintf(stderr, "Errors found in extent allocation tree\n"); + goto out; + fprintf(stderr, "checking fs roots\n"); ret = check_fs_roots(root, &root_cache); if (ret) ---> checking fs roots > btrfsck: btrfsck.c:397: process_inode_item: Assertion `!(rec->ino !> key->objectid || rec->refs > 1)'' failed. > AbortedThis would need more information to analyze further, like full output of btrfs-debug-tree, and capture actual values from the Assertion to match them against the output. Additionally, I''m still interested in details about conditions that lead to this corruption, it''s less painful to reproduce it locally than the remote debugging ping-pong (though sometimes nothing else is available). I think it''d be interesting to dig deeper, but you were able to get to your data so it''s not probably that urgent for both of us. david -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi The full output of the btrfs-debug-tree is 190MB compressed, did you want it still ? As far as the conditions, I was running a repo sync which I had CTRL-Z, I then got distracted and mistakenly started the sync again (not sure if you are familiar with repo command, it spawns git processes to checkout project). The crash did not occur immediately but it was probably within 2 minutes of me starting this second repo sync. The lock up was bad enough that BUSIER did not reboot the PC, I had to power down. After the patch to btrfsck, I got this error: # ./work/builds/btrfs-progs/btrfsck --repair /dev/sda2 enabling repair mode checking extents leaf parent key incorrect 46329503744 bad block 46329503744 owner ref check failed [46329503744 4096] repair deleting extent record: key 46329503744 168 4096 adding new tree backref on start 46329503744 len 4096 parent 256 root 256 repaired damaged extent references *** glibc detected *** ./work/builds/btrfs-progs/btrfsck: corrupted double-linked list: 0x000000001202b220 *** ======= Backtrace: ========/lib64/libc.so.6(+0x77896)[0x7f0008c59896] /lib64/libc.so.6(+0x77cfb)[0x7f0008c59cfb] /lib64/libc.so.6(+0x784a8)[0x7f0008c5a4a8] /lib64/libc.so.6(cfree+0x6c)[0x7f0008c5d84c] ./work/builds/btrfs-progs/btrfsck[0x415db8] ./work/builds/btrfs-progs/btrfsck[0x415e15] ./work/builds/btrfs-progs/btrfsck[0x40aa9e] ./work/builds/btrfs-progs/btrfsck[0x4046c7] /lib64/libc.so.6(__libc_start_main+0xed)[0x7f0008c0636d] ./work/builds/btrfs-progs/btrfsck[0x4017f9] ======= Memory map: =======00400000-00427000 r-xp 00000000 00:22 1375631 /mnt/DevSystem/Work/builds/btrfs-progs/btrfsck 00626000-00627000 r--p 00026000 00:22 1375631 /mnt/DevSystem/Work/builds/btrfs-progs/btrfsck 00627000-00628000 rw-p 00027000 00:22 1375631 /mnt/DevSystem/Work/builds/btrfs-progs/btrfsck 01f27000-2576c000 rw-p 00000000 00:00 0 [heap] 7f0004000000-7f0004021000 rw-p 00000000 00:00 0 7f0004021000-7f0008000000 ---p 00000000 00:00 0 7f00089cb000-7f00089e0000 r-xp 00000000 08:22 298053 /lib64/libgcc_s.so.1 7f00089e0000-7f0008be0000 ---p 00015000 08:22 298053 /lib64/libgcc_s.so.1 7f0008be0000-7f0008be1000 r--p 00015000 08:22 298053 /lib64/libgcc_s.so.1 7f0008be1000-7f0008be2000 rw-p 00016000 08:22 298053 /lib64/libgcc_s.so.1 7f0008be2000-7f0008d64000 r-xp 00000000 08:22 2883622 /lib64/libc-2.14.1.so 7f0008d64000-7f0008f64000 ---p 00182000 08:22 2883622 /lib64/libc-2.14.1.so 7f0008f64000-7f0008f68000 r--p 00182000 08:22 2883622 /lib64/libc-2.14.1.so 7f0008f68000-7f0008f69000 rw-p 00186000 08:22 2883622 /lib64/libc-2.14.1.so 7f0008f69000-7f0008f6e000 rw-p 00000000 00:00 0 7f0008f6e000-7f0008fef000 r-xp 00000000 08:22 2883678 /lib64/libm-2.14.1.so 7f0008fef000-7f00091ee000 ---p 00081000 08:22 2883678 /lib64/libm-2.14.1.so 7f00091ee000-7f00091ef000 r--p 00080000 08:22 2883678 /lib64/libm-2.14.1.so 7f00091ef000-7f00091f0000 rw-p 00081000 08:22 2883678 /lib64/libm-2.14.1.so 7f00091f0000-7f00091f4000 r-xp 00000000 08:22 394806 /lib64/libuuid.so.1.3.0 7f00091f4000-7f00093f3000 ---p 00004000 08:22 394806 /lib64/libuuid.so.1.3.0 7f00093f3000-7f00093f4000 r--p 00003000 08:22 394806 /lib64/libuuid.so.1.3.0 7f00093f4000-7f00093f5000 rw-p 00004000 08:22 394806 /lib64/libuuid.so.1.3.0 7f00093f5000-7f0009415000 r-xp 00000000 08:22 2883603 /lib64/ld-2.14.1.so 7f00095d1000-7f00095d4000 rw-p 00000000 00:00 0 7f0009612000-7f0009615000 rw-p 00000000 00:00 0 7f0009615000-7f0009616000 r--p 00020000 08:22 2883603 /lib64/ld-2.14.1.so 7f0009616000-7f0009617000 rw-p 00021000 08:22 2883603 /lib64/ld-2.14.1.so 7f0009617000-7f0009618000 rw-p 00000000 00:00 0 7fff70f01000-7fff70f23000 rw-p 00000000 00:00 0 [stack] 7fff70fff000-7fff71000000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Aborted -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Aug 22, 2012 at 10:19:13AM -0700, Not Zippy wrote:> The full output of the btrfs-debug-tree is 190MB compressed, did you > want it still ?That''s not the biggest one I''ve received :) (2.5G xz compressed) Please note that it contains filenames, so you may want to filter them out with sed -e ''s/\(.* name: \).*/\1FILENAME/'' (though sometimes the filename can be useful to give further hints).> As far as the conditions, I was running a repo sync which I had > CTRL-Z, I then got distracted and mistakenly started the sync again > (not sure if you are familiar with repo command, it spawns git > processes to checkout project). The crash did not occur immediately > but it was probably within 2 minutes of me starting this second repo > sync. The lock up was bad enough that BUSIER did not reboot the PC, I > had to power down.Thanks. Does not seem to be doing anything special and uncommon. The hard power off could have caught the filesystem in some intermediate state (ie. a bug to resolve, this should not happen) or the disk wrote bad data. smartctl may show if there are hard erros on the disk.> After the patch to btrfsck, I got this error: > > # ./work/builds/btrfs-progs/btrfsck --repair /dev/sda2[...] I''m afraid that the corrupted block confuses fsck too much. david -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html