Just happened while writing a huge avi file to my usb3 backup disk: [356036.596292] ------------[ cut here ]------------ [356036.596300] kernel BUG at fs/btrfs/inode.c:1588! [356036.596304] invalid opcode: 0000 [#1] SMP [356036.596307] CPU 2 [356036.596309] Modules linked in: vmnet(O) vmblock(O) vsock(O) vmci(O) vmmon(O) af_packet snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss nls_iso8859_15 nls_cp437 vfat fat btusb bluetooth zram(C) loop snd_usb_audio snd_hwdep snd_usbmidi_lib snd_rawmidi snd_seq_device gspca_sonixj gspca_main videodev v4l2_compat_ioctl32 pcspkr i2c_i801 evdev unix fuse xfs nfs nfs_acl auth_rpcgss lockd sunrpc reiserfs scsi_wait_scan hid_monterey hid_microsoft hid_logitech hid_ezkey hid_cypress hid_chicony hid_cherry hid_belkin hid_apple hid_a4tech usbhid usb_storage hid sr_mod cdrom sg pata_cmd64x [last unloaded: microcode] [356036.596346] [356036.596349] Pid: 28747, comm: btrfs-fixup-1 Tainted: G C O 3.2.1-gentoo-r2 #1 To Be Filled By O.E.M. To Be Filled By O.E.M./Z68 Pro3 [356036.596355] RIP: 0010:[<ffffffff811605fe>] [<ffffffff811605fe>] btrfs_writepage_fixup_worker+0xde/0x121 [356036.596363] RSP: 0000:ffff8801e2379de0 EFLAGS: 00010246 [356036.596366] RAX: 0000000000000000 RBX: ffffea00019b1a00 RCX: 0000000000000000 [356036.596370] RDX: 0000000000000000 RSI: ffffffffffffffff RDI: ffff88008a1bbb40 [356036.596373] RBP: 00000000033fd000 R08: ffff8801e2379d2c R09: 0000000180240024 [356036.596377] R10: 0000000000000000 R11: bf800000bf800000 R12: ffff88008a1bbc10 [356036.596380] R13: 0000000000000000 R14: ffff8801e2379df8 R15: 00000000033fdfff [356036.596384] FS: 0000000000000000(0000) GS:ffff88043fb00000(0000) knlGS:0000000000000000 [356036.596387] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [356036.596391] CR2: 00007f5ef9660000 CR3: 00000003253f2000 CR4: 00000000000406e0 [356036.596394] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [356036.596398] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [356036.596401] Process btrfs-fixup-1 (pid: 28747, threadinfo ffff8801e2378000, task ffff8802d2160650) [356036.596405] Stack: [356036.596407] ffff88008a1bbab0 ffff88026d847540 ffffffffffffffff ffff88003a7c1f50 [356036.596412] 0000000000000000 ffff88019bf62d80 ffff88019bf62dd0 ffff88019bf62d98 [356036.596417] ffff88019bf62da8 ffff8802d2160650 ffff88019bf62d88 ffffffff8117f23f [356036.596421] Call Trace: [356036.596426] [<ffffffff8117f23f>] ? worker_loop+0x170/0x485 [356036.596431] [<ffffffff8117f0cf>] ? btrfs_queue_worker+0x272/0x272 [356036.596435] [<ffffffff8117f0cf>] ? btrfs_queue_worker+0x272/0x272 [356036.596439] [<ffffffff810489fb>] ? kthread+0x7a/0x82 [356036.596445] [<ffffffff81444634>] ? kernel_thread_helper+0x4/0x10 [356036.596449] [<ffffffff81048981>] ? kthread_worker_fn+0x135/0x135 [356036.596453] [<ffffffff81444630>] ? gs_change+0xb/0xb [356036.596456] Code: 00 00 4c 89 f1 48 8b 3c 24 e8 67 4f 01 00 48 89 df e8 b2 70 f2 ff ba 01 00 00 00 4c 89 ee 4c 89 e7 e8 bf 29 01 00 e9 4b ff ff ff <0f> 0b 41 b8 50 00 00 00 4c 89 f1 4c 89 fa 48 89 ee 48 8b 3c 24 [356036.596478] RIP [<ffffffff811605fe>] btrfs_writepage_fixup_worker+0xde/0x121 [356036.596483] RSP <ffff8801e2379de0> [356036.653626] ---[ end trace 9fa19a7644192fb6 ]--- btrfsck now finds many of these: jupiter ~ # btrfsck /dev/sde1 root 256 inode 12746 errors 400 root 256 inode 12747 errors 400 root 256 inode 12748 errors 400 root 256 inode 12749 errors 400 root 256 inode 17141 errors 400 root 256 inode 219966 errors 400 root 256 inode 224243 errors 400 root 256 inode 225245 errors 400 root 256 inode 225354 errors 400 root 256 inode 290639 errors 2000 root 256 inode 291751 errors 2000 This disk is used for nothing else then the following cycle: 1. mount it (compress-force=gzip) 2. run rsync to backup my system (using cow-friendly rsync flags) 3. create a snapshot 4. unmount it So that error must have been introduced simply by running rsync. It has plenty of free space (about 800 GB). -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Interestingly, the filesystem was not unmountable - system hung. After
reisub and checking again with "btrfs scrub" no errors where reported
and it
just rsync''ed fine this time. This does not make sense to me.
In any case here''s my backup script although I see nothing special with
it:
#!/bin/bash
DATE=$(date +%Y%m%d-%H%M)
BASEDIR=/mnt/usb-backup
ionice -c3 -p$$
mount ${BASEDIR} && (
[ -d "${BASEDIR}/snapshots/system-${DATE}" ] || (
time rsync -avAXH --delete --inplace --no-whole-file --stats \
--exclude "/proc/" \
--exclude "/dev/" \
--exclude "/sys/" \
--exclude "/boot/" \
--exclude "/media/" \
--exclude "/mnt/" \
/ ${BASEDIR}/current/
btrfs subvolume snapshot \
"${BASEDIR}/current" \
"${BASEDIR}/snapshots/system-${DATE}"
btrfs filesystem sync "${BASEDIR}"
)
umount /mnt/usb-backup
)
Kai Krakow <hurikhan77+btrfs@gmail.com> schrieb:
> Just happened while writing a huge avi file to my usb3 backup disk:
>
> [356036.596292] ------------[ cut here ]------------
> [356036.596300] kernel BUG at fs/btrfs/inode.c:1588!
> [356036.596304] invalid opcode: 0000 [#1] SMP
> [356036.596307] CPU 2
> [356036.596309] Modules linked in: vmnet(O) vmblock(O) vsock(O) vmci(O)
> vmmon(O) af_packet snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss
> snd_mixer_oss nls_iso8859_15 nls_cp437 vfat fat btusb bluetooth zram(C)
> loop snd_usb_audio snd_hwdep snd_usbmidi_lib snd_rawmidi snd_seq_device
> gspca_sonixj gspca_main videodev v4l2_compat_ioctl32 pcspkr i2c_i801 evdev
> unix fuse xfs nfs nfs_acl auth_rpcgss lockd sunrpc reiserfs scsi_wait_scan
> hid_monterey hid_microsoft hid_logitech hid_ezkey hid_cypress hid_chicony
> hid_cherry hid_belkin hid_apple hid_a4tech usbhid usb_storage hid sr_mod
> cdrom sg pata_cmd64x [last unloaded: microcode]
> [356036.596346]
> [356036.596349] Pid: 28747, comm: btrfs-fixup-1 Tainted: G C O
> 3.2.1-gentoo-r2 #1 To Be Filled By O.E.M. To Be Filled By O.E.M./Z68 Pro3
> [356036.596355] RIP: 0010:[<ffffffff811605fe>]
[<ffffffff811605fe>]
> btrfs_writepage_fixup_worker+0xde/0x121
> [356036.596363] RSP: 0000:ffff8801e2379de0 EFLAGS: 00010246
> [356036.596366] RAX: 0000000000000000 RBX: ffffea00019b1a00 RCX:
> 0000000000000000
> [356036.596370] RDX: 0000000000000000 RSI: ffffffffffffffff RDI:
> ffff88008a1bbb40
> [356036.596373] RBP: 00000000033fd000 R08: ffff8801e2379d2c R09:
> 0000000180240024
> [356036.596377] R10: 0000000000000000 R11: bf800000bf800000 R12:
> ffff88008a1bbc10
> [356036.596380] R13: 0000000000000000 R14: ffff8801e2379df8 R15:
> 00000000033fdfff
> [356036.596384] FS: 0000000000000000(0000) GS:ffff88043fb00000(0000)
> knlGS:0000000000000000
> [356036.596387] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [356036.596391] CR2: 00007f5ef9660000 CR3: 00000003253f2000 CR4:
> 00000000000406e0
> [356036.596394] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [356036.596398] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [356036.596401] Process btrfs-fixup-1 (pid: 28747, threadinfo
> ffff8801e2378000, task ffff8802d2160650)
> [356036.596405] Stack:
> [356036.596407] ffff88008a1bbab0 ffff88026d847540 ffffffffffffffff
> ffff88003a7c1f50
> [356036.596412] 0000000000000000 ffff88019bf62d80 ffff88019bf62dd0
> ffff88019bf62d98
> [356036.596417] ffff88019bf62da8 ffff8802d2160650 ffff88019bf62d88
> ffffffff8117f23f
> [356036.596421] Call Trace:
> [356036.596426] [<ffffffff8117f23f>] ? worker_loop+0x170/0x485
> [356036.596431] [<ffffffff8117f0cf>] ?
btrfs_queue_worker+0x272/0x272
> [356036.596435] [<ffffffff8117f0cf>] ?
btrfs_queue_worker+0x272/0x272
> [356036.596439] [<ffffffff810489fb>] ? kthread+0x7a/0x82
> [356036.596445] [<ffffffff81444634>] ? kernel_thread_helper+0x4/0x10
> [356036.596449] [<ffffffff81048981>] ? kthread_worker_fn+0x135/0x135
> [356036.596453] [<ffffffff81444630>] ? gs_change+0xb/0xb
> [356036.596456] Code: 00 00 4c 89 f1 48 8b 3c 24 e8 67 4f 01 00 48 89 df
> [e8
> b2 70 f2 ff ba 01 00 00 00 4c 89 ee 4c 89 e7 e8 bf 29 01 00 e9 4b ff ff ff
> <0f> 0b 41 b8 50 00 00 00 4c 89 f1 4c 89 fa 48 89 ee 48 8b 3c 24
> [356036.596478] RIP [<ffffffff811605fe>]
> btrfs_writepage_fixup_worker+0xde/0x121
> [356036.596483] RSP <ffff8801e2379de0>
> [356036.653626] ---[ end trace 9fa19a7644192fb6 ]---
>
> btrfsck now finds many of these:
>
> jupiter ~ # btrfsck /dev/sde1
> root 256 inode 12746 errors 400
> root 256 inode 12747 errors 400
> root 256 inode 12748 errors 400
> root 256 inode 12749 errors 400
> root 256 inode 17141 errors 400
> root 256 inode 219966 errors 400
> root 256 inode 224243 errors 400
> root 256 inode 225245 errors 400
> root 256 inode 225354 errors 400
> root 256 inode 290639 errors 2000
> root 256 inode 291751 errors 2000
>
> This disk is used for nothing else then the following cycle:
>
> 1. mount it (compress-force=gzip)
> 2. run rsync to backup my system (using cow-friendly rsync flags)
> 3. create a snapshot
> 4. unmount it
>
> So that error must have been introduced simply by running rsync. It has
> plenty of free space (about 800 GB).
>
> --
> To unsubscribe from this list: send the line "unsubscribe
linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Kai Krakow <hurikhan77+btrfs@gmail.com> schrieb:> Interestingly, the filesystem was not unmountable - system hung. After > reisub and checking again with "btrfs scrub" no errors where reported and > it just rsync''ed fine this time. This does not make sense to me.btrfsck still shows a lot of errors, while scrubbing says everything is okay... *sigh Now, how should one fix it if there is still no repair utility? I''m pretty sure sooner or later I will run into a BUG_ON again... Thanks, Kai -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Kai Krakow posted on Thu, 02 Feb 2012 04:54:45 +0100 as excerpted:> Kai Krakow <hurikhan77+btrfs@gmail.com> schrieb: > >> Interestingly, the filesystem was not unmountable - system hung. After >> reisub and checking again with "btrfs scrub" no errors where reported >> and it just rsync''ed fine this time. This does not make sense to me. > > btrfsck still shows a lot of errors, while scrubbing says everything is > okay... *sigh > > Now, how should one fix it if there is still no repair utility? I''m > pretty sure sooner or later I will run into a BUG_ON again...I had hoped someone else better qualified would answer, and they may still do so, but in the meantime, a couple notes... 1) This is unfortunately quite handwavy as I don''t understand the details myself, but if I''m reading the list right, there''s a known "phantom ENOSPC bug" that some are hitting when trying to write a large file (gigs, think dvd image), or do an rsync of several gigs, or... It may be that you hit it, and on retry, enough of the file was already there that it didn''t trigger the second time around. I gather they''ve not traced what is apparently a timeout or race condition fully and are working on a temporary throttling-based workaround. That''s supposed to work for the time being, but it''s only a workaround, and the throttling controls themselves are apparently rather rough at this point, so part of the testing is to make it a bit easier for ordinary users without the esoteric knowledge of a btrfs dev to use. So there''s a short-to-medium-term workaround coming and a longer term fix, once they trace down the problem itself. Meanwhile, don''t be too worried about ENOSPC errors the occur under heavy write load and that go away on retry, as there''s apparently others having the same issue. 2) Just a couple days ago I read an article that claimed Oracle has a Feb 16 deadline for a working btrfsck as that''s the deadline for getting it in their next shipping Unbreakable Linux release. I won''t claim to know if the article is correct or not, but if so, a reasonably working btrfsck should be available within two weeks. =:^) Of course it may continue to improve after that... Meanwhile, there''s a tool already available that should allow retrieving the undamaged data off of unmountable filesystems, at least, and there''s another tool that allows rollback to an earlier root node if necessary, thus allowing recovery of most filesystems at the cost of losing the last few seconds of work. Given the experimental nature of btrfs and the known lack of a proper btrfsck at this point anyway, that''s... actually quite reasonable, and the reason I decided it was time to start checking out btrfs myself (I''m still researching but have been on the list about a week now and had read a couple weeks worth of posts before I responded to anything). Don''t ask me what the names of those tools are. I could certainly look them up, but so can you, now that you know they exist. =:^) (assuming you didn''t before, of course.) Hopefully that''s somewhat helpful in pointing you in the right direction, at least. =:^) -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Thank you, Duncon... Duncan <1i5t5.duncan@cox.net> schrieb:> Kai Krakow posted on Thu, 02 Feb 2012 04:54:45 +0100 as excerpted: > >> Kai Krakow <hurikhan77+btrfs@gmail.com> schrieb: >> >>> Interestingly, the filesystem was not unmountable - system hung. After >>> reisub and checking again with "btrfs scrub" no errors where reported >>> and it just rsync''ed fine this time. This does not make sense to me. >> >> btrfsck still shows a lot of errors, while scrubbing says everything is >> okay... *sigh >> >> Now, how should one fix it if there is still no repair utility? I''m >> pretty sure sooner or later I will run into a BUG_ON again... > > I had hoped someone else better qualified would answer, and they may > still do so, but in the meantime, a couple notes...Still I think you gained good insight by reading all those posts. I''m using btrfs for a few weeks now and it is pretty solid since 3.2. I''ve been reading the list a few weeks before starting btrfs but only looked at articles about corruption and data loss. I started using btrfs when rescue tools became available and the most annoying corruption bugs had been fixed. But I''ve been hit by corruption and freezes a few times so I decided to have that big usb3 disk which I rsync every now and then using snapshots for rollback.> 1) This is unfortunately quite handwavy as I don''t understand the details > myself, but if I''m reading the list right, there''s a known "phantom ENOSPC > bug" that some are hitting when trying to write a large file (gigs, think > dvd image), or do an rsync of several gigs, or... It may be that you hit > it, and on retry, enough of the file was already there that it didn''t > trigger the second time around.On my first thought this was my suspicion, too. But otoh there was no ENOSPC message, neither in dmesg nor in rsync. Rsync just froze, I was able to kill it and my script continued to create a snapshot afterwards and unmount. I tried to mount again after btrfsck, it worked fine, I unmounted, system hung. I rebooted, scrubbed my two-disk array, no problems, I mounted the backup disk again, rsync''ed it, went fine, unmounted. But btrfsck still shows the same errors for this disk. *sigh [...]> So there''s a short-to-medium-term workaround coming and a longer term > fix, once they trace down the problem itself. Meanwhile, don''t be too > worried about ENOSPC errors the occur under heavy write load and that go > away on retry, as there''s apparently others having the same issue.I reported it here in the hope that it helps tracking down the bug or find people with similar experiences. Since "only" my backup disk is affected it is not a big problem. I can create it from scratch but try not to do it until someone guides me in tracking it down a bit. I think btrfs should try to fix such corruptions online while using it. From what I''ve learned here this is the long-term target and a working btrfsck should just be a helper tool. And the reason for the long delayed btrfsck is that Chris wants to have proper online fixing in place first. At least I can tell this corruption was introduced by bad logic in the kernel, and not by some crash. The usb3 disk is solely mounted for the purpose of rsync''ing and unmounted all the other times.> 2) Just a couple days ago I read an article that claimed Oracle has a Feb > 16 deadline for a working btrfsck as that''s the deadline for getting it > in their next shipping Unbreakable Linux release. I won''t claim to know > if the article is correct or not, but if so, a reasonably working btrfsck > should be available within two weeks. =:^) Of course it may continue to > improve after that...Sounds good. I wonder if Chris could tell anything on that point. ;-)> Meanwhile, there''s a tool already available that should allow retrieving > the undamaged data off of unmountable filesystems, at least, and there''s > another tool that allows rollback to an earlier root node if necessary, > thus allowing recovery of most filesystems at the cost of losing the last > few seconds of work. Given the experimental nature of btrfs and the > known lack of a proper btrfsck at this point anyway, that''s... actually > quite reasonable, and the reason I decided it was time to start checking > out btrfs myself (I''m still researching but have been on the list about a > week now and had read a couple weeks worth of posts before I responded to > anything).The tools are btrfs-rescue and btrfs-repair from Josef''s btrfs-progs available from github.> Hopefully that''s somewhat helpful in pointing you in the right direction, > at least. =:^)Well, no real news to me. But it is good having another soul here interested in gaining some knowledge. I think btrfs will become a great filesystem. But if you could provide a link for the Feb 16 deadline I''d be eager to read the article. Regards, Kai -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Kai Krakow <hurikhan77+btrfs@gmail.com> schrieb:> Just happened while writing a huge avi file to my usb3 backup disk: > > [356036.596292] ------------[ cut here ]------------ > [356036.596300] kernel BUG at fs/btrfs/inode.c:1588![...]> > btrfsck now finds many of these: > > jupiter ~ # btrfsck /dev/sde1 > root 256 inode 12746 errors 400 > root 256 inode 12747 errors 400 > root 256 inode 12748 errors 400 > root 256 inode 12749 errors 400 > root 256 inode 17141 errors 400 > root 256 inode 219966 errors 400 > root 256 inode 224243 errors 400 > root 256 inode 225245 errors 400 > root 256 inode 225354 errors 400 > root 256 inode 290639 errors 2000 > root 256 inode 291751 errors 2000It looks like the latter isn''t a consequence of the former. I found kernel commit f70a9a6b9 which is supposed to do something about "inode errors 400": commit f70a9a6b94af86fca069a7552ab672c31b457786 Author: Miao Xie <miaox@cn.fujitsu.com> Date: Thu Jan 12 19:10:12 2012 -0500 Btrfs: fix btrfsck error 400 when truncating a compressed It''s actually the case for me that rsync writes to the device using mount options "compress-force=zlib" and that rsync probably truncates files sometimes when using the inplace option. So that occurence is explained. Can anyone tell how bad it is to have this error? May the fs explode at some point or is it just an error I could safely ignore for the moment? And what about the "inode errors 2000"? What''s the 2000 standing for? Thanks, Kai -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sat, Feb 4, 2012 at 5:40 AM, Kai Krakow <hurikhan77+btrfs@gmail.com> wrote:> Kai Krakow <hurikhan77+btrfs@gmail.com> schrieb: > >> Just happened while writing a huge avi file to my usb3 backup disk: >> >> [356036.596292] ------------[ cut here ]------------ >> [356036.596300] kernel BUG at fs/btrfs/inode.c:1588! > [...] >> >> btrfsck now finds many of these: >> >> jupiter ~ # btrfsck /dev/sde1 >> root 256 inode 12746 errors 400 >> root 256 inode 12747 errors 400 >> root 256 inode 12748 errors 400 >> root 256 inode 12749 errors 400 >> root 256 inode 17141 errors 400 >> root 256 inode 219966 errors 400 >> root 256 inode 224243 errors 400 >> root 256 inode 225245 errors 400 >> root 256 inode 225354 errors 400 >> root 256 inode 290639 errors 2000 >> root 256 inode 291751 errors 2000 > > It looks like the latter isn''t a consequence of the former. I found kernel > commit f70a9a6b9 which is supposed to do something about "inode errors 400": > > commit f70a9a6b94af86fca069a7552ab672c31b457786 > Author: Miao Xie <miaox@cn.fujitsu.com> > Date: Thu Jan 12 19:10:12 2012 -0500 > Btrfs: fix btrfsck error 400 when truncating a compressed > > It''s actually the case for me that rsync writes to the device using mount > options "compress-force=zlib" and that rsync probably truncates files > sometimes when using the inplace option. > > So that occurence is explained. Can anyone tell how bad it is to have this > error? May the fs explode at some point or is it just an error I could > safely ignore for the moment? > > And what about the "inode errors 2000"? What''s the 2000 standing for? >As I understand it, Miao Xie''s "Btrfs: fix btrfsck error 400 when truncating a compressed" patch was intended to address only one scenario that would lead to 400 errors. It was not intended to comprehensively address problems that generate inode 400 errors, nor will this patch fix the error once it encountered. As it stands right now, if you have errors reported by btrfsck, and you''ve exhausted the tools available for addressing errors (scrub and zero-ing out the log are the only two I know of), you only really have two options (until further tools such as btrfsck are released): (1) Run with the errors and cross your fingers. (2) Backup and restore onto a freshly formatted volume (there are some new tools available to extract data if you encounter errors). My personal preference is to backup and restore. With respect to the error codes, you have to look in the source for btrfsck.c. Inode errors are defined as I_ERR_<error>. 0x400 (1 << 8) errors are I_ERR_FILE_NBYTES_WRONG 0x2000 (1 << 13) errors are I_ERR_LINK_COUNT_WRONG The 400 inode errors have been common lately. I haven''t seen as many 2000 inode errors. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Kai Krakow posted on Fri, 03 Feb 2012 00:25:51 +0100 as excerpted:> Duncan <1i5t5.duncan@cox.net> schrieb: > >> I had hoped someone else better qualified would answer, and they may >> still do so, but in the meantime, a couple notes... > > Still I think you gained good insight by reading all those posts. I''m > using btrfs for a few weeks now and it is pretty solid since 3.2. I''ve > been reading the list a few weeks before starting btrfs but only looked > at articles about corruption and data loss. I started using btrfs when > rescue tools became available and the most annoying corruption bugs had > been fixed. > But I''ve been hit by corruption and freezes a few times so I decided to > have that big usb3 disk which I rsync every now and then using snapshots > for rollback.Agreed on the insight from reading, and good to read that it has been pretty solid for you since 3.2 What I seem to be seeing is that the normal single-disk/dual-metadata setup seems to be reasonable, a few weird reports here and there including the ENOSPC stuff, but nothing huge. But my primary interest is raid1 both data and metadata, more than two- copies (3-4), and > 2-copy just doesn''t appear to be available yet (but see the article linked below, which says 3.4 timeframe). Even the two- copy setup looks like it still has major problems, including a recent thread reporting an inability to allocate further chunks when in degraded mode, so as soon as currently in-use chunks get full (maybe a gig or so instead of 20 in that test), ENOSPC. So I think I''ll wait another kernel cycle or two... but now that I''m here, I''m going to continue tracking the list, so I''ll be ready to go when the time comes.>> 1) "phantom ENOSPC bug"> On my first thought this was my suspicion, too. But otoh there was no > ENOSPC message, neither in dmesg nor in rsync. Rsync just froze, I was > able to kill it and my script continued to create a snapshot afterwards > and unmount. I tried to mount again after btrfsck, it worked fine, I > unmounted, system hung. I rebooted, scrubbed my two-disk array, no > problems, I mounted the backup disk again, rsync''ed it, went fine, > unmounted. But btrfsck still shows the same errors for this disk. *sighGood point. It looks similar to the ENOSPC bug, but without the ENOSPC. But keep in mind that they''re apparently simply throttling as a near-term workaround and haven''t fully traced the bug, yet. Given the otherwise similar trigger and symptoms, your reported problem could thus be a variant of the same bug, that happened to freeze rsync instead of erroring out with ENOSPC. If so, when they do finally nail that one, it could well either nail yours or at least make it easier to trace, as well.> I think btrfs should try to fix such corruptions online while using it. > From what I''ve learned here this is the long-term target and a working > btrfsck should just be a helper tool. And the reason for the long > delayed btrfsck is that Chris wants to have proper online fixing in > place first.I had seen articles pointing out that the mount-time and online fixing tools were indeed taking up some of the slack, but this is the first time I''ve seen it claimed as a major strategy, vs. the problems they''ve been seeing simply being easy enough to fix online once they track them down sufficiently to fix them at all, online or off. However, it does make a lot of sense to do what you can online, and until the last couple weeks I could have easily missed that it was deliberate since I wasn''t following btrfs closely enough to be sure to catch it before that, so it well may /be/ a deliberate strategy. I believe you''re correct.> At least I can tell this corruption was introduced by bad logic in the > kernel, and not by some crash. The usb3 disk is solely mounted for the > purpose of rsync''ing and unmounted all the other times.That''s a good point, as well.>> 2) Just a couple days ago I read an article that claimed Oracle has a >> Feb 16 deadline for a working btrfsck as that''s the deadline for >> getting it in their next shipping Unbreakable Linux release. I won''t >> claim to know if the article is correct or not, but if so, a reasonably >> working btrfsck should be available within two weeks. =:^) Of course >> it may continue to improve after that... > > Sounds good. I wonder if Chris could tell anything on that point. ;-) > >> Meanwhile, there''s a tool already available that should allow >> retrieving the undamaged data off of unmountable filesystems, at least, >> and there''s another tool that allows rollback to an earlier root node >> if necessary> The tools are btrfs-rescue and btrfs-repair from Josef''s btrfs-progs > available from github.Thanks.> But if you could provide a link for the Feb 16 deadline I''d be eager to > read the article.It was a couple days before I could go looking, thus the delay in this post, and it might be Feb 14 not 16, but... The basis seems to be Chris Mason''s talk at SCALE 10x LA, so there should be independent coverage on various Linux new sites. Here''s the one I googled up first (using the Feb 16 date, that at least here appears to be Feb 14, which might explain why I had trouble googling it). Phoronix. http://www.phoronix.com/scan.php?page=news_item&px=MTA0ODU That might have been the one I read, originally. (I subscribe to lxer.com ''s feed, which covers Linux and Android stories from around the net, including phoronix, and would have clicked that if it had come up, but don''t remember for sure whether that was it or if there was another.) There''s a bit more tech detail, including the new tidbit about multiple- mirroring that I mentioned above, in a different article. http://www.phoronix.com/scan.php?page=news_item&px=MTA0Njk I had discovered much to my dismay that so-called btrfs-raid1 only does dual-copy, not full raid1 to an arbitrary number of copies. My current disks are old enough that I really don''t want to risk two-copy-only, especially since I''m currently running 4-spindle md/raid1 for most of my system so I already have the disks. I originally installed md/raid6 for most of my data, thus the quad-spindle, but after running it for awhile, decided raid-1 fit my needs better. If I''d have known about raid5/6 at purchase and setup time what I know now, I''d have probably gone 2-spindle raid1 then, with a third as a hot-spare, and saved myself the money on the 4th one. The two-way would have been fine for current btrfs, but given that I''m running 4-way raid1 now and the disks are about mid-life operating hours, according to SMART, I simply don''t want to risk switching to two-way-only mirroring, only to then have both mirrors of after all aging disks die at once. So I''ve been debating whether btrfs DUAL mode (dual metadata on the same device, single data) on 4-way md/raid1s would be better, or btrfs-raid1s (two-way-mirrored data and metadata both, since two-way is all that''s possible ATM) layered on pairs of 2-way md/raids would be better. The latter would play to btrfs'' ability to recover data from a different mirror when necessary. But I already run a dozen md/raids on partitions across the same four physical devices, and that would double it to two- dozen. At some point it''s no longer a workable solution... But if triple-way mirroring (and one assumes N-way mirroring can''t be far behind that if it''s not what was meant) will show up in 3.4 or 3.5, as that article suggests, and with the writing-fsck being out for awhile by then if it''s coming out later this month, that might well be my upgrade time. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Mitch Harder <mitch.harder@sabayonlinux.org> schrieb:> On Sat, Feb 4, 2012 at 5:40 AM, Kai Krakow <hurikhan77+btrfs@gmail.com> > wrote: >> It''s actually the case for me that rsync writes to the device using mount >> options "compress-force=zlib" and that rsync probably truncates files >> sometimes when using the inplace option. >> >> So that occurence is explained. Can anyone tell how bad it is to have >> this error? May the fs explode at some point or is it just an error I >> could safely ignore for the moment? >> >> And what about the "inode errors 2000"? What''s the 2000 standing for? >> > > As I understand it, Miao Xie''s "Btrfs: fix btrfsck error 400 when > truncating a compressed" patch was intended to address only one > scenario that would lead to 400 errors. It was not intended to > comprehensively address problems that generate inode 400 errors, nor > will this patch fix the error once it encountered.I already understood that this patch wont magically fix errors already on disk. But my hope is, in 3.3, it would not introduce such errors any longer. But from what you write it could still generate inode 400 errors? I conclude that there exist many points in the btrfs code which could generate inode 400 errors, and Miao Xie''s patch only fixes the one scenario about truncating compressed files? Well, that should at least fix the problems for me because I believe this is coming from using compress-force with rsync-inplace... I don''t think these errors are originating from somewhere else because this disk is only mounted for the few minutes I''m running the rsync to it, then unmounted. Any maybe 3-4 weeks ago there have been no errors in btrfsck...> As it stands right now, if you have errors reported by btrfsck, and > you''ve exhausted the tools available for addressing errors (scrub and > zero-ing out the log are the only two I know of),I think I will try btrfs-repair from Josef... That''s one other tool for addressing errors.> you only really have > two options (until further tools such as btrfsck are released): > > (1) Run with the errors and cross your fingers.I don''t like crossing fingers for computers... ;-)> (2) Backup and restore onto a freshly formatted volume (there are > some new tools available to extract data if you encounter errors).This is my backup disk so I could easily purge it and restart with a backup from scratch - it will just take some hours for the first sync.> My personal preference is to backup and restore.This is why I got this external backup disk. ;-)> With respect to the error codes, you have to look in the source for > btrfsck.c. Inode errors are defined as I_ERR_<error>. > > 0x400 (1 << 8) errors are I_ERR_FILE_NBYTES_WRONG > 0x2000 (1 << 13) errors are I_ERR_LINK_COUNT_WRONGThanks for the pointers. I probably could live with the wrong link count. Not sure what kind of problems and wrong nbytes value introduces but at least my latest snapshot seems to be okay. Well, and at least these kind of errors seem like a proper candidate for btrfs-repair from Josef''s btrs-progs tree.> The 400 inode errors have been common lately. I haven''t seen as many > 2000 inode errors.I report back if btrfs-repair could fix it for me... Thanks, Kai -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Kai Krakow posted on Sun, 05 Feb 2012 09:01:25 +0100 as excerpted:> This is why I got this external backup disk. ;-)You''ve tested that backup, I hope? If you''re using it as a system disk, be sure you can boot from it without the main disk running so the system can''t pull anything from it. I mention this because I have an external that''s not directly bootable. I had to create a USB stick to boot from, that can then load the system off the unbootable USB drive after the kernel is loaded. People could be caught out if they weren''t aware it wasn''t bootable... -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
2012/2/1 Kai Krakow <hurikhan77+btrfs@gmail.com>:> Just happened while writing a huge avi file to my usb3 backup disk:Same problem here, I try to give the filesystem history: a) three days ago I format a 219GB partition: 1) latest Linus'' git kernel tree; 2) two nested subvolumes; 3) options: defaults,noatime,nobarrier,ssd,noacl,compress,subvol=root,autodefrag b) I copy ~90GB of data; c) I mount same as above, without compress; d) I copy other data, to ~140GB; e) run balance, after a while I had to poweroff; f) two days ago, I boot and it finishes the balance; g) I put in cron a snapshot every hour; h) today (after a lot of clean resume/suspend in RAM) I run VirtualBox and start an Ubuntu 12.04 install in Guest; i) near the end of installation VirtualBox get stuck (but everything else works) and kernel complain: [16661.706465] ------------[ cut here ]------------ [16661.706514] kernel BUG at fs/btrfs/inode.c:1588! [16661.706556] invalid opcode: 0000 [#1] SMP [16661.706597] CPU 0 [16661.706615] Modules linked in: zram(C) xfs exportfs usbhid hid binfmt_misc pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) rfcomm bnep joydev snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep uvcvideo r852 sm_common nand videobuf2_core snd_pcm videodev nand_ids btusb v4l2_compat_ioctl32 bluetooth mtd videobuf2_vmalloc videobuf2_memops nand_bch bch option usb_wwan nand_ecc usbserial psmouse snd_timer iwl3945 snd_page_alloc iwlegacy raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov raid6_pq async_tx raid0 linear dm_mirror dm_region_hash dm_log usb_storage sdhci_pci sdhci i915 drm_kms_helper mmc_core drm intel_agp intel_gtt sky2 agpgart [16661.707016] [16661.707016] Pid: 710, comm: btrfs-fixup-1 Tainted: G C O 3.3.0-rc3g+ #13 SAMSUNG ELECTRONICS CO., LTD. SQ45S70S/SQ45S70S [16661.707016] RIP: 0010:[<ffffffff811aedc5>] [<ffffffff811aedc5>] btrfs_writepage_fixup_worker+0x145/0x150 [16661.707016] RSP: 0000:ffff8800b9bb3df0 EFLAGS: 00010246 [16661.707016] RAX: 0000000000000000 RBX: ffffea0002185900 RCX: 0000000002488000 [16661.707016] RDX: ffff8800897ae2a8 RSI: ffffffffffffffff RDI: ffff8800897ae428 [16661.707016] RBP: 0000000002488000 R08: 0000000000000008 R09: ffff8800b9bb3da8 [16661.707016] R10: 0000000000001000 R11: 0000000000000000 R12: ffff8800b75f5ff0 [16661.707016] R13: 0000000000000000 R14: 0000000002488fff R15: ffff8800b75f5e70 [16661.707016] FS: 0000000000000000(0000) GS:ffff8800bf400000(0000) knlGS:0000000000000000 [16661.707016] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [16661.707016] CR2: 00007fa504fd40e0 CR3: 00000000ba515000 CR4: 00000000000006f0 [16661.707016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16661.707016] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16661.707016] Process btrfs-fixup-1 (pid: 710, threadinfo ffff8800b9bb2000, task ffff8800373edcd0) [16661.707016] Stack: [16661.707016] ffffffff81046a30 ffff880085a0d960 00000000ffffffff ffff88008e3602a0 [16661.707016] ffff8800bf410b40 ffff8800b9ce3840 ffff880085a0d968 ffff8800b9ce3858 [16661.707016] ffff8800b9ce3890 ffff8800b9bb3e90 ffff8800b9ce3858 ffffffff811d6972 [16661.707016] Call Trace: [16661.707016] [<ffffffff81046a30>] ? run_timer_softirq+0x220/0x220 [16661.707016] [<ffffffff811d6972>] ? worker_loop+0xa2/0x500 [16661.707016] [<ffffffff811d68d0>] ? btrfs_queue_worker+0x340/0x340 [16661.707016] [<ffffffff81056966>] ? kthread+0x96/0xa0 [16661.707016] [<ffffffff8150b534>] ? kernel_thread_helper+0x4/0x10 [16661.707016] [<ffffffff810568d0>] ? kthread_freezable_should_stop+0x60/0x60 [16661.707016] [<ffffffff8150b530>] ? gs_change+0xb/0xb [16661.707016] Code: 41 5f c3 0f 1f 00 41 b8 50 00 00 00 48 8d 4c 24 18 4c 89 f2 48 89 ee 4c 89 ff e8 d7 a4 01 00 eb b8 48 89 df e8 3d 01 ef ff eb 9c <0f> 0b 66 0f 1f 84 00 00 00 00 00 41 55 4c 8d 6e 40 41 54 55 48 [16661.707016] RIP [<ffffffff811aedc5>] btrfs_writepage_fixup_worker+0x145/0x150 [16661.707016] RSP <ffff8800b9bb3df0> [16661.731968] ---[ end trace 9b36ae9483fc03e3 ]--- l) I can''t remove ~/VirtualBox VMs/Ubuntu (command "rm -rf" doesn''t return), but I can cleanly close others apps; m) booting from recovery partition I can mount BTRFS and rm the directory above; n) run btrfsck fs tree 454 refs 12 unresolved ref root 454 dir 844801 index 5 namelen 9 name snap-0212 error 600 unresolved ref root 455 dir 844801 index 5 namelen 9 name snap-0212 error 600 unresolved ref root 458 dir 844801 index 5 namelen 9 name snap-0212 error 600 unresolved ref root 459 dir 844801 index 5 namelen 9 name snap-0212 error 600 unresolved ref root 466 dir 844801 index 5 namelen 9 name snap-0212 error 600 unresolved ref root 498 dir 844801 index 5 namelen 9 name snap-0212 error 600 unresolved ref root 500 dir 844801 index 5 namelen 9 name snap-0212 error 600 unresolved ref root 501 dir 844801 index 5 namelen 9 name snap-0212 error 600 unresolved ref root 503 dir 844801 index 5 namelen 9 name snap-0212 error 600 unresolved ref root 504 dir 844801 index 5 namelen 9 name snap-0212 error 600 unresolved ref root 507 dir 844801 index 5 namelen 9 name snap-0212 error 600 fs tree 455 refs 11 unresolved ref root 455 dir 844801 index 6 namelen 24 name snap-2012-02-12.15:23:24 error 600 unresolved ref root 458 dir 844801 index 6 namelen 24 name snap-2012-02-12.15:23:24 error 600 unresolved ref root 459 dir 844801 index 6 namelen 24 name snap-2012-02-12.15:23:24 error 600 unresolved ref root 466 dir 844801 index 6 namelen 24 name snap-2012-02-12.15:23:24 error 600 unresolved ref root 498 dir 844801 index 6 namelen 24 name snap-2012-02-12.15:23:24 error 600 unresolved ref root 500 dir 844801 index 6 namelen 24 name snap-2012-02-12.15:23:24 error 600 unresolved ref root 501 dir 844801 index 6 namelen 24 name snap-2012-02-12.15:23:24 error 600 unresolved ref root 503 dir 844801 index 6 namelen 24 name snap-2012-02-12.15:23:24 error 600 unresolved ref root 504 dir 844801 index 6 namelen 24 name snap-2012-02-12.15:23:24 error 600 unresolved ref root 507 dir 844801 index 6 namelen 24 name snap-2012-02-12.15:23:24 error 600 fs tree 458 refs 10 unresolved ref root 458 dir 844801 index 7 namelen 24 name snap-2012-02-13.00:44:53 error 600 unresolved ref root 459 dir 844801 index 7 namelen 24 name snap-2012-02-13.00:44:53 error 600 unresolved ref root 466 dir 844801 index 7 namelen 24 name snap-2012-02-13.00:44:53 error 600 unresolved ref root 498 dir 844801 index 7 namelen 24 name snap-2012-02-13.00:44:53 error 600 unresolved ref root 500 dir 844801 index 7 namelen 24 name snap-2012-02-13.00:44:53 error 600 unresolved ref root 501 dir 844801 index 7 namelen 24 name snap-2012-02-13.00:44:53 error 600 unresolved ref root 503 dir 844801 index 7 namelen 24 name snap-2012-02-13.00:44:53 error 600 unresolved ref root 504 dir 844801 index 7 namelen 24 name snap-2012-02-13.00:44:53 error 600 unresolved ref root 507 dir 844801 index 7 namelen 24 name snap-2012-02-13.00:44:53 error 600 fs tree 459 refs 9 unresolved ref root 459 dir 844801 index 8 namelen 24 name snap-2012-02-13.01:00:01 error 600 unresolved ref root 466 dir 844801 index 8 namelen 24 name snap-2012-02-13.01:00:01 error 600 unresolved ref root 498 dir 844801 index 8 namelen 24 name snap-2012-02-13.01:00:01 error 600 unresolved ref root 500 dir 844801 index 8 namelen 24 name snap-2012-02-13.01:00:01 error 600 unresolved ref root 501 dir 844801 index 8 namelen 24 name snap-2012-02-13.01:00:01 error 600 unresolved ref root 503 dir 844801 index 8 namelen 24 name snap-2012-02-13.01:00:01 error 600 unresolved ref root 504 dir 844801 index 8 namelen 24 name snap-2012-02-13.01:00:01 error 600 unresolved ref root 507 dir 844801 index 8 namelen 24 name snap-2012-02-13.01:00:01 error 600 fs tree 466 refs 8 unresolved ref root 466 dir 844801 index 9 namelen 24 name snap-2012-02-13.11:00:01 error 600 unresolved ref root 498 dir 844801 index 9 namelen 24 name snap-2012-02-13.11:00:01 error 600 unresolved ref root 500 dir 844801 index 9 namelen 24 name snap-2012-02-13.11:00:01 error 600 unresolved ref root 501 dir 844801 index 9 namelen 24 name snap-2012-02-13.11:00:01 error 600 unresolved ref root 503 dir 844801 index 9 namelen 24 name snap-2012-02-13.11:00:01 error 600 unresolved ref root 504 dir 844801 index 9 namelen 24 name snap-2012-02-13.11:00:01 error 600 unresolved ref root 507 dir 844801 index 9 namelen 24 name snap-2012-02-13.11:00:01 error 600 fs tree 498 refs 7 unresolved ref root 498 dir 844801 index 10 namelen 24 name snap-2012-02-13.12:00:01 error 600 unresolved ref root 500 dir 844801 index 10 namelen 24 name snap-2012-02-13.12:00:01 error 600 unresolved ref root 501 dir 844801 index 10 namelen 24 name snap-2012-02-13.12:00:01 error 600 unresolved ref root 503 dir 844801 index 10 namelen 24 name snap-2012-02-13.12:00:01 error 600 unresolved ref root 504 dir 844801 index 10 namelen 24 name snap-2012-02-13.12:00:01 error 600 unresolved ref root 507 dir 844801 index 10 namelen 24 name snap-2012-02-13.12:00:01 error 600 fs tree 500 refs 6 unresolved ref root 500 dir 844801 index 11 namelen 24 name snap-2012-02-13.14:00:01 error 600 unresolved ref root 501 dir 844801 index 11 namelen 24 name snap-2012-02-13.14:00:01 error 600 unresolved ref root 503 dir 844801 index 11 namelen 24 name snap-2012-02-13.14:00:01 error 600 unresolved ref root 504 dir 844801 index 11 namelen 24 name snap-2012-02-13.14:00:01 error 600 unresolved ref root 507 dir 844801 index 11 namelen 24 name snap-2012-02-13.14:00:01 error 600 fs tree 501 refs 5 unresolved ref root 501 dir 844801 index 12 namelen 24 name snap-2012-02-13.18:00:01 error 600 unresolved ref root 503 dir 844801 index 12 namelen 24 name snap-2012-02-13.18:00:01 error 600 unresolved ref root 504 dir 844801 index 12 namelen 24 name snap-2012-02-13.18:00:01 error 600 unresolved ref root 507 dir 844801 index 12 namelen 24 name snap-2012-02-13.18:00:01 error 600 fs tree 503 refs 4 unresolved ref root 503 dir 844801 index 13 namelen 24 name snap-2012-02-13.19:00:01 error 600 unresolved ref root 504 dir 844801 index 13 namelen 24 name snap-2012-02-13.19:00:01 error 600 unresolved ref root 507 dir 844801 index 13 namelen 24 name snap-2012-02-13.19:00:01 error 600 fs tree 504 refs 3 unresolved ref root 504 dir 844801 index 14 namelen 24 name snap-2012-02-13.20:00:01 error 600 unresolved ref root 507 dir 844801 index 14 namelen 24 name snap-2012-02-13.20:00:01 error 600 fs tree 507 refs 2 unresolved ref root 507 dir 844801 index 15 namelen 24 name snap-2012-02-13.21:00:01 error 600 found 148321431552 bytes used err is 1 total csum bytes: 143183508 total tree bytes: 1695162368 total fs tree bytes: 1380012032 btree space waste bytes: 395268619 file data blocks allocated: 181930553344 referenced 190255210496 o) remove all snapshots, so btrsck stop complain: found 144518303744 bytes used err is 0 total csum bytes: 139577232 total tree bytes: 1588137984 total fs tree bytes: 1281392640 btree space waste bytes: 370101450 file data blocks allocated: 143005290496 referenced 152667561984 p) scrub is still running, but everything seems fine. Thanks a lot for your time, Andrea -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html