Chris Murphy
2014-Jul-23 01:34 UTC
BUGS: bogus out of space reported when mounted raid1 degraded, btrfs replace failure, then oops
3.16.0-0.rc6.git0.1.fc21.1.x86_64 btfs-progs 3.14.2 Fortunately this is a test system so it is dispensable. But in just an hour I ran into 5 bugs, and managed to apparently completely destroy a btrfs file system beyond repair, and it wasn't intentional. 1. mkfs.btrfs /dev/sda6 ## volume's life starts as single device, on an SSD 2. btrfs device add /dev/sdb1 / ## added an HDD partition 3. btrfs balance start -dconvert=raid1 -mconvert=raid1 4. clean shutdown, remove device 1 (leaving device 0) 5. poweron, mount degraded 6. gdm/gnome comes up very slowly, then I see a sad face graphic, with a message that there's only 60MB of space left. # df -h Filesystem Size Used Avail Use% Mounted on /dev/sda6 26G 13G 20M 100% / /dev/sda6 26G 13G 20M 100% /home /dev/sda6 26G 13G 20M 100% /var /dev/sda6 26G 13G 20M 100% /boot # btrfs fi df Data, RAID1: total=6.00GiB, used=5.99GiB System, RAID1: total=32.00MiB, used=32.00KiB Metadata, RAID1: total=768.00MiB, used=412.41MiB unknown, single: total=160.00MiB, used=0.00 # btrfs fi show Label: 'Rawhide2' uuid: f857c336-b8f5-4f5d-9500-a705ee1b6977 Total devices 2 FS bytes used 6.39GiB devid 1 size 12.58GiB used 6.78GiB path /dev/sda6 *** Some devices missing Btrfs v3.14.2 BUG 1: The df command is clearly bogus six ways to Sunday. It's a 12.58 GiB partition, only 6.78GiB used, thus 5.8GiB free, yet df and apparently gvfs think it's full, maybe systemd too because the journal wigged out and stopped logging events while also kept stopping and starting. So whatever changes occurred to clean up the df reporting, are very problematic at best when mounting degraded. ============so then he gets curious about replacing the missing disk============= 7. btrfs replace start 2 /dev/sdb1 / ## this is a ~13GB partition that matches the size of the missing device This completes, no disk activity for a little over a minute, and then I see a call trace with btrfs_replace implicated. Unfortunately the system becomes so unstable at this point, I can't even capture a dmesg to a separate volume. After 30 minutes of unresponsive local shells, I force a poweroff. 8. Power on. Dropped to a dracut shell, as the btrfs volume will not mount: [ 53.890761] rawhide kernel: BTRFS: failed to read the system array on sda6 [ 53.905058] rawhide kernel: BTRFS: open_ctree failed 9. mount with -o recovery, same message 10. Reboot using vbox pointed to these partitions as raw devices so I can better capture data, and not use a degraded fs as root; the devices are sdb and sdc. # mount -o ro /dev/sdb /mnt mount: wrong fs type, bad option, bad superblock on /dev/sdb, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so. [ 216.819927] BTRFS: failed to read the system array on sdc [ 216.835570] BTRFS: open_ctree failed So it's the same message as in dracut shell. Same message with ro,recovery. 11. mount -o degraded,ro /dev/sdb /mnt This works. Somehow the replace hasn't completed on some level. Very weird. And not intuitive. [root@localhost ~]# btrfs fi show Label: 'Rawhide2' uuid: f857c336-b8f5-4f5d-9500-a705ee1b6977 Total devices 2 FS bytes used 6.39GiB devid 0 size 12.58GiB used 6.78GiB path /dev/sdc devid 1 size 12.58GiB used 6.78GiB path /dev/sdb Btrfs v3.14.2 Does not show any missing devices. I vaguely recall in the dracut shell when booted baremetal that btrfs fi show did still show a missing devices along with the original and replacement devices, i.e. the replace didn't complete. I suspect that my 'btrfs replace start 2' is wrong, that devid 2 did not exist, it was actually devid 0 and 1 like above; but the problem is that btrfs fi show does not show devid for missing devices. I only saw the devid 1 for the remaining device, and assumed the missing one was 2. So that's why I did 'btrfs replace start 2' yet I didn't get an error message. The replace started, but apparently didn't complete. BUG 2: btrfs fi show needs to show the devid of the missing device. BUG 3: btrfs replace start should fail when specifying a non-existent devid. BUG 4: btrfs replace start can fail to complete (possibly related to bug 2 and 3). BUG 4: When mounting -degraded (rw), I get a major oops resulting in a completely unresponsive system. # mount -o degraded /dev/sdb /mnt [ 16.466995] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs [ 55.081687] BTRFS info (device sdb): allowing degraded mounts [ 55.082107] BTRFS info (device sdb): disk space caching is enabled [ 55.117702] SELinux: initialized (dev sdb, type btrfs), uses xattr [ 55.117717] BTRFS: continuing dev_replace from <missing disk> (devid 2) to /dev/sdc @72% [ 55.530810] BTRFS: dev_replace from <missing disk> (devid 2) to /dev/sdc) finished [ 55.532149] BUG: unable to handle kernel NULL pointer dereference at 0000000000000088 [ 55.533087] IP: [<ffffffffa0268551>] btrfs_kobj_rm_device+0x21/0x40 [btrfs] [ 55.533087] PGD 0 [ 55.533087] Oops: 0000 [#1] SMP [ 55.533087] Modules linked in: cfg80211 rfkill btrfs snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device ppdev xor raid6_pq snd_pcm microcode snd_timer serio_raw parport_pc snd i2c_piix4 parport soundcore i2c_core xfs libcrc32c virtio_net virtio_pci virtio_ring ata_generic virtio pata_acpi [ 55.533087] CPU: 2 PID: 821 Comm: btrfs-devrepl Not tainted 3.16.0-0.rc6.git0.1.fc21.1.x86_64 #1 [ 55.533087] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [ 55.533087] task: ffff880099b5eca0 ti: ffff88009983c000 task.ti: ffff88009983c000 [ 55.533087] RIP: 0010:[<ffffffffa0268551>] [<ffffffffa0268551>] btrfs_kobj_rm_device+0x21/0x40 [btrfs] [ 55.533087] RSP: 0018:ffff88009983fe08 EFLAGS: 00010286 [ 55.533087] RAX: 0000000000000000 RBX: 0000000000000000 RCX: bbb3527a6b299586 [ 55.533087] RDX: ffff880036b6e410 RSI: ffff88009b4a2800 RDI: ffff880035f6cac0 [ 55.533087] RBP: ffff88009983fe10 R08: ffff880036b6e410 R09: 0000000000000234 [ 55.533087] R10: ffffe8ffffd01090 R11: ffffffff818675c0 R12: ffff880099a2cdc8 [ 55.533087] R13: ffff88009b4a2800 R14: ffff880099eaa000 R15: ffff880036acf200 [ 55.533087] FS: 0000000000000000(0000) GS:ffff88009fb00000(0000) knlGS:0000000000000000 [ 55.533087] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 55.533087] CR2: 0000000000000088 CR3: 000000009aefe000 CR4: 00000000000006e0 [ 55.533087] Stack: [ 55.533087] ffff880099a2c000 ffff88009983fe90 ffffffffa02bf93d ffff880099a2c100 [ 55.533087] ffff880099a2ce38 00000006baa50000 ffffffff00000028 ffff88009983fea0 [ 55.533087] ffff88009983fe58 000000002909d417 ffff880099a2c000 000000002909d417 [ 55.533087] Call Trace: [ 55.533087] [<ffffffffa02bf93d>] btrfs_dev_replace_finishing+0x32d/0x5c0 [btrfs] [ 55.533087] [<ffffffffa02c0130>] ? btrfs_dev_replace_status+0x110/0x110 [btrfs] [ 55.533087] [<ffffffffa02c019d>] btrfs_dev_replace_kthread+0x6d/0x130 [btrfs] [ 55.533087] [<ffffffff810b311a>] kthread+0xea/0x100 [ 55.533087] [<ffffffff810b3030>] ? insert_kthread_work+0x40/0x40 [ 55.533087] [<ffffffff8172253c>] ret_from_fork+0x7c/0xb0 [ 55.533087] [<ffffffff810b3030>] ? insert_kthread_work+0x40/0x40 [ 55.533087] Code: 5f 5d c3 0f 1f 80 00 00 00 00 66 66 66 66 90 55 48 89 e5 53 48 8b bf f0 09 00 00 48 85 ff 74 20 31 db 48 85 f6 74 14 48 8b 46 78 <48> 8b 80 88 00 00 00 48 8b 70 38 e8 2f 23 01 e1 89 d8 5b 5d c3 [ 55.533087] RIP [<ffffffffa0268551>] btrfs_kobj_rm_device+0x21/0x40 [btrfs] [ 55.533087] RSP <ffff88009983fe08> [ 55.533087] CR2: 0000000000000088 [ 55.533087] ---[ end trace a34670f31a1db59e ]--- [root@localhost ~]# btrfs check /dev/sdb warning, device 2 is missing warning devid 2 not found already Checking filesystem on /dev/sdb UUID: f857c336-b8f5-4f5d-9500-a705ee1b6977 checking extents checking free space cache Error reading 22597402624, -1 failed to load free space cache for block group 21619867648 Error reading 25839001600, -1 failed to load free space cache for block group 22693609472 free space inode generation (0) did not match free space cache generation (858) Error reading 22597664768, -1 failed to load free space cache for block group 24841093120 Error reading 28045934592, -1 failed to load free space cache for block group 25914834944 Error reading 25849696256, -1 failed to load free space cache for block group 26988576768 Error reading 22595305472, -1 failed to load free space cache for block group 28095873024 Error reading 25688473600, -1 failed to load free space cache for block group 28364308480 checking fs roots checking csums checking root refs found 1449851186 bytes used err is 0 total csum bytes: 6233932 total tree bytes: 432472064 total fs tree bytes: 415531008 total extent tree bytes: 9240576 btree space waste bytes: 68632283 file data blocks allocated: 10542505984 referenced 8114642944 Btrfs v3.14.2 BUG 5: # btrfs-image -c9 -t3 /dev/sdb image.bin warning, device 2 is missing warning devid 2 not found already btrfs-image: disk-io.c:155: readahead_tree_block: Assertion `!(ret)' failed. Aborted (core dumped) Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html