I ahve been using btrfs as my primary root and home partitions for a few mo enths now, and so far that is going well. I run Arch Linux with the latest Linus git kernels and use the latest btrfs-progs-unstable git trees to manage my btrfs filesystems. My filesystem latout consists of a single 500GB SATA drive with an ext2 /boot partition and a dm-crypt''ed LVM volume group taking up the rest of the space. Some earlier playing around with the multivolume capabilities of btrfs convinced me that LVM is still needed for serious work - my regular hard drive upgrade habits involve pvmoving everything to a larger drive when I feel like I am running out of space on whatever drive I happen to be using at the time. I decided to see if btrfs could take over that capability right now, and my tests were negative Here is a transcript of my testing: [root@studio-arch ~]# uname -a Linux studio-arch 2.6.34-rc6 #13 SMP PREEMPT Tue May 4 16:38:20 CDT 2010 x86_64 Intel(R) Core(TM)2 Duo CPU P8600 @ 2.40GHz GenuineIntel GNU/Linux [root@studio-arch ~]# lvm lvm> lvcreate -L 10G -n test1 gentoo Logical volume "test1" created lvm> lvcreate -L 10G -n test2 gentoo Logical volume "test2" created lvm> quit Exiting. [root@studio-arch ~]# mkfs.btrfs /dev/gentoo/test1 WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL WARNING! - see http://btrfs.wiki.kernel.org before using fs created label (null) on /dev/gentoo/test1 nodesize 4096 leafsize 4096 sectorsize 4096 size 10.00GB Btrfs Btrfs v0.19 [root@studio-arch ~]# mount /dev/gentoo/test1 /mnt [root@studio-arch ~]# cp -a /boot/* /mnt/ [root@studio-arch ~]# ls /mnt grub initramfs-linux-2.6.34-rc6.img vmlinuz-2.6.34-rc3 initramfs-2.6.34-rc3.img initramfs-newroot.img vmlinuz-2.6.34-rc4 initramfs-2.6.34-rc4.img lost+found vmlinuz-2.6.34-rc6 initramfs-2.6.34-rc6.img memtest86+.bin [root@studio-arch ~]# btrfs fi show Label: none uuid: edf442ee-729c-4868-acc9-3b45e2621262 Total devices 1 FS bytes used 28.00KB devid 1 size 10.00GB used 2.04GB path /dev/dm-6 Btrfs Btrfs v0.19 [root@studio-arch ~]# btrfs fi df /mnt Metadata, DUP: total=1.00GB, used=540.00KB System, DUP: total=8.00MB, used=4.00KB Data: total=1.01GB, used=26.93MB Metadata: total=8.00MB, used=0.00 System: total=4.00MB, used=0.00 [root@studio-arch ~]# btrfs fi show Label: none uuid: edf442ee-729c-4868-acc9-3b45e2621262 Total devices 2 FS bytes used 27.46MB devid 1 size 10.00GB used 3.04GB path /dev/dm-6 devid 2 size 10.00GB used 0.00 path /dev/dm-7 Btrfs Btrfs v0.19 [root@studio-arch ~]# btrfs fi bal /mnt [root@studio-arch ~]# dmesg <...> [ 6841.202960] btrfs: relocating block group 1103101952 flags 1 [ 6841.754905] btrfs: found 8 extents [ 6842.373712] btrfs: found 8 extents [ 6842.506431] btrfs: relocating block group 29360128 flags 36 [ 6842.893517] btrfs: found 18 extents [ 6843.075464] btrfs: relocating block group 20971520 flags 34 [ 6843.342993] btrfs: found 1 extents [ 6843.529189] btrfs: relocating block group 12582912 flags 1 [ 6843.875434] btrfs: found 106 extents [ 6844.874100] btrfs: found 106 extents [ 6845.057757] btrfs: relocating block group 4194304 flags 4 [root@studio-arch ~]# btrfs fi df /mnt System, RAID1: total=8.00MB, used=4.00KB Metadata, RAID1: total=128.00MB, used=540.00KB Data, RAID0: total=4.00GB, used=26.93MB Metadata, DUP: total=0.00, used=0.00 System, DUP: total=0.00, used=0.00 Data: total=0.00, used=0.00 Metadata: total=0.00, used=0.00 System: total=4.00MB, used=0.00 [root@studio-arch ~]# btrfs fi show failed to read /dev/sr0 Label: none uuid: 9413c8ab-ad91-4330-8875-1b607df815e8 Total devices 1 FS bytes used 4.20GB devid 1 size 20.00GB used 8.54GB path /dev/dm-3 Label: none uuid: 4a19cf55-c0ca-4941-9202-a02fb312da99 Total devices 1 FS bytes used 29.53GB devid 1 size 70.00GB used 70.00GB path /dev/dm-4 Label: none uuid: edf442ee-729c-4868-acc9-3b45e2621262 Total devices 2 FS bytes used 27.46MB devid 1 size 10.00GB used 2.14GB path /dev/dm-6 devid 2 size 10.00GB used 2.13GB path /dev/dm-7 Btrfs Btrfs v0.19 [root@studio-arch ~]# So far, so good -- 2 devices in the filesystem, metadata mirrored across them, data striped across them -- about what I would expect. Here is where I run into the main glitch: [root@studio-arch ~]# btrfs dev del /dev/gentoo/test1 /mnt ERROR: error removing the device ''/dev/gentoo/test1'' [root@studio-arch ~]# dmesg <...> [ 7181.202675] btrfs: unable to go below two devices on raid1 [root@studio-arch ~]# That error should not happen. The filesystem was not created as a RAID1 (it was a single disk only), and the data is not redundant across the devices. Maybe the btrfs command is not doing the Right Thing, so try the older btrfs-vol command: [root@studio-arch ~]# btrfs-vol -r /dev/gentoo/test1 /mnt ioctl returns -1 [root@studio-arch ~]# dmesg <...> [ 7471.892633] btrfs: unable to go below two devices on raid1 [root@studio-arch ~]# At this point, I am stuck. There does not appear to be a --yes-really-do-it switch for either btrfs or btrfs-vol that will proceed to rebalance all the data off test1 and remove it from the filesystem. The offending bit of kernel code in btrfs_rm_device seems to be all_avail = root->fs_info->avail_data_alloc_bits | root->fs_info->avail_system_alloc_bits | root->fs_info->avail_metadata_alloc_bits; and the checks that follow it. We should arguably only care about the redundancy imposed by the data allocation policy (not system or metadata) when doing device adds and removes, otherwise we are caught in a bit of a trap -- you can grow your filesystem from one device to many, but you can only shrink it back down to two. Trying the semi obvious fix of all_avail = root->fs_info->avail_data_alloc_bits; fails with the following oops: [ 242.083305] btrfs allocation failed flags 4, wanted 4096 [ 242.083335] BUG: unable to handle kernel NULL pointer dereference at 00000000000000b8 [ 242.083346] IP: [<ffffffff81060c79>] do_raw_spin_lock+0x9/0x1a [ 242.083364] PGD 10c9b8067 PUD 10c987067 PMD 0 [ 242.083375] Oops: 0002 [#1] PREEMPT SMP [ 242.083385] last sysfs file: /sys/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0A:00/power_supply/BAT0/energy_full [ 242.083393] CPU 1 [ 242.083397] Modules linked in: loop fuse ip6table_filter ip6_tables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT iptable_filter ip_tables bridge stp llc af_packet ext2 mbcache xfs exportfs uvcvideo videodev v4l1_compat v4l2_compat_ioctl32 dell_wmi wmi battery ac sdhci_pci sdhci iwlagn iwlcore mmc_core thermal mac80211 cfg80211 snd_hda_codec_intelhdmi snd_hda_codec_idt dell_laptop rfkill dcdbas evdev joydev snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc uhci_hcd tg3 libphy ehci_hcd usbcore ipv6 kvm_intel kvm rtc_cmos rtc_core rtc_lib [ 242.083530] [ 242.083538] Pid: 4119, comm: btrfs Not tainted 2.6.34-rc6 #14 0D176M/Studio 1555 [ 242.083545] RIP: 0010:[<ffffffff81060c79>] [<ffffffff81060c79>] do_raw_spin_lock+0x9/0x1a [ 242.083558] RSP: 0018:ffff88010ca13508 EFLAGS: 00010213 [ 242.083564] RAX: 0000000000000100 RBX: 00000000000000b8 RCX: 0000000000012e3f [ 242.083570] RDX: 0000000000000000 RSI: 0000000000001000 RDI: 00000000000000b8 [ 242.083577] RBP: ffff88010ca13508 R08: 0000000000000002 R09: 00000000000000d3 [ 242.083583] R10: ffffffff814f180a R11: 00000000ffffffff R12: 00000000000000b8 [ 242.083589] R13: 0000000000000001 R14: 0000000000001000 R15: 0000000000001000 [ 242.083597] FS: 00007f9b2f344740(0000) GS:ffff880001900000(0000) knlGS:0000000000000000 [ 242.083605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 242.083611] CR2: 00000000000000b8 CR3: 0000000115e5d000 CR4: 00000000000406e0 [ 242.083618] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 242.083624] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 242.083632] Process btrfs (pid: 4119, threadinfo ffff88010ca12000, task ffff880114df4d40) [ 242.083637] Stack: [ 242.083641] ffff88010ca13528 ffffffff8136a853 0000000000000004 0000000000000000 [ 242.083651] <0> ffff88010ca13588 ffffffff8111c43d 0000000000000018 ffff88010ca13598 [ 242.083663] <0> ffff88010ca13558 ffffffff8111e10d ffff88010ca13588 0000000000000000 [ 242.083677] Call Trace: [ 242.083688] [<ffffffff8136a853>] _raw_spin_lock+0x1e/0x23 [ 242.083700] [<ffffffff8111c43d>] dump_space_info+0x29/0x158 [ 242.083710] [<ffffffff8111e10d>] ? rcu_read_unlock+0x23/0x2e [ 242.083721] [<ffffffff8112120c>] btrfs_reserve_extent+0x141/0x155 [ 242.083731] [<ffffffff8112295c>] btrfs_alloc_free_block+0x59/0x192 [ 242.083742] [<ffffffff81147310>] ? read_extent_buffer+0xbe/0xde [ 242.083752] [<ffffffff81116b2e>] __btrfs_cow_block+0xfc/0x312 [ 242.083761] [<ffffffff81117a1e>] ? comp_keys+0x26/0x28 [ 242.083771] [<ffffffff81143c38>] ? pagefault_enable+0x23/0x2e [ 242.083782] [<ffffffff8111532f>] ? kmap_atomic+0x16/0x4b [ 242.083791] [<ffffffff81116e40>] btrfs_cow_block+0xfc/0x10b [ 242.083800] [<ffffffff81119b98>] btrfs_search_slot+0x137/0x4e4 [ 242.083811] [<ffffffff8111aace>] btrfs_insert_empty_items+0x5b/0xa1 [ 242.083819] [<ffffffff811162ef>] ? btrfs_alloc_path+0x15/0x26 [ 242.083828] [<ffffffff8111ab66>] btrfs_insert_item+0x52/0x9d [ 242.083839] [<ffffffff811239b2>] btrfs_make_block_group+0x223/0x254 [ 242.083849] [<ffffffff8114a72f>] __btrfs_alloc_chunk+0x54e/0x5d0 [ 242.083859] [<ffffffff81143c38>] ? pagefault_enable+0x23/0x2e [ 242.083870] [<ffffffff8114b074>] btrfs_alloc_chunk+0x4d/0x7d [ 242.083879] [<ffffffff8111e2fb>] ? update_space_info+0xe3/0x16f [ 242.083889] [<ffffffff8111f3d1>] do_chunk_alloc+0x174/0x1d3 [ 242.083900] [<ffffffff81123116>] btrfs_prepare_block_group_relocation+0xe5/0x113 [ 242.083912] [<ffffffff8115f2d4>] btrfs_relocate_block_group+0xb2/0x2cc [ 242.083922] [<ffffffff81148564>] btrfs_relocate_chunk+0x65/0x4a2 [ 242.083930] [<ffffffff8114746c>] ? map_extent_buffer+0x67/0xa1 [ 242.083940] [<ffffffff81143c38>] ? pagefault_enable+0x23/0x2e [ 242.083949] [<ffffffff811474af>] ? unmap_extent_buffer+0x9/0xb [ 242.083958] [<ffffffff8113cd74>] ? btrfs_dev_extent_chunk_offset+0xb6/0xc6 [ 242.083969] [<ffffffff8114ad14>] btrfs_shrink_device+0x1ce/0x2f1 [ 242.083980] [<ffffffff8114c7f9>] btrfs_rm_device+0x210/0x4aa [ 242.083989] [<ffffffff81150b88>] ? btrfs_ioctl+0x60c/0x7d0 [ 242.084000] [<ffffffff8109db7a>] ? copy_from_user+0x9/0xb [ 242.084010] [<ffffffff81150baa>] btrfs_ioctl+0x62e/0x7d0 [ 242.084022] [<ffffffff810ca834>] vfs_ioctl+0x31/0xa2 [ 242.084032] [<ffffffff810cb150>] do_vfs_ioctl+0x457/0x48a [ 242.084042] [<ffffffff810cb1d4>] sys_ioctl+0x51/0x75 [ 242.084053] [<ffffffff8100212b>] system_call_fastpath+0x16/0x1b [ 242.084059] Code: 00 00 01 74 05 e8 28 e9 13 00 c9 c3 55 48 89 e5 f0 ff 07 c9 c3 55 48 89 e5 f0 81 07 00 00 00 01 c9 c3 55 b8 00 01 00 00 48 89 e5 <f0> 66 0f c1 07 38 e0 74 06 f3 90 8a 07 eb f6 c9 c3 55 48 89 e5 [ 242.084155] RIP [<ffffffff81060c79>] do_raw_spin_lock+0x9/0x1a [ 242.084165] RSP <ffff88010ca13508> [ 242.084170] CR2: 00000000000000b8 [ 242.084177] ---[ end trace 368ef32f0f0e8e74 ]--- [ 242.084184] note: btrfs[4119] exited with preempt_count 1 [root@studio-arch ~]# What to do to fix? -- Victor Lowther LPIC2 UCP RHCE -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html