How do I replace failed disks in RAID-1 mode? I followed the http://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices, but it triggered a kernel BUG (using 2.6.30.1 kernel): # mount -o degraded /dev/sdb4 /mnt/btrfs/ # btrfs-vol -r missing /mnt/btrfs/ removing missing devices from /mnt/btrfs/ ioctl returns -1 # dmesg -c btrfs: unable to go below two devices on raid1 # btrfs-vol -a /dev/sda4 /mnt/btrfs/ Speicherzugriffsfehler # dmesg btrfs allocation failed flags 18, wanted 4096 space_info has 8384512 free, is not full space_info total=8388608, pinned=0, delalloc=0, may_use=0, used=4096 block group 20971520 has 8388608 bytes, 4096 used 0 pinned 0 reserved entry offset 20975616, bytes 8384512 1 blocks of free space at or bigger than bytes is ------------[ cut here ]------------ kernel BUG at fs/btrfs/extent-tree.c:2930! invalid opcode: 0000 [#1] SMP last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map CPU 1 Modules linked in: btrfs zlib_deflate crc32c libcrc32c cpufreq_stats tun af_packet fglrx(P) ipv6 coretemp binfmt_misc loop ext3 jbd joydev dm_mod hid_multilaser usbhid cpufreq_ondemand cpufreq_conservative cpufreq_powersave acpi_cpufreq freq_table hid kvm_intel kvm snd_hda_codec_atihdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_seq_dummy snd_hwdep snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_pcm snd_timer snd_mixer_oss snd pcspkr soundcore e1000e iTCO_wdt iTCO_vendor_support serio_raw snd_page_alloc i2c_i801 rtc_cmos i2c_core evdev thermal processor button wmi ehci_hcd uhci_hcd sr_mod sg usbcore ata_piix ahci libata sd_mod scsi_mod crc_t10dif raid1 ext4 jbd2 crc16 Pid: 23996, comm: btrfs-vol Tainted: P 2.6.30.1-desktop-1mnb #1 RIP: 0010:[<ffffffffa06a5426>] [<ffffffffa06a5426>] __btrfs_reserve_extent+0x296/0x310 [btrfs] RSP: 0018:ffff8800bb441808 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88023cd9da2c RCX: 000000000001ffff RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000246 RBP: ffff8800bb4418b8 R08: 000000000000ddd9 R09: 000000000000000a R10: 0000000000000000 R11: 0000000000000000 R12: ffff88023d644c68 R13: 0000000000001000 R14: ffff88023d644d30 R15: ffff88023d644d18 FS: 00007ffc25a67740(0000) GS:ffff88002f21b000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f0645383ea4 CR3: 00000000be85c000 CR4: 00000000000026e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process btrfs-vol (pid: 23996, threadinfo ffff8800bb440000, task ffff88013a1a0000) Stack: 0000000000000000 ffff8800bb441988 0000000000000000 0000000000000000 0000000000000012 ffff88023fc016d0 ffff8800bb4418a8 ffff880149532000 0000000000000000 ffff8800bb441988 ffffffffffffffff 0000000000000000 Call Trace: [<ffffffffa06a550e>] btrfs_alloc_extent+0x6e/0x100 [btrfs] [<ffffffffa06a5619>] btrfs_alloc_free_block+0x79/0xd0 [btrfs] [<ffffffffa0693779>] __btrfs_cow_block+0x199/0x880 [btrfs] [<ffffffffa0693f7d>] btrfs_cow_block+0x11d/0x250 [btrfs] [<ffffffffa069a9b6>] btrfs_search_slot+0x476/0x8c0 [btrfs] [<ffffffffa069b5bd>] btrfs_insert_empty_items+0x8d/0x100 [btrfs] [<ffffffffa0691718>] ? btrfs_alloc_path+0x28/0x50 [btrfs] [<ffffffffa06d8d29>] btrfs_add_device+0x89/0x210 [btrfs] [<ffffffff80346db4>] ? kill_bdev+0x44/0x70 [<ffffffffa06ddc4a>] btrfs_init_new_device+0x65a/0xb20 [btrfs] [<ffffffffa06e0b8d>] ? btrfs_ioctl+0x3cd/0x910 [btrfs] [<ffffffffa06e0baf>] btrfs_ioctl+0x3ef/0x910 [btrfs] [<ffffffff80325680>] vfs_ioctl+0x30/0xd0 [<ffffffff8034de53>] ? inotify_inode_queue_event+0xf3/0x130 [<ffffffff80213ece>] ? apic_timer_interrupt+0xe/0x20 [<ffffffff80325e5b>] do_vfs_ioctl+0x31b/0x5b0 [<ffffffff8027cd6c>] ? up_read+0x1c/0x40 [<ffffffff80326189>] sys_ioctl+0x99/0xc0 [<ffffffff80213342>] system_call_fastpath+0x16/0x1b Code: 4c 8b 63 58 49 81 ec b0 00 00 00 49 8b 84 24 b0 00 00 00 0f 18 08 49 8d 84 24 b0 00 00 00 49 39 c7 75 27 4c 89 f7 e8 2a 79 bd df <0f> 0b eb fe be 5c 0b 00 00 48 c7 c7 9b d2 6e a0 e8 05 5d bb df RIP [<ffffffffa06a5426>] __btrfs_reserve_extent+0x296/0x310 [btrfs] RSP <ffff8800bb441808> ---[ end trace a51370eaca1eb6d6 ]--- -- Tomasz Chmielewski http://wpkg.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, 2009-07-13 at 12:28 +0200, Tomasz Chmielewski wrote:> How do I replace failed disks in RAID-1 mode?I don''t think you can. In theory you can remove the broken one, and you can add a _new_ empty one -- I say ''in theory'' because you seem to have demonstrated both of those actions failing. But I don''t believe we have yet implemented anything to let you _replace_ a failed disk and recreate its original contents. I had that on my TODO list for some time after I get the basic RAID[56] operation working. -- David Woodhouse Open Source Technology Centre David.Woodhouse@intel.com Intel Corporation -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Tomasz Chmielewski
2009-Jul-13 13:35 UTC
Re: replacing failed disks in RAID-1 (kernel BUG)?
David Woodhouse wrote:> On Mon, 2009-07-13 at 12:28 +0200, Tomasz Chmielewski wrote: >> How do I replace failed disks in RAID-1 mode? > > I don''t think you can. In theory you can remove the broken one, and you > can add a _new_ empty one -- I say ''in theory'' because you seem to have > demonstrated both of those actions failing. > > But I don''t believe we have yet implemented anything to let you > _replace_ a failed disk and recreate its original contents. > > I had that on my TODO list for some time after I get the basic RAID[56] > operation working.It would be also interesting to have a tool to monitor the state of the RAID (i.e. similar to what /proc/mdstat provides for md). I also tried to compare what happens when we do writes to md-raid and to btrfs-raid (RAID-1 in both cases) and it looks... strange for btrfs. Or perhaps this is how RAID-1 works in btrfs? I used iostat to monitor the writes on both devices. With md RAID-1, when we do: # dd if=/dev/zero of=/mnt/md-raid-1/testfile and # iostat -dk 1 We can see the write speed on both devices is more or less the same. With btrfs RAID-1, when we do the same, I can see that writes go to one drive, while the second drive receives 0 kb/s writes; then it changes (one drive is written to, the second isn''t). Only sometimes, writes happen concurrently to both drives, like with md RAID-1. Is it intended? -- Tomasz Chmielewski http://wpkg.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
I have no idea if this would work with btrfs, but the first thing I would do is use ''dd'' to clone one drive over to the new drive. I''ve done that to repair real failed RAID1 arrays. -rb On Mon, Jul 13, 2009 at 3:28 AM, Tomasz Chmielewski<mangoo@wpkg.org> wrote:> How do I replace failed disks in RAID-1 mode? > > I followed the > http://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices, > but it triggered a kernel BUG (using 2.6.30.1 kernel): > > > # mount -o degraded /dev/sdb4 /mnt/btrfs/ > > # btrfs-vol -r missing /mnt/btrfs/ > removing missing devices from /mnt/btrfs/ > ioctl returns -1 > > # dmesg -c > btrfs: unable to go below two devices on raid1 > > # btrfs-vol -a /dev/sda4 /mnt/btrfs/ > Speicherzugriffsfehler > > # dmesg > btrfs allocation failed flags 18, wanted 4096 > space_info has 8384512 free, is not full > space_info total=8388608, pinned=0, delalloc=0, may_use=0, used=4096 > block group 20971520 has 8388608 bytes, 4096 used 0 pinned 0 reserved > entry offset 20975616, bytes 8384512 > 1 blocks of free space at or bigger than bytes is > ------------[ cut here ]------------ > kernel BUG at fs/btrfs/extent-tree.c:2930! > invalid opcode: 0000 [#1] SMP > last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map > CPU 1 > Modules linked in: btrfs zlib_deflate crc32c libcrc32c cpufreq_stats tun > af_packet fglrx(P) ipv6 coretemp binfmt_misc loop ext3 jbd joydev dm_mod > hid_multilaser usbhid cpufreq_ondemand cpufreq_conservative > cpufreq_powersave acpi_cpufreq freq_table hid kvm_intel kvm > snd_hda_codec_atihdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec > snd_seq_dummy snd_hwdep snd_seq_oss snd_seq_midi_event snd_seq > snd_seq_device snd_pcm_oss snd_pcm snd_timer snd_mixer_oss snd pcspkr > soundcore e1000e iTCO_wdt iTCO_vendor_support serio_raw snd_page_alloc > i2c_i801 rtc_cmos i2c_core evdev thermal processor button wmi ehci_hcd > uhci_hcd sr_mod sg usbcore ata_piix ahci libata sd_mod scsi_mod crc_t10dif > raid1 ext4 jbd2 crc16 > Pid: 23996, comm: btrfs-vol Tainted: P 2.6.30.1-desktop-1mnb #1 > RIP: 0010:[<ffffffffa06a5426>] [<ffffffffa06a5426>] > __btrfs_reserve_extent+0x296/0x310 [btrfs] > RSP: 0018:ffff8800bb441808 EFLAGS: 00010246 > RAX: 0000000000000000 RBX: ffff88023cd9da2c RCX: 000000000001ffff > RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000246 > RBP: ffff8800bb4418b8 R08: 000000000000ddd9 R09: 000000000000000a > R10: 0000000000000000 R11: 0000000000000000 R12: ffff88023d644c68 > R13: 0000000000001000 R14: ffff88023d644d30 R15: ffff88023d644d18 > FS: 00007ffc25a67740(0000) GS:ffff88002f21b000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > CR2: 00007f0645383ea4 CR3: 00000000be85c000 CR4: 00000000000026e0 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Process btrfs-vol (pid: 23996, threadinfo ffff8800bb440000, task > ffff88013a1a0000) > Stack: > 0000000000000000 ffff8800bb441988 0000000000000000 0000000000000000 > 0000000000000012 ffff88023fc016d0 ffff8800bb4418a8 ffff880149532000 > 0000000000000000 ffff8800bb441988 ffffffffffffffff 0000000000000000 > Call Trace: > [<ffffffffa06a550e>] btrfs_alloc_extent+0x6e/0x100 [btrfs] > [<ffffffffa06a5619>] btrfs_alloc_free_block+0x79/0xd0 [btrfs] > [<ffffffffa0693779>] __btrfs_cow_block+0x199/0x880 [btrfs] > [<ffffffffa0693f7d>] btrfs_cow_block+0x11d/0x250 [btrfs] > [<ffffffffa069a9b6>] btrfs_search_slot+0x476/0x8c0 [btrfs] > [<ffffffffa069b5bd>] btrfs_insert_empty_items+0x8d/0x100 [btrfs] > [<ffffffffa0691718>] ? btrfs_alloc_path+0x28/0x50 [btrfs] > [<ffffffffa06d8d29>] btrfs_add_device+0x89/0x210 [btrfs] > [<ffffffff80346db4>] ? kill_bdev+0x44/0x70 > [<ffffffffa06ddc4a>] btrfs_init_new_device+0x65a/0xb20 [btrfs] > [<ffffffffa06e0b8d>] ? btrfs_ioctl+0x3cd/0x910 [btrfs] > [<ffffffffa06e0baf>] btrfs_ioctl+0x3ef/0x910 [btrfs] > [<ffffffff80325680>] vfs_ioctl+0x30/0xd0 > [<ffffffff8034de53>] ? inotify_inode_queue_event+0xf3/0x130 > [<ffffffff80213ece>] ? apic_timer_interrupt+0xe/0x20 > [<ffffffff80325e5b>] do_vfs_ioctl+0x31b/0x5b0 > [<ffffffff8027cd6c>] ? up_read+0x1c/0x40 > [<ffffffff80326189>] sys_ioctl+0x99/0xc0 > [<ffffffff80213342>] system_call_fastpath+0x16/0x1b > Code: 4c 8b 63 58 49 81 ec b0 00 00 00 49 8b 84 24 b0 00 00 00 0f 18 08 49 > 8d 84 24 b0 00 00 00 49 39 c7 75 27 4c 89 f7 e8 2a 79 bd df <0f> 0b eb fe be > 5c 0b 00 00 48 c7 c7 9b d2 6e a0 e8 05 5d bb df > RIP [<ffffffffa06a5426>] __btrfs_reserve_extent+0x296/0x310 [btrfs] > RSP <ffff8800bb441808> > ---[ end trace a51370eaca1eb6d6 ]--- > > > -- > Tomasz Chmielewski > http://wpkg.org > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html >-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Tomasz Chmielewski
2009-Jul-14 08:36 UTC
Re: replacing failed disks in RAID-1 (kernel BUG)?
Roger Bumgarner wrote:> I have no idea if this would work with btrfs, but the first thing I > would do is use ''dd'' to clone one drive over to the new drive. I''ve > done that to repair real failed RAID1 arrays.I don''t think it''s a good idea. Well, it could be a temporary workaround (_if_ it works) for btrfs until it stabilizes. But certainly not for md RAID-1. _If_ it did work for you, you would have to take the whole array down as dd works, which is not good. md provides its own tools (mdadm) to replace/rebuild the RAID. -- Tomasz Chmielewski http://wpkg.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Jul 13, 2009 at 12:28:09PM +0200, Tomasz Chmielewski wrote:> How do I replace failed disks in RAID-1 mode? > > I followed the http://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices, but it triggered a kernel BUG (using 2.6.30.1 kernel): > > > # mount -o degraded /dev/sdb4 /mnt/btrfs/ > > # btrfs-vol -r missing /mnt/btrfs/ > removing missing devices from /mnt/btrfs/ > ioctl returns -1 > > # dmesg -c > btrfs: unable to go below two devices on raid1 > > # btrfs-vol -a /dev/sda4 /mnt/btrfs/ > SpeicherzugriffsfehlerSo, this should have worked. I''ll dig into why the btrfs-vol -a failed. -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Tomasz Chmielewski
2009-Jul-14 17:29 UTC
Re: replacing failed disks in RAID-1 (kernel BUG)?
Chris Mason wrote:> On Mon, Jul 13, 2009 at 12:28:09PM +0200, Tomasz Chmielewski wrote: >> How do I replace failed disks in RAID-1 mode? >> >> I followed the http://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices, but it triggered a kernel BUG (using 2.6.30.1 kernel): >> >> >> # mount -o degraded /dev/sdb4 /mnt/btrfs/ >> >> # btrfs-vol -r missing /mnt/btrfs/ >> removing missing devices from /mnt/btrfs/ >> ioctl returns -1 >> >> # dmesg -c >> btrfs: unable to go below two devices on raid1 >> >> # btrfs-vol -a /dev/sda4 /mnt/btrfs/ >> Speicherzugriffsfehler > > So, this should have worked. I''ll dig into why the btrfs-vol -a failed.Mind that I was using 2.6.30.1 kernel, so the code was not very fresh. -- Tomasz Chmielewski http://wpkg.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html