Hi, Doing some extensive benchmarks on BTRFS, I encountered a kernel bug in BTRFS (as reported in dmesg) Maybe the information below can help you making btrfs better. Situation Doing an intensive sequential write on a SAS 3TB disk drive (SEAGATE ST33000652SS) with 128 threads with Sysbench. Device is connected through an HBA. Blocksize was 256k ; Kernel is 3.3.0 (x86_64) ; Btrfs is version v0.19 Write is done through an LVS volume formated with BTRFS of course; Mount options are : rw,noatime,nodiratime,compress=lzo,nospace_cache Dmesg gives me following error : [370517.203926] sd 0:0:15:0: [sdo] Device not ready [370517.203930] sd 0:0:15:0: [sdo] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [370517.203935] sd 0:0:15:0: [sdo] Sense Key : Not Ready [current] [370517.203940] sd 0:0:15:0: [sdo] ASC=0x4 <<vendor>> ASCQ=0xf2 [370517.203946] sd 0:0:15:0: [sdo] CDB: Write(10): 2a 00 00 00 6a 00 00 00 80 00 [370517.203955] end_request: I/O error, dev sdo, sector 27136 [370517.204236] sd 0:0:15:0: [sdo] Device not ready [370517.204240] sd 0:0:15:0: [sdo] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [370517.204244] sd 0:0:15:0: [sdo] Sense Key : Not Ready [current] [370517.204249] sd 0:0:15:0: [sdo] ASC=0x4 <<vendor>> ASCQ=0xf2 [370517.204255] sd 0:0:15:0: [sdo] CDB: Write(10): 2a 00 00 00 08 80 00 00 08 00 [370517.204265] end_request: I/O error, dev sdo, sector 2176 [370517.204283] lost page write due to I/O error on dm-5 [370517.204350] btrfs: 1 errors while writing supers [370517.204376] ------------[ cut here ]------------ [370517.204391] kernel BUG at fs/btrfs/disk-io.c:2880! [370517.204406] invalid opcode: 0000 [#1] SMP [370517.204423] CPU 2 [370517.204436] Pid: 8790, comm: sysbench Not tainted 3.3.0-oxeva #2 Supermicro X8DTT-H/X8DTT-H [370517.204482] RIP: 0010:[<ffffffff81389e63>] [<ffffffff81389e63>] write_all_supers+0x833/0x840 [370517.204508] RSP: 0018:ffff880e52f63b68 EFLAGS: 00010292 [370517.204522] RAX: 000000000000003b RBX: ffff88100ef81f38 RCX: 0000000000000068 [370517.204539] RDX: 0000000000000000 RSI: 0000000000000046 RDI: ffffffff81f9c6b0 [370517.204556] RBP: ffff880e52f63bd8 R08: 0000000000000000 R09: ffffffff81bba980 [370517.204573] R10: 0000000000000001 R11: 0000000000000000 R12: ffff880e52f63ba0 [370517.204592] R13: ffff88100ef81f38 R14: ffff881010902000 R15: 0000000000000001 [370517.204829] FS: 00007f3a2300c700(0000) GS:ffff88081fc40000(0000) knlGS:0000000000000000 [370517.204954] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [370517.205024] CR2: 00007f3a24b9e998 CR3: 0000000ce9963000 CR4: 00000000000006e0 [370517.205147] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [370517.205269] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [370517.205393] Process sysbench (pid: 8790, threadinfo ffff880e52f62000, task ffff881008902880) [370517.205519] Stack: [370517.205580] ffff88101090210b ffff88100ef81ec0 0000000052f63ba8 0000000100000001 [370517.205716] ffff881010d59000 ffff881000000001 ffff880811d87400 ffff88100ef81f38 [370517.205849] 00000013c4a03fff ffff880ff058d000 ffff880811d87400 ffff881002a915a0 [370517.205984] Call Trace: [370517.206049] [<ffffffff81389e7e>] write_ctree_super+0xe/0x10 [370517.206119] [<ffffffff813c24b4>] btrfs_sync_log+0x424/0x5c0 [370517.206190] [<ffffffff8139e9cb>] btrfs_sync_file+0x17b/0x1e0 [370517.206260] [<ffffffff81160703>] vfs_fsync_range+0x23/0x30 [370517.206329] [<ffffffff8116076c>] generic_write_sync+0x3c/0x40 [370517.206399] [<ffffffff8139f4a7>] btrfs_file_aio_write+0x317/0x530 [370517.206472] [<ffffffff8113556a>] do_sync_write+0xda/0x120 [370517.206544] [<ffffffff8110037f>] ? handle_mm_fault+0x1af/0x320 [370517.206614] [<ffffffff81135ab8>] vfs_write+0xc8/0x190 [370517.206682] [<ffffffff81135c12>] sys_pwrite64+0x92/0xa0 [370517.206753] [<ffffffff81aa82e2>] system_call_fastpath+0x16/0x1b [370517.206820] Code: e9 72 fe ff ff 44 89 d6 48 c7 c7 b8 68 d2 81 31 c0 e8 43 b1 71 00 0f 0b eb fe 8b 75 b8 48 c7 c7 b8 68 d2 81 31 c0 e8 2e b1 71 00 <0f> 0b eb fe 66 0f 1f 84 00 00 00 00 00 55 48 89 f7 48 89 e5 89 [370517.207167] RIP [<ffffffff81389e63>] write_all_supers+0x833/0x840 [370517.207239] RSP <ffff880e52f63b68> [370517.207673] ---[ end trace f208fa157676276c ]--- I don''t know if the device failed because BTRFS did something wrong, or if BTRFS failed because of the device. After this error, device cannot be accessed anymore (stuck and fdisk returned no info, neither do smartctl - just ''device busy''). By the way, I''m actually doing some extensive benchmarks, comparing BTRFS with XFS or EXT4. I''ll post results on this mailing list in a few days, I''m sure it can be interested for you. Olivier -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Olivier
> [370517.204350] btrfs: 1 errors while writing supers
> [370517.204376] ------------[ cut here ]------------
> [370517.204391] kernel BUG at fs/btrfs/disk-io.c:2880!
-----------
if (total_errors > max_errors) {
printk(KERN_ERR "btrfs: %d errors while writing supers\n",
total_errors);
/* This shouldn''t happen. FUA is masked off if unsupported
*/
BUG();
}
-----------
accessible disk(s) is(are) below the critical number of disk(s),
btrfs has limited choice. IMO instead of BUG, we could have a
user configurable choice to either panic or fail-pending-IO,
dump and force-unmount.
> or if BTRFS failed because of the device.
yes. Replacing the disk should help here.
HTH
regds, Anand
On 17/04/2012 17:23, Olivier Doucet wrote:> Hi,
>
> Doing some extensive benchmarks on BTRFS, I encountered a kernel bug
> in BTRFS (as reported in dmesg)
>
> Maybe the information below can help you making btrfs better.
>
> Situation
> Doing an intensive sequential write on a SAS 3TB disk drive (SEAGATE
> ST33000652SS) with 128 threads with Sysbench.
> Device is connected through an HBA. Blocksize was 256k ; Kernel is
> 3.3.0 (x86_64) ; Btrfs is version v0.19
>
> Write is done through an LVS volume formated with BTRFS of course;
> Mount options are : rw,noatime,nodiratime,compress=lzo,nospace_cache
>
> Dmesg gives me following error :
>
> [370517.203926] sd 0:0:15:0: [sdo] Device not ready
> [370517.203930] sd 0:0:15:0: [sdo] Result: hostbyte=DID_OK
> driverbyte=DRIVER_SENSE
> [370517.203935] sd 0:0:15:0: [sdo] Sense Key : Not Ready [current]
> [370517.203940] sd 0:0:15:0: [sdo] ASC=0x4<<vendor>>
ASCQ=0xf2
> [370517.203946] sd 0:0:15:0: [sdo] CDB: Write(10): 2a 00 00 00 6a 00 00 00
80 00
> [370517.203955] end_request: I/O error, dev sdo, sector 27136
> [370517.204236] sd 0:0:15:0: [sdo] Device not ready
> [370517.204240] sd 0:0:15:0: [sdo] Result: hostbyte=DID_OK
> driverbyte=DRIVER_SENSE
> [370517.204244] sd 0:0:15:0: [sdo] Sense Key : Not Ready [current]
> [370517.204249] sd 0:0:15:0: [sdo] ASC=0x4<<vendor>>
ASCQ=0xf2
> [370517.204255] sd 0:0:15:0: [sdo] CDB: Write(10): 2a 00 00 00 08 80 00 00
08 00
> [370517.204265] end_request: I/O error, dev sdo, sector 2176
> [370517.204283] lost page write due to I/O error on dm-5
> [370517.204350] btrfs: 1 errors while writing supers
> [370517.204376] ------------[ cut here ]------------
> [370517.204391] kernel BUG at fs/btrfs/disk-io.c:2880!
> [370517.204406] invalid opcode: 0000 [#1] SMP
> [370517.204423] CPU 2
> [370517.204436] Pid: 8790, comm: sysbench Not tainted 3.3.0-oxeva #2
> Supermicro X8DTT-H/X8DTT-H
> [370517.204482] RIP: 0010:[<ffffffff81389e63>]
[<ffffffff81389e63>]
> write_all_supers+0x833/0x840
> [370517.204508] RSP: 0018:ffff880e52f63b68 EFLAGS: 00010292
> [370517.204522] RAX: 000000000000003b RBX: ffff88100ef81f38 RCX:
> 0000000000000068
> [370517.204539] RDX: 0000000000000000 RSI: 0000000000000046 RDI:
> ffffffff81f9c6b0
> [370517.204556] RBP: ffff880e52f63bd8 R08: 0000000000000000 R09:
> ffffffff81bba980
> [370517.204573] R10: 0000000000000001 R11: 0000000000000000 R12:
> ffff880e52f63ba0
> [370517.204592] R13: ffff88100ef81f38 R14: ffff881010902000 R15:
> 0000000000000001
> [370517.204829] FS: 00007f3a2300c700(0000) GS:ffff88081fc40000(0000)
> knlGS:0000000000000000
> [370517.204954] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [370517.205024] CR2: 00007f3a24b9e998 CR3: 0000000ce9963000 CR4:
> 00000000000006e0
> [370517.205147] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [370517.205269] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [370517.205393] Process sysbench (pid: 8790, threadinfo
> ffff880e52f62000, task ffff881008902880)
> [370517.205519] Stack:
> [370517.205580] ffff88101090210b ffff88100ef81ec0 0000000052f63ba8
> 0000000100000001
> [370517.205716] ffff881010d59000 ffff881000000001 ffff880811d87400
> ffff88100ef81f38
> [370517.205849] 00000013c4a03fff ffff880ff058d000 ffff880811d87400
> ffff881002a915a0
> [370517.205984] Call Trace:
> [370517.206049] [<ffffffff81389e7e>] write_ctree_super+0xe/0x10
> [370517.206119] [<ffffffff813c24b4>] btrfs_sync_log+0x424/0x5c0
> [370517.206190] [<ffffffff8139e9cb>] btrfs_sync_file+0x17b/0x1e0
> [370517.206260] [<ffffffff81160703>] vfs_fsync_range+0x23/0x30
> [370517.206329] [<ffffffff8116076c>] generic_write_sync+0x3c/0x40
> [370517.206399] [<ffffffff8139f4a7>]
btrfs_file_aio_write+0x317/0x530
> [370517.206472] [<ffffffff8113556a>] do_sync_write+0xda/0x120
> [370517.206544] [<ffffffff8110037f>] ? handle_mm_fault+0x1af/0x320
> [370517.206614] [<ffffffff81135ab8>] vfs_write+0xc8/0x190
> [370517.206682] [<ffffffff81135c12>] sys_pwrite64+0x92/0xa0
> [370517.206753] [<ffffffff81aa82e2>] system_call_fastpath+0x16/0x1b
> [370517.206820] Code: e9 72 fe ff ff 44 89 d6 48 c7 c7 b8 68 d2 81 31
> c0 e8 43 b1 71 00 0f 0b eb fe 8b 75 b8 48 c7 c7 b8 68 d2 81 31 c0 e8
> 2e b1 71 00<0f> 0b eb fe 66 0f 1f 84 00 00 00 00 00 55 48 89 f7 48
89
> e5 89
> [370517.207167] RIP [<ffffffff81389e63>]
write_all_supers+0x833/0x840
> [370517.207239] RSP<ffff880e52f63b68>
> [370517.207673] ---[ end trace f208fa157676276c ]---
>
>
> I don''t know if the device failed because BTRFS did something
wrong,
> or if BTRFS failed because of the device.
>
> After this error, device cannot be accessed anymore (stuck and fdisk
> returned no info, neither do smartctl - just ''device
busy'').
>
> By the way, I''m actually doing some extensive benchmarks,
comparing
> BTRFS with XFS or EXT4. I''ll post results on this mailing list in
a
> few days, I''m sure it can be interested for you.
>
> Olivier
> --
> To unsubscribe from this list: send the line "unsubscribe
linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Apr 18, 2012 at 12:22:34PM +0800, anand jain wrote:> > [370517.204350] btrfs: 1 errors while writing supers > > [370517.204376] ------------[ cut here ]------------ > > [370517.204391] kernel BUG at fs/btrfs/disk-io.c:2880! > > ----------- > if (total_errors > max_errors) { > printk(KERN_ERR "btrfs: %d errors while writing supers\n", > total_errors); > /* This shouldn''t happen. FUA is masked off if unsupported */ > BUG(); > } > -----------This is what versions up to 3.3 do.> > accessible disk(s) is(are) below the critical number of disk(s), > btrfs has limited choice. IMO instead of BUG, we could have a > user configurable choice to either panic or fail-pending-IO, > dump and force-unmount.With 3.4-rc the BUG is not there, -EIO is returned and the filesystem goes readonly. Something like the following output, triggered by echo 1 > /sys/block/DEV/device/delete [27712.760344] sd 6:0:0:0: [sdg] Synchronizing SCSI cache [27712.760448] sd 6:0:0:0: [sdg] Stopping disk [27712.762771] ata6.00: disabled [27732.780063] BTRFS error (device sdg1) in btree_writepage_io_failed_hook:3735: IO failure (Error occured while writing out btree at 29442048) [27732.780072] btrfs is forced readonly [27732.780081] BTRFS error (device sdg1) in btree_writepage_io_failed_hook:3735: IO failure (Error occured while writing out btree at 29437952) [27732.780090] BTRFS error (device sdg1) in btree_writepage_io_failed_hook:3735: IO failure (Error occured while writing out btree at 29433856) [27732.780121] BTRFS error (device sdg1) in btrfs_commit_transaction:1455: IO failure (Error while writing out transaction.) [27732.780131] BTRFS warning (device sdg1): Skipping commit of aborted transaction. [27732.780137] delayed_refs has NO entry [27732.780714] ata6: hard resetting link david -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html