Tomasz Chmielewski
2013-Oct-21 03:21 UTC
WARNING: CPU: 2 PID: 1543 at fs/btrfs/ctree.c:1322 btrfs_search_old_slot+0x338/0x81d [btrfs]()
Got this with Linux 3.12-rc5: [67893.388276] ------------[ cut here ]------------ [67893.388335] WARNING: CPU: 2 PID: 1543 at fs/btrfs/ctree.c:1322 btrfs_search_old_slot+0x338/0x81d [btrfs]() [67893.388437] Modules linked in: veth ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop ehci_pci ehci_hcd pcspkr lpc_ich mfd_core i2c_i801 acpi_cpufreq freq_table i2c_core video button ext4 crc16 jbd2 mbcache raid1 sg sd_mod ahci libahci libata scsi_mod r8169 mii [67893.388749] CPU: 2 PID: 1543 Comm: btrfs-endio-wri Not tainted 3.12.0-rc5 #1 [67893.388798] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013 [67893.388887] 0000000000000009 ffff88054a2fb898 ffffffff8138190b 0000000000000006 [67893.388976] 0000000000000000 ffff88054a2fb8d8 ffffffff810366a9 000002527354d000 [67893.389065] ffffffffa027ce23 ffff8807ff3ddd80 ffff8803fd093800 ffff8802289b0140 [67893.389154] Call Trace: [67893.389200] [<ffffffff8138190b>] dump_stack+0x46/0x58 [67893.389248] [<ffffffff810366a9>] warn_slowpath_common+0x77/0x91 [67893.389300] [<ffffffffa027ce23>] ? btrfs_search_old_slot+0x338/0x81d [btrfs] [67893.389349] [<ffffffff810366d8>] warn_slowpath_null+0x15/0x17 [67893.389408] [<ffffffffa027ce23>] btrfs_search_old_slot+0x338/0x81d [btrfs] [67893.389464] [<ffffffffa02b1d01>] ? alloc_extent_buffer+0x2e4/0x30f [btrfs] [67893.389519] [<ffffffffa02e06e0>] __resolve_indirect_refs+0x10f/0x481 [btrfs] [67893.389574] [<ffffffffa02ad8f3>] ? release_extent_buffer+0x7d/0x8f [btrfs] [67893.389628] [<ffffffffa02e0b44>] ? __add_missing_keys.isra.16+0xf2/0x10d [btrfs] [67893.389721] [<ffffffffa02e10d5>] find_parent_nodes+0x33b/0x5d6 [btrfs] [67893.389774] [<ffffffffa02e1403>] btrfs_find_all_roots+0x7c/0xd7 [btrfs] [67893.389828] [<ffffffffa02e1bbf>] iterate_extent_inodes+0x105/0x1d6 [btrfs] [67893.389881] [<ffffffffa0299aa3>] ? record_extent_backrefs+0xc1/0xc1 [btrfs] [67893.389935] [<ffffffffa0299aa3>] ? record_extent_backrefs+0xc1/0xc1 [btrfs] [67893.389989] [<ffffffffa02e1d0f>] iterate_inodes_from_logical+0x7f/0x95 [btrfs] [67893.390081] [<ffffffffa0299a3d>] record_extent_backrefs+0x5b/0xc1 [btrfs] [67893.390134] [<ffffffffa02a213d>] btrfs_finish_ordered_io+0x79b/0x8e4 [btrfs] [67893.390188] [<ffffffffa02a2296>] finish_ordered_fn+0x10/0x12 [btrfs] [67893.390241] [<ffffffffa02bd31a>] worker_loop+0x15e/0x494 [btrfs] [67893.390294] [<ffffffffa02bd1bc>] ? btrfs_queue_worker+0x26d/0x26d [btrfs] [67893.390343] [<ffffffff8104e6be>] kthread+0xb5/0xbd [67893.390390] [<ffffffff8104e609>] ? kthread_freezable_should_stop+0x43/0x43 [67893.390439] [<ffffffff813866bc>] ret_from_fork+0x7c/0xb0 [67893.390486] [<ffffffff8104e609>] ? kthread_freezable_should_stop+0x43/0x43 [67893.390534] ---[ end trace 2b74aa1647d2ae0b ]--- [67893.390592] BTRFS critical (device sdb5): unable to find logical 69225652694147072 len 4096 [67893.390686] ------------[ cut here ]------------ [67893.390731] kernel BUG at fs/btrfs/inode.c:1595! [67893.390788] invalid opcode: 0000 [#1] SMP [67893.391952] Modules linked in: veth ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop ehci_pci ehci_hcd pcspkr lpc_ich mfd_core i2c_i801 acpi_cpufreq freq_table i2c_core video button ext4 crc16 jbd2 mbcache raid1 sg sd_mod ahci libahci libata scsi_mod r8169 mii [67893.392262] CPU: 2 PID: 1543 Comm: btrfs-endio-wri Tainted: G W 3.12.0-rc5 #1 [67893.392351] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013 [67893.392440] task: ffff8807eb66c530 ti: ffff88054a2fa000 task.ti: ffff88054a2fa000 [67893.392527] RIP: 0010:[<ffffffffa029b325>] [<ffffffffa029b325>] btrfs_merge_bio_hook+0x53/0x68 [btrfs] [67893.392624] RSP: 0018:ffff88054a2fb508 EFLAGS: 00010282 [67893.392670] RAX: 00000000ffffffea RBX: 0000000000001000 RCX: 0000000000000046 [67893.392727] RDX: 0000000000000006 RSI: 0000000000000046 RDI: ffff88081fa8d080 [67893.392783] RBP: ffff88054a2fb528 R08: 0000000000000000 R09: 0000000000000000 [67893.392850] R10: 0000000000000000 R11: ffff8807ff420000 R12: 0000000000001000 [67893.392907] R13: ffff880480ecae28 R14: 00007af82ea316f8 R15: ffff8807ff1e0040 [67893.392955] FS: 0000000000000000(0000) GS:ffff88081fa80000(0000) knlGS:0000000000000000 [67893.393043] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [67893.393089] CR2: 00007fc7b8e8f000 CR3: 000000000160b000 CR4: 00000000001407e0 [67893.393137] Stack: [67893.393179] 0000000000000004 0000000000001000 0000000000001000 ffff88054a2fb778 [67893.393268] ffff88054a2fb588 ffffffffa02af74a ffff88054a2fb588 0000000000000000 [67893.393357] ffffea0002aa7ec0 0000002000000000 ffff88054a2fb5d8 00f5f05d462df000 [67893.393446] Call Trace: [67893.393495] [<ffffffffa02af74a>] submit_extent_page.isra.40+0xea/0x17e [btrfs] [67893.393587] [<ffffffffa02b0744>] __do_readpage+0x4cc/0x56d [btrfs] [67893.393639] [<ffffffffa02af9fb>] ? repair_io_failure+0x1dc/0x1dc [btrfs] [67893.393692] [<ffffffffa0290ac8>] ? verify_parent_transid+0x146/0x146 [btrfs] [67893.393746] [<ffffffffa02ac82a>] ? btrfs_lookup_ordered_extent+0x5d/0xb3 [btrfs] [67893.393838] [<ffffffffa02b089a>] __extent_read_full_page+0xb5/0xc4 [btrfs] [67893.393891] [<ffffffffa0290ac8>] ? verify_parent_transid+0x146/0x146 [btrfs] [67893.393945] [<ffffffffa02b29b2>] read_extent_buffer_pages+0x208/0x222 [btrfs] [67893.394033] [<ffffffff8104ed3e>] ? __wake_up_bit+0x2c/0x2e [67893.394084] [<ffffffffa0290ac8>] ? verify_parent_transid+0x146/0x146 [btrfs] [67893.394136] [<ffffffffa02922bd>] btree_read_extent_buffer_pages.constprop.123+0x61/0xf9 [btrfs] [67893.394230] [<ffffffffa0293789>] read_tree_block+0x2c/0x45 [btrfs] [67893.394280] [<ffffffffa027a868>] read_block_for_search.isra.44+0x2b4/0x2fb [btrfs] [67893.394369] [<ffffffff810366b8>] ? warn_slowpath_common+0x86/0x91 [67893.394419] [<ffffffffa027cfb3>] btrfs_search_old_slot+0x4c8/0x81d [btrfs] [67893.394478] [<ffffffffa02b1d01>] ? alloc_extent_buffer+0x2e4/0x30f [btrfs] [67893.394532] [<ffffffffa02e06e0>] __resolve_indirect_refs+0x10f/0x481 [btrfs] [67893.394586] [<ffffffffa02ad8f3>] ? release_extent_buffer+0x7d/0x8f [btrfs] [67893.394639] [<ffffffffa02e0b44>] ? __add_missing_keys.isra.16+0xf2/0x10d [btrfs] [67893.394732] [<ffffffffa02e10d5>] find_parent_nodes+0x33b/0x5d6 [btrfs] [67893.394784] [<ffffffffa02e1403>] btrfs_find_all_roots+0x7c/0xd7 [btrfs] [67893.394837] [<ffffffffa02e1bbf>] iterate_extent_inodes+0x105/0x1d6 [btrfs] [67893.394890] [<ffffffffa0299aa3>] ? record_extent_backrefs+0xc1/0xc1 [btrfs] [67893.394943] [<ffffffffa0299aa3>] ? record_extent_backrefs+0xc1/0xc1 [btrfs] [67893.394996] [<ffffffffa02e1d0f>] iterate_inodes_from_logical+0x7f/0x95 [btrfs] [67893.395088] [<ffffffffa0299a3d>] record_extent_backrefs+0x5b/0xc1 [btrfs] [67893.395144] [<ffffffffa02a213d>] btrfs_finish_ordered_io+0x79b/0x8e4 [btrfs] [67893.395198] [<ffffffffa02a2296>] finish_ordered_fn+0x10/0x12 [btrfs] [67893.395250] [<ffffffffa02bd31a>] worker_loop+0x15e/0x494 [btrfs] [67893.395303] [<ffffffffa02bd1bc>] ? btrfs_queue_worker+0x26d/0x26d [btrfs] [67893.395354] [<ffffffff8104e6be>] kthread+0xb5/0xbd [67893.395402] [<ffffffff8104e609>] ? kthread_freezable_should_stop+0x43/0x43 [67893.395451] [<ffffffff813866bc>] ret_from_fork+0x7c/0xb0 [67893.395498] [<ffffffff8104e609>] ? kthread_freezable_should_stop+0x43/0x43 [67893.395546] Code: 8b 58 30 41 89 fa 48 8b b9 e8 01 00 00 48 8d 4d e8 48 c1 e2 09 45 31 c9 45 31 c0 44 89 d6 48 89 5d e8 e8 12 e1 01 00 85 c0 79 02 <0f> 0b 31 c0 4c 01 e3 48 39 5d e8 0f 92 c0 5a 59 5b 41 5c 5d c3 [67893.395718] RIP [<ffffffffa029b325>] btrfs_merge_bio_hook+0x53/0x68 [btrfs] [67893.395771] RSP <ffff88054a2fb508> [67893.396087] ---[ end trace 2b74aa1647d2ae0c ]--- -- Tomasz Chmielewski http://wpkg.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Tomasz Chmielewski
2013-Oct-21 06:10 UTC
Re: WARNING: CPU: 2 PID: 1543 at fs/btrfs/ctree.c:1322 btrfs_search_old_slot+0x338/0x81d [btrfs]()
And yet again on 3.12-rc6. Note the "BTRFS critical (device sdd1): unable to find logical 7598242613674844160 len 4096" there; after that, the fs freezes and system won''t reboot; has to be power cycled. [ 9138.478015] ------------[ cut here ]------------ [ 9138.478039] WARNING: CPU: 4 PID: 22882 at fs/btrfs/ctree.c:1322 btrfs_search_old_slot+0x338/0x81d [btrfs]() [ 9138.478053] Modules linked in: veth ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop pcspkr ehci_pci i2c_i801 button video ehci_hcd i2c_core lpc_ich mfd_core acpi_cpufreq freq_table ext4 crc16 jbd2 mbcache raid1 sg sd_mod ahci libahci libata scsi_mod r8169 mii [ 9138.478147] CPU: 4 PID: 22882 Comm: btrfs-endio-wri Not tainted 3.12.0-rc6 #1 [ 9138.478158] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013 [ 9138.478171] 0000000000000009 ffff880279dc38d8 ffffffff8138145b 0000000000000006 [ 9138.478266] 0000000000000000 ffff880279dc3918 ffffffff810366a9 00000062953b9000 [ 9138.478362] ffffffffa0240e23 ffff8807ee2ebab0 ffff880433255800 ffff88067ed69e00 [ 9138.478454] Call Trace: [ 9138.478503] [<ffffffff8138145b>] dump_stack+0x46/0x58 [ 9138.478554] [<ffffffff810366a9>] warn_slowpath_common+0x77/0x91 [ 9138.478613] [<ffffffffa0240e23>] ? btrfs_search_old_slot+0x338/0x81d [btrfs] [ 9138.478665] [<ffffffff810366d8>] warn_slowpath_null+0x15/0x17 [ 9138.478728] [<ffffffffa0240e23>] btrfs_search_old_slot+0x338/0x81d [btrfs] [ 9138.478786] [<ffffffffa02a46e8>] __resolve_indirect_refs+0x10f/0x481 [btrfs] [ 9138.478848] [<ffffffffa0275df3>] ? free_extent_buffer+0x77/0x7c [btrfs] [ 9138.478908] [<ffffffffa02a50dd>] find_parent_nodes+0x33b/0x5d6 [btrfs] [ 9138.478962] [<ffffffffa02a540b>] btrfs_find_all_roots+0x7c/0xd7 [btrfs] [ 9138.479018] [<ffffffffa02a6300>] ? ulist_add_merge+0x7e/0x14e [btrfs] [ 9138.479072] [<ffffffffa02a8562>] btrfs_qgroup_account_ref+0x109/0x40d [btrfs] [ 9138.479170] [<ffffffffa0248805>] btrfs_delayed_refs_qgroup_accounting+0xac/0xe0 [btrfs] [ 9138.479272] [<ffffffffa025c475>] __btrfs_end_transaction+0x51/0x2bd [btrfs] [ 9138.479329] [<ffffffffa0249ccd>] ? btrfs_block_rsv_release+0x49/0x4b [btrfs] [ 9138.479388] [<ffffffffa025c70c>] btrfs_end_transaction+0xb/0xd [btrfs] [ 9138.479445] [<ffffffffa0266066>] btrfs_finish_ordered_io+0x6c4/0x8e4 [btrfs] [ 9138.479499] [<ffffffff8103ee15>] ? del_timer_sync+0x2c/0x4b [ 9138.479549] [<ffffffff8103e6ed>] ? usleep_range+0x3d/0x3d [ 9138.479601] [<ffffffffa0266296>] finish_ordered_fn+0x10/0x12 [btrfs] [ 9138.479658] [<ffffffffa0281322>] worker_loop+0x15e/0x494 [btrfs] [ 9138.479711] [<ffffffffa02811c4>] ? btrfs_queue_worker+0x26d/0x26d [btrfs] [ 9138.479760] [<ffffffff8104e6be>] kthread+0xb5/0xbd [ 9138.479809] [<ffffffff8104e609>] ? kthread_freezable_should_stop+0x43/0x43 [ 9138.479858] [<ffffffff813861fc>] ret_from_fork+0x7c/0xb0 [ 9138.479905] [<ffffffff8104e609>] ? kthread_freezable_should_stop+0x43/0x43 [ 9138.479954] ---[ end trace 963bf814d16095aa ]--- [ 9138.480011] BTRFS critical (device sdd1): unable to find logical 7598242613674844160 len 4096 [ 9138.480105] ------------[ cut here ]------------ [ 9138.480150] kernel BUG at fs/btrfs/inode.c:1595! [ 9138.480197] invalid opcode: 0000 [#1] SMP [ 9138.480243] Modules linked in: veth ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop pcspkr ehci_pci i2c_i801 button video ehci_hcd i2c_core lpc_ich mfd_core acpi_cpufreq freq_table ext4 crc16 jbd2 mbcache raid1 sg sd_mod ahci libahci libata scsi_mod r8169 mii [ 9138.480557] CPU: 4 PID: 22882 Comm: btrfs-endio-wri Tainted: G W 3.12.0-rc6 #1 [ 9138.480645] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013 [ 9138.480736] task: ffff8807eacf5c80 ti: ffff880279dc2000 task.ti: ffff880279dc2000 [ 9138.480824] RIP: 0010:[<ffffffffa025f325>] [<ffffffffa025f325>] btrfs_merge_bio_hook+0x53/0x68 [btrfs] [ 9138.480920] RSP: 0018:ffff880279dc3548 EFLAGS: 00010282 [ 9138.480966] RAX: 00000000ffffffea RBX: 0000000000001000 RCX: 0000000000000046 [ 9138.481015] RDX: 0000000000000006 RSI: 0000000000000046 RDI: ffff88081fb0d080 [ 9138.481063] RBP: ffff880279dc3568 R08: 0000000000000000 R09: 0000000000000000 [ 9138.481112] R10: 0000000000000000 R11: ffff8807ff420000 R12: 0000000000001000 [ 9138.481160] R13: ffff8801e3435aa8 R14: 0034b930b330a998 R15: ffff8807ee090448 [ 9138.481209] FS: 0000000000000000(0000) GS:ffff88081fb00000(0000) knlGS:0000000000000000 [ 9138.481299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9138.481348] CR2: 00000000015d0ff0 CR3: 000000000160b000 CR4: 00000000001407e0 [ 9138.481398] Stack: [ 9138.481442] 0000000000000004 0000000000001000 0000000000001000 ffff880279dc37b8 [ 9138.481535] ffff880279dc35c8 ffffffffa0273752 ffff880279dc35c8 0000000000000000 [ 9138.481629] ffffea0013434ec0 0000002000000000 ffff880279dc3618 6972616661533000 [ 9138.481722] Call Trace: [ 9138.481772] [<ffffffffa0273752>] submit_extent_page.isra.40+0xea/0x17e [btrfs] [ 9138.481866] [<ffffffffa027474c>] __do_readpage+0x4cc/0x56d [btrfs] [ 9138.481919] [<ffffffffa0273a03>] ? repair_io_failure+0x1dc/0x1dc [btrfs] [ 9138.481972] [<ffffffffa0254ac8>] ? verify_parent_transid+0x146/0x146 [btrfs] [ 9138.482027] [<ffffffffa0270832>] ? btrfs_lookup_ordered_extent+0x5d/0xb3 [btrfs] [ 9138.482130] [<ffffffffa02748a2>] __extent_read_full_page+0xb5/0xc4 [btrfs] [ 9138.482184] [<ffffffffa0254ac8>] ? verify_parent_transid+0x146/0x146 [btrfs] [ 9138.482241] [<ffffffffa02769ba>] read_extent_buffer_pages+0x208/0x222 [btrfs] [ 9138.482340] [<ffffffffa0254ac8>] ? verify_parent_transid+0x146/0x146 [btrfs] [ 9138.482397] [<ffffffffa02562bd>] btree_read_extent_buffer_pages.constprop.123+0x61/0xf9 [btrfs] [ 9138.482498] [<ffffffffa0257789>] read_tree_block+0x2c/0x45 [btrfs] [ 9138.482554] [<ffffffffa023e868>] read_block_for_search.isra.44+0x2b4/0x2fb [btrfs] [ 9138.482647] [<ffffffff810366b8>] ? warn_slowpath_common+0x86/0x91 [ 9138.482703] [<ffffffffa0240fb3>] btrfs_search_old_slot+0x4c8/0x81d [btrfs] [ 9138.482763] [<ffffffffa02a46e8>] __resolve_indirect_refs+0x10f/0x481 [btrfs] [ 9138.482822] [<ffffffffa0275df3>] ? free_extent_buffer+0x77/0x7c [btrfs] [ 9138.482881] [<ffffffffa02a50dd>] find_parent_nodes+0x33b/0x5d6 [btrfs] [ 9138.482939] [<ffffffffa02a540b>] btrfs_find_all_roots+0x7c/0xd7 [btrfs] [ 9138.482996] [<ffffffffa02a6300>] ? ulist_add_merge+0x7e/0x14e [btrfs] [ 9138.483050] [<ffffffffa02a8562>] btrfs_qgroup_account_ref+0x109/0x40d [btrfs] [ 9138.483144] [<ffffffffa0248805>] btrfs_delayed_refs_qgroup_accounting+0xac/0xe0 [btrfs] [ 9138.483239] [<ffffffffa025c475>] __btrfs_end_transaction+0x51/0x2bd [btrfs] [ 9138.483292] [<ffffffffa0249ccd>] ? btrfs_block_rsv_release+0x49/0x4b [btrfs] [ 9138.483346] [<ffffffffa025c70c>] btrfs_end_transaction+0xb/0xd [btrfs] [ 9138.483400] [<ffffffffa0266066>] btrfs_finish_ordered_io+0x6c4/0x8e4 [btrfs] [ 9138.483451] [<ffffffff8103ee15>] ? del_timer_sync+0x2c/0x4b [ 9138.483499] [<ffffffff8103e6ed>] ? usleep_range+0x3d/0x3d [ 9138.483551] [<ffffffffa0266296>] finish_ordered_fn+0x10/0x12 [btrfs] [ 9138.483605] [<ffffffffa0281322>] worker_loop+0x15e/0x494 [btrfs] [ 9138.483659] [<ffffffffa02811c4>] ? btrfs_queue_worker+0x26d/0x26d [btrfs] [ 9138.483708] [<ffffffff8104e6be>] kthread+0xb5/0xbd [ 9138.483756] [<ffffffff8104e609>] ? kthread_freezable_should_stop+0x43/0x43 [ 9138.483807] [<ffffffff813861fc>] ret_from_fork+0x7c/0xb0 [ 9138.483857] [<ffffffff8104e609>] ? kthread_freezable_should_stop+0x43/0x43 [ 9138.483908] Code: 8b 58 30 41 89 fa 48 8b b9 e8 01 00 00 48 8d 4d e8 48 c1 e2 09 45 31 c9 45 31 c0 44 89 d6 48 89 5d e8 e8 1a e1 01 00 85 c0 79 02 <0f> 0b 31 c0 4c 01 e3 48 39 5d e8 0f 92 c0 5a 59 5b 41 5c 5d c3 [ 9138.484085] RIP [<ffffffffa025f325>] btrfs_merge_bio_hook+0x53/0x68 [btrfs] [ 9138.484140] RSP <ffff880279dc3548> [ 9138.484457] ---[ end trace 963bf814d16095ab ]--- -- Tomasz Chmielewski http://wpkg.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Liu Bo
2013-Oct-21 06:46 UTC
Re: WARNING: CPU: 2 PID: 1543 at fs/btrfs/ctree.c:1322 btrfs_search_old_slot+0x338/0x81d [btrfs]()
On Mon, Oct 21, 2013 at 03:10:32PM +0900, Tomasz Chmielewski wrote:> And yet again on 3.12-rc6.Looks like this can be reproduced reliable, can you show the steps here? thanks, -liubo> > Note the "BTRFS critical (device sdd1): unable to find logical > 7598242613674844160 len 4096" there; after that, the fs freezes and > system won''t reboot; has to be power cycled. > > [ 9138.478015] ------------[ cut here ]------------ > [ 9138.478039] WARNING: CPU: 4 PID: 22882 at fs/btrfs/ctree.c:1322 btrfs_search_old_slot+0x338/0x81d [btrfs]() > [ 9138.478053] Modules linked in: veth ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop pcspkr ehci_pci i2c_i801 button video ehci_hcd i2c_core lpc_ich mfd_core acpi_cpufreq freq_table ext4 crc16 jbd2 mbcache raid1 sg sd_mod ahci libahci libata scsi_mod r8169 mii > [ 9138.478147] CPU: 4 PID: 22882 Comm: btrfs-endio-wri Not tainted 3.12.0-rc6 #1 > [ 9138.478158] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013 > [ 9138.478171] 0000000000000009 ffff880279dc38d8 ffffffff8138145b 0000000000000006 > [ 9138.478266] 0000000000000000 ffff880279dc3918 ffffffff810366a9 00000062953b9000 > [ 9138.478362] ffffffffa0240e23 ffff8807ee2ebab0 ffff880433255800 ffff88067ed69e00 > [ 9138.478454] Call Trace: > [ 9138.478503] [<ffffffff8138145b>] dump_stack+0x46/0x58 > [ 9138.478554] [<ffffffff810366a9>] warn_slowpath_common+0x77/0x91 > [ 9138.478613] [<ffffffffa0240e23>] ? btrfs_search_old_slot+0x338/0x81d [btrfs] > [ 9138.478665] [<ffffffff810366d8>] warn_slowpath_null+0x15/0x17 > [ 9138.478728] [<ffffffffa0240e23>] btrfs_search_old_slot+0x338/0x81d [btrfs] > [ 9138.478786] [<ffffffffa02a46e8>] __resolve_indirect_refs+0x10f/0x481 [btrfs] > [ 9138.478848] [<ffffffffa0275df3>] ? free_extent_buffer+0x77/0x7c [btrfs] > [ 9138.478908] [<ffffffffa02a50dd>] find_parent_nodes+0x33b/0x5d6 [btrfs] > [ 9138.478962] [<ffffffffa02a540b>] btrfs_find_all_roots+0x7c/0xd7 [btrfs] > [ 9138.479018] [<ffffffffa02a6300>] ? ulist_add_merge+0x7e/0x14e [btrfs] > [ 9138.479072] [<ffffffffa02a8562>] btrfs_qgroup_account_ref+0x109/0x40d [btrfs] > [ 9138.479170] [<ffffffffa0248805>] btrfs_delayed_refs_qgroup_accounting+0xac/0xe0 [btrfs] > [ 9138.479272] [<ffffffffa025c475>] __btrfs_end_transaction+0x51/0x2bd [btrfs] > [ 9138.479329] [<ffffffffa0249ccd>] ? btrfs_block_rsv_release+0x49/0x4b [btrfs] > [ 9138.479388] [<ffffffffa025c70c>] btrfs_end_transaction+0xb/0xd [btrfs] > [ 9138.479445] [<ffffffffa0266066>] btrfs_finish_ordered_io+0x6c4/0x8e4 [btrfs] > [ 9138.479499] [<ffffffff8103ee15>] ? del_timer_sync+0x2c/0x4b > [ 9138.479549] [<ffffffff8103e6ed>] ? usleep_range+0x3d/0x3d > [ 9138.479601] [<ffffffffa0266296>] finish_ordered_fn+0x10/0x12 [btrfs] > [ 9138.479658] [<ffffffffa0281322>] worker_loop+0x15e/0x494 [btrfs] > [ 9138.479711] [<ffffffffa02811c4>] ? btrfs_queue_worker+0x26d/0x26d [btrfs] > [ 9138.479760] [<ffffffff8104e6be>] kthread+0xb5/0xbd > [ 9138.479809] [<ffffffff8104e609>] ? kthread_freezable_should_stop+0x43/0x43 > [ 9138.479858] [<ffffffff813861fc>] ret_from_fork+0x7c/0xb0 > [ 9138.479905] [<ffffffff8104e609>] ? kthread_freezable_should_stop+0x43/0x43 > [ 9138.479954] ---[ end trace 963bf814d16095aa ]--- > [ 9138.480011] BTRFS critical (device sdd1): unable to find logical 7598242613674844160 len 4096 > [ 9138.480105] ------------[ cut here ]------------ > [ 9138.480150] kernel BUG at fs/btrfs/inode.c:1595! > [ 9138.480197] invalid opcode: 0000 [#1] SMP > [ 9138.480243] Modules linked in: veth ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop pcspkr ehci_pci i2c_i801 button video ehci_hcd i2c_core lpc_ich mfd_core acpi_cpufreq freq_table ext4 crc16 jbd2 mbcache raid1 sg sd_mod ahci libahci libata scsi_mod r8169 mii > [ 9138.480557] CPU: 4 PID: 22882 Comm: btrfs-endio-wri Tainted: G W 3.12.0-rc6 #1 > [ 9138.480645] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013 > [ 9138.480736] task: ffff8807eacf5c80 ti: ffff880279dc2000 task.ti: ffff880279dc2000 > [ 9138.480824] RIP: 0010:[<ffffffffa025f325>] [<ffffffffa025f325>] btrfs_merge_bio_hook+0x53/0x68 [btrfs] > [ 9138.480920] RSP: 0018:ffff880279dc3548 EFLAGS: 00010282 > [ 9138.480966] RAX: 00000000ffffffea RBX: 0000000000001000 RCX: 0000000000000046 > [ 9138.481015] RDX: 0000000000000006 RSI: 0000000000000046 RDI: ffff88081fb0d080 > [ 9138.481063] RBP: ffff880279dc3568 R08: 0000000000000000 R09: 0000000000000000 > [ 9138.481112] R10: 0000000000000000 R11: ffff8807ff420000 R12: 0000000000001000 > [ 9138.481160] R13: ffff8801e3435aa8 R14: 0034b930b330a998 R15: ffff8807ee090448 > [ 9138.481209] FS: 0000000000000000(0000) GS:ffff88081fb00000(0000) knlGS:0000000000000000 > [ 9138.481299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 9138.481348] CR2: 00000000015d0ff0 CR3: 000000000160b000 CR4: 00000000001407e0 > [ 9138.481398] Stack: > [ 9138.481442] 0000000000000004 0000000000001000 0000000000001000 ffff880279dc37b8 > [ 9138.481535] ffff880279dc35c8 ffffffffa0273752 ffff880279dc35c8 0000000000000000 > [ 9138.481629] ffffea0013434ec0 0000002000000000 ffff880279dc3618 6972616661533000 > [ 9138.481722] Call Trace: > [ 9138.481772] [<ffffffffa0273752>] submit_extent_page.isra.40+0xea/0x17e [btrfs] > [ 9138.481866] [<ffffffffa027474c>] __do_readpage+0x4cc/0x56d [btrfs] > [ 9138.481919] [<ffffffffa0273a03>] ? repair_io_failure+0x1dc/0x1dc [btrfs] > [ 9138.481972] [<ffffffffa0254ac8>] ? verify_parent_transid+0x146/0x146 [btrfs] > [ 9138.482027] [<ffffffffa0270832>] ? btrfs_lookup_ordered_extent+0x5d/0xb3 [btrfs] > [ 9138.482130] [<ffffffffa02748a2>] __extent_read_full_page+0xb5/0xc4 [btrfs] > [ 9138.482184] [<ffffffffa0254ac8>] ? verify_parent_transid+0x146/0x146 [btrfs] > [ 9138.482241] [<ffffffffa02769ba>] read_extent_buffer_pages+0x208/0x222 [btrfs] > [ 9138.482340] [<ffffffffa0254ac8>] ? verify_parent_transid+0x146/0x146 [btrfs] > [ 9138.482397] [<ffffffffa02562bd>] btree_read_extent_buffer_pages.constprop.123+0x61/0xf9 [btrfs] > [ 9138.482498] [<ffffffffa0257789>] read_tree_block+0x2c/0x45 [btrfs] > [ 9138.482554] [<ffffffffa023e868>] read_block_for_search.isra.44+0x2b4/0x2fb [btrfs] > [ 9138.482647] [<ffffffff810366b8>] ? warn_slowpath_common+0x86/0x91 > [ 9138.482703] [<ffffffffa0240fb3>] btrfs_search_old_slot+0x4c8/0x81d [btrfs] > [ 9138.482763] [<ffffffffa02a46e8>] __resolve_indirect_refs+0x10f/0x481 [btrfs] > [ 9138.482822] [<ffffffffa0275df3>] ? free_extent_buffer+0x77/0x7c [btrfs] > [ 9138.482881] [<ffffffffa02a50dd>] find_parent_nodes+0x33b/0x5d6 [btrfs] > [ 9138.482939] [<ffffffffa02a540b>] btrfs_find_all_roots+0x7c/0xd7 [btrfs] > [ 9138.482996] [<ffffffffa02a6300>] ? ulist_add_merge+0x7e/0x14e [btrfs] > [ 9138.483050] [<ffffffffa02a8562>] btrfs_qgroup_account_ref+0x109/0x40d [btrfs] > [ 9138.483144] [<ffffffffa0248805>] btrfs_delayed_refs_qgroup_accounting+0xac/0xe0 [btrfs] > [ 9138.483239] [<ffffffffa025c475>] __btrfs_end_transaction+0x51/0x2bd [btrfs] > [ 9138.483292] [<ffffffffa0249ccd>] ? btrfs_block_rsv_release+0x49/0x4b [btrfs] > [ 9138.483346] [<ffffffffa025c70c>] btrfs_end_transaction+0xb/0xd [btrfs] > [ 9138.483400] [<ffffffffa0266066>] btrfs_finish_ordered_io+0x6c4/0x8e4 [btrfs] > [ 9138.483451] [<ffffffff8103ee15>] ? del_timer_sync+0x2c/0x4b > [ 9138.483499] [<ffffffff8103e6ed>] ? usleep_range+0x3d/0x3d > [ 9138.483551] [<ffffffffa0266296>] finish_ordered_fn+0x10/0x12 [btrfs] > [ 9138.483605] [<ffffffffa0281322>] worker_loop+0x15e/0x494 [btrfs] > [ 9138.483659] [<ffffffffa02811c4>] ? btrfs_queue_worker+0x26d/0x26d [btrfs] > [ 9138.483708] [<ffffffff8104e6be>] kthread+0xb5/0xbd > [ 9138.483756] [<ffffffff8104e609>] ? kthread_freezable_should_stop+0x43/0x43 > [ 9138.483807] [<ffffffff813861fc>] ret_from_fork+0x7c/0xb0 > [ 9138.483857] [<ffffffff8104e609>] ? kthread_freezable_should_stop+0x43/0x43 > [ 9138.483908] Code: 8b 58 30 41 89 fa 48 8b b9 e8 01 00 00 48 8d 4d e8 48 c1 e2 09 45 31 c9 45 31 c0 44 89 d6 48 89 5d e8 e8 1a e1 01 00 85 c0 79 02 <0f> 0b 31 c0 4c 01 e3 48 39 5d e8 0f 92 c0 5a 59 5b 41 5c 5d c3 > [ 9138.484085] RIP [<ffffffffa025f325>] btrfs_merge_bio_hook+0x53/0x68 [btrfs] > [ 9138.484140] RSP <ffff880279dc3548> > [ 9138.484457] ---[ end trace 963bf814d16095ab ]--- > > > -- > Tomasz Chmielewski > http://wpkg.org > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Tomasz Chmielewski
2013-Oct-21 07:06 UTC
Re: WARNING: CPU: 2 PID: 1543 at fs/btrfs/ctree.c:1322 btrfs_search_old_slot+0x338/0x81d [btrfs]()
Not really. I''m running "backup" in a following way: - rsync --inplace --delete -H ... from remote to local btrfs filesystem - snapshot - remove old snapshots - repeat for a different server While previously this system would hang every few days (oom, allocation failures; see a different thread, "Lots of trouble hanging when rm files with many extents") - now I''ve added "autodefrag" mount option, and it hanged twice in a row. -- Tomasz Chmielewski http://wpkg.org On Mon, 21 Oct 2013 14:46:12 +0800 Liu Bo <bo.li.liu@oracle.com> wrote:> On Mon, Oct 21, 2013 at 03:10:32PM +0900, Tomasz Chmielewski wrote: > > And yet again on 3.12-rc6. > > Looks like this can be reproduced reliable, can you show the steps > here? > > thanks, > -liubo > > > > > Note the "BTRFS critical (device sdd1): unable to find logical > > 7598242613674844160 len 4096" there; after that, the fs freezes and > > system won''t reboot; has to be power cycled. > > > > [ 9138.478015] ------------[ cut here ]------------ > > [ 9138.478039] WARNING: CPU: 4 PID: 22882 at fs/btrfs/ctree.c:1322 > > btrfs_search_old_slot+0x338/0x81d [btrfs]() [ 9138.478053] Modules > > linked in: veth ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 > > nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables > > cpufreq_ondemand cpufreq_conservative cpufreq_powersave > > cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate > > loop pcspkr ehci_pci i2c_i801 button video ehci_hcd i2c_core > > lpc_ich mfd_core acpi_cpufreq freq_table ext4 crc16 jbd2 mbcache > > raid1 sg sd_mod ahci libahci libata scsi_mod r8169 mii > > [ 9138.478147] CPU: 4 PID: 22882 Comm: btrfs-endio-wri Not tainted > > 3.12.0-rc6 #1 [ 9138.478158] Hardware name: System manufacturer > > System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013 > > [ 9138.478171] 0000000000000009 ffff880279dc38d8 ffffffff8138145b > > 0000000000000006 [ 9138.478266] 0000000000000000 ffff880279dc3918 > > ffffffff810366a9 00000062953b9000 [ 9138.478362] ffffffffa0240e23 > > ffff8807ee2ebab0 ffff880433255800 ffff88067ed69e00 [ 9138.478454] > > Call Trace: [ 9138.478503] [<ffffffff8138145b>] > > dump_stack+0x46/0x58 [ 9138.478554] [<ffffffff810366a9>] > > warn_slowpath_common+0x77/0x91 [ 9138.478613] > > [<ffffffffa0240e23>] ? btrfs_search_old_slot+0x338/0x81d [btrfs] > > [ 9138.478665] [<ffffffff810366d8>] warn_slowpath_null+0x15/0x17 > > [ 9138.478728] [<ffffffffa0240e23>] > > btrfs_search_old_slot+0x338/0x81d [btrfs] [ 9138.478786] > > [<ffffffffa02a46e8>] __resolve_indirect_refs+0x10f/0x481 [btrfs] > > [ 9138.478848] [<ffffffffa0275df3>] ? free_extent_buffer+0x77/0x7c > > [btrfs] [ 9138.478908] [<ffffffffa02a50dd>] > > find_parent_nodes+0x33b/0x5d6 [btrfs] [ 9138.478962] > > [<ffffffffa02a540b>] btrfs_find_all_roots+0x7c/0xd7 [btrfs] > > [ 9138.479018] [<ffffffffa02a6300>] ? ulist_add_merge+0x7e/0x14e > > [btrfs] [ 9138.479072] [<ffffffffa02a8562>] > > btrfs_qgroup_account_ref+0x109/0x40d [btrfs] [ 9138.479170] > > [<ffffffffa0248805>] btrfs_delayed_refs_qgroup_accounting+0xac/0xe0 > > [btrfs] [ 9138.479272] [<ffffffffa025c475>] > > __btrfs_end_transaction+0x51/0x2bd [btrfs] [ 9138.479329] > > [<ffffffffa0249ccd>] ? btrfs_block_rsv_release+0x49/0x4b [btrfs] > > [ 9138.479388] [<ffffffffa025c70c>] btrfs_end_transaction+0xb/0xd > > [btrfs] [ 9138.479445] [<ffffffffa0266066>] > > btrfs_finish_ordered_io+0x6c4/0x8e4 [btrfs] [ 9138.479499] > > [<ffffffff8103ee15>] ? del_timer_sync+0x2c/0x4b [ 9138.479549] > > [<ffffffff8103e6ed>] ? usleep_range+0x3d/0x3d [ 9138.479601] > > [<ffffffffa0266296>] finish_ordered_fn+0x10/0x12 [btrfs] > > [ 9138.479658] [<ffffffffa0281322>] worker_loop+0x15e/0x494 > > [btrfs] [ 9138.479711] [<ffffffffa02811c4>] ? > > btrfs_queue_worker+0x26d/0x26d [btrfs] [ 9138.479760] > > [<ffffffff8104e6be>] kthread+0xb5/0xbd [ 9138.479809] > > [<ffffffff8104e609>] ? kthread_freezable_should_stop+0x43/0x43 > > [ 9138.479858] [<ffffffff813861fc>] ret_from_fork+0x7c/0xb0 > > [ 9138.479905] [<ffffffff8104e609>] ? > > kthread_freezable_should_stop+0x43/0x43 [ 9138.479954] ---[ end > > trace 963bf814d16095aa ]--- [ 9138.480011] BTRFS critical (device > > sdd1): unable to find logical 7598242613674844160 len 4096 > > [ 9138.480105] ------------[ cut here ]------------ [ 9138.480150] > > kernel BUG at fs/btrfs/inode.c:1595! [ 9138.480197] invalid opcode: > > 0000 [#1] SMP [ 9138.480243] Modules linked in: veth ipt_MASQUERADE > > iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat > > nf_conntrack ip_tables x_tables cpufreq_ondemand > > cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc > > ipv6 btrfs xor raid6_pq zlib_deflate loop pcspkr ehci_pci i2c_i801 > > button video ehci_hcd i2c_core lpc_ich mfd_core acpi_cpufreq > > freq_table ext4 crc16 jbd2 mbcache raid1 sg sd_mod ahci libahci > > libata scsi_mod r8169 mii [ 9138.480557] CPU: 4 PID: 22882 Comm: > > btrfs-endio-wri Tainted: G W 3.12.0-rc6 #1 [ 9138.480645] > > Hardware name: System manufacturer System Product Name/P8H77-M PRO, > > BIOS 1101 02/04/2013 [ 9138.480736] task: ffff8807eacf5c80 ti: > > ffff880279dc2000 task.ti: ffff880279dc2000 [ 9138.480824] RIP: > > 0010:[<ffffffffa025f325>] [<ffffffffa025f325>] > > btrfs_merge_bio_hook+0x53/0x68 [btrfs] [ 9138.480920] RSP: > > 0018:ffff880279dc3548 EFLAGS: 00010282 [ 9138.480966] RAX: > > 00000000ffffffea RBX: 0000000000001000 RCX: 0000000000000046 > > [ 9138.481015] RDX: 0000000000000006 RSI: 0000000000000046 RDI: > > ffff88081fb0d080 [ 9138.481063] RBP: ffff880279dc3568 R08: > > 0000000000000000 R09: 0000000000000000 [ 9138.481112] R10: > > 0000000000000000 R11: ffff8807ff420000 R12: 0000000000001000 > > [ 9138.481160] R13: ffff8801e3435aa8 R14: 0034b930b330a998 R15: > > ffff8807ee090448 [ 9138.481209] FS: 0000000000000000(0000) > > GS:ffff88081fb00000(0000) knlGS:0000000000000000 [ 9138.481299] > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9138.481348] > > CR2: 00000000015d0ff0 CR3: 000000000160b000 CR4: 00000000001407e0 > > [ 9138.481398] Stack: [ 9138.481442] 0000000000000004 > > 0000000000001000 0000000000001000 ffff880279dc37b8 [ 9138.481535] > > ffff880279dc35c8 ffffffffa0273752 ffff880279dc35c8 0000000000000000 > > [ 9138.481629] ffffea0013434ec0 0000002000000000 ffff880279dc3618 > > 6972616661533000 [ 9138.481722] Call Trace: [ 9138.481772] > > [<ffffffffa0273752>] submit_extent_page.isra.40+0xea/0x17e [btrfs] > > [ 9138.481866] [<ffffffffa027474c>] __do_readpage+0x4cc/0x56d > > [btrfs] [ 9138.481919] [<ffffffffa0273a03>] ? > > repair_io_failure+0x1dc/0x1dc [btrfs] [ 9138.481972] > > [<ffffffffa0254ac8>] ? verify_parent_transid+0x146/0x146 [btrfs] > > [ 9138.482027] [<ffffffffa0270832>] ? > > btrfs_lookup_ordered_extent+0x5d/0xb3 [btrfs] [ 9138.482130] > > [<ffffffffa02748a2>] __extent_read_full_page+0xb5/0xc4 [btrfs] > > [ 9138.482184] [<ffffffffa0254ac8>] ? > > verify_parent_transid+0x146/0x146 [btrfs] [ 9138.482241] > > [<ffffffffa02769ba>] read_extent_buffer_pages+0x208/0x222 [btrfs] > > [ 9138.482340] [<ffffffffa0254ac8>] ? > > verify_parent_transid+0x146/0x146 [btrfs] [ 9138.482397] > > [<ffffffffa02562bd>] > > btree_read_extent_buffer_pages.constprop.123+0x61/0xf9 [btrfs] > > [ 9138.482498] [<ffffffffa0257789>] read_tree_block+0x2c/0x45 > > [btrfs] [ 9138.482554] [<ffffffffa023e868>] > > read_block_for_search.isra.44+0x2b4/0x2fb [btrfs] [ 9138.482647] > > [<ffffffff810366b8>] ? warn_slowpath_common+0x86/0x91 > > [ 9138.482703] [<ffffffffa0240fb3>] > > btrfs_search_old_slot+0x4c8/0x81d [btrfs] [ 9138.482763] > > [<ffffffffa02a46e8>] __resolve_indirect_refs+0x10f/0x481 [btrfs] > > [ 9138.482822] [<ffffffffa0275df3>] ? free_extent_buffer+0x77/0x7c > > [btrfs] [ 9138.482881] [<ffffffffa02a50dd>] > > find_parent_nodes+0x33b/0x5d6 [btrfs] [ 9138.482939] > > [<ffffffffa02a540b>] btrfs_find_all_roots+0x7c/0xd7 [btrfs] > > [ 9138.482996] [<ffffffffa02a6300>] ? ulist_add_merge+0x7e/0x14e > > [btrfs] [ 9138.483050] [<ffffffffa02a8562>] > > btrfs_qgroup_account_ref+0x109/0x40d [btrfs] [ 9138.483144] > > [<ffffffffa0248805>] btrfs_delayed_refs_qgroup_accounting+0xac/0xe0 > > [btrfs] [ 9138.483239] [<ffffffffa025c475>] > > __btrfs_end_transaction+0x51/0x2bd [btrfs] [ 9138.483292] > > [<ffffffffa0249ccd>] ? btrfs_block_rsv_release+0x49/0x4b [btrfs] > > [ 9138.483346] [<ffffffffa025c70c>] btrfs_end_transaction+0xb/0xd > > [btrfs] [ 9138.483400] [<ffffffffa0266066>] > > btrfs_finish_ordered_io+0x6c4/0x8e4 [btrfs] [ 9138.483451] > > [<ffffffff8103ee15>] ? del_timer_sync+0x2c/0x4b [ 9138.483499] > > [<ffffffff8103e6ed>] ? usleep_range+0x3d/0x3d [ 9138.483551] > > [<ffffffffa0266296>] finish_ordered_fn+0x10/0x12 [btrfs] > > [ 9138.483605] [<ffffffffa0281322>] worker_loop+0x15e/0x494 > > [btrfs] [ 9138.483659] [<ffffffffa02811c4>] ? > > btrfs_queue_worker+0x26d/0x26d [btrfs] [ 9138.483708] > > [<ffffffff8104e6be>] kthread+0xb5/0xbd [ 9138.483756] > > [<ffffffff8104e609>] ? kthread_freezable_should_stop+0x43/0x43 > > [ 9138.483807] [<ffffffff813861fc>] ret_from_fork+0x7c/0xb0 > > [ 9138.483857] [<ffffffff8104e609>] ? > > kthread_freezable_should_stop+0x43/0x43 [ 9138.483908] Code: 8b 58 > > 30 41 89 fa 48 8b b9 e8 01 00 00 48 8d 4d e8 48 c1 e2 09 45 31 c9 > > 45 31 c0 44 89 d6 48 89 5d e8 e8 1a e1 01 00 85 c0 79 02 <0f> 0b 31 > > c0 4c 01 e3 48 39 5d e8 0f 92 c0 5a 59 5b 41 5c 5d c3 > > [ 9138.484085] RIP [<ffffffffa025f325>] > > btrfs_merge_bio_hook+0x53/0x68 [btrfs] [ 9138.484140] RSP > > <ffff880279dc3548> [ 9138.484457] ---[ end trace 963bf814d16095ab > > ]--- > > > > > > -- > > Tomasz Chmielewski > > http://wpkg.org > > -- > > To unsubscribe from this list: send the line "unsubscribe > > linux-btrfs" in the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
David Sterba
2013-Oct-21 12:53 UTC
Re: WARNING: CPU: 2 PID: 1543 at fs/btrfs/ctree.c:1322 btrfs_search_old_slot+0x338/0x81d [btrfs]()
On Mon, Oct 21, 2013 at 03:10:32PM +0900, Tomasz Chmielewski wrote:> Note the "BTRFS critical (device sdd1): unable to find logical > 7598242613674844160 len 4096" there; after that, the fs freezes and > system won''t reboot; has to be power cycled.From the first mail: [67893.390592] BTRFS critical (device sdb5): unable to find logical 69225652694147072 len 4096> [ 9138.480011] BTRFS critical (device sdd1): unable to find logical 7598242613674844160 len 409669225652694147072 = 0xf5f05d462de000 = something that looks like inctruction fragments 7598242613674844160 = 0x6972616661532000 = "irafaS" -> Safari? Something has corrupted the memory and stored the bytes on disk where btrfs finds the garbage and chokes. Try read-only mount and run scrub. david -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Tomasz Chmielewski
2013-Oct-21 17:50 UTC
Re: WARNING: CPU: 2 PID: 1543 at fs/btrfs/ctree.c:1322 btrfs_search_old_slot+0x338/0x81d [btrfs]()
On Mon, 21 Oct 2013 14:53:17 +0200 David Sterba <dsterba@suse.cz> wrote:> On Mon, Oct 21, 2013 at 03:10:32PM +0900, Tomasz Chmielewski wrote: > > Note the "BTRFS critical (device sdd1): unable to find logical > > 7598242613674844160 len 4096" there; after that, the fs freezes and > > system won''t reboot; has to be power cycled. > > From the first mail: > [67893.390592] BTRFS critical (device sdb5): unable to find logical > 69225652694147072 len 4096 > > > [ 9138.480011] BTRFS critical (device sdd1): unable to find logical > > 7598242613674844160 len 4096 > > 69225652694147072 = 0xf5f05d462de000 = something that looks like > inctruction fragments 7598242613674844160 = 0x6972616661532000 > "irafaS" -> Safari? > > Something has corrupted the memory and stored the bytes on disk where > btrfs finds the garbage and chokes. Try read-only mount and run scrub.I''m getting these every few seconds when running scrub - expected? kernel:[ 3809.964073] BUG: soft lockup - CPU#6 stuck for 23s! [btrfs:13127] Message from syslogd@bkp010 at Oct 21 17:49:22 ... kernel:[ 3809.964075] BUG: soft lockup - CPU#3 stuck for 23s! [btrfs:13128] Message from syslogd@bkp010 at Oct 21 17:49:30 ... kernel:[ 3817.959359] BUG: soft lockup - CPU#4 stuck for 22s! [btrfs:13189] Message from syslogd@bkp010 at Oct 21 17:49:30 ... kernel:[ 3817.959362] BUG: soft lockup - CPU#1 stuck for 22s! [btrfs:13190] Message from syslogd@bkp010 at Oct 21 17:49:50 ... kernel:[ 3837.947582] BUG: soft lockup - CPU#6 stuck for 22s! [btrfs:13127] Message from syslogd@bkp010 at Oct 21 17:49:50 ... kernel:[ 3837.947585] BUG: soft lockup - CPU#3 stuck for 22s! [btrfs:13128] -- Tomasz Chmielewski http://wpkg.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
David Sterba
2013-Oct-22 15:46 UTC
Re: WARNING: CPU: 2 PID: 1543 at fs/btrfs/ctree.c:1322 btrfs_search_old_slot+0x338/0x81d [btrfs]()
On Tue, Oct 22, 2013 at 02:50:25AM +0900, Tomasz Chmielewski wrote:> I''m getting these every few seconds when running scrub - expected?The system probably locked up again due to the corrupted data. Looking at the stacktraces again, this is due to a BUG_ON in btrfs_merge_bio_hook, and it does not look like an easy one to fix. The filesystem is damaged, but we don''t know to what extent. It''s possible that ''btrfs restore'' will be able to get to the data. david -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Tomasz Chmielewski
2013-Oct-22 16:04 UTC
Re: WARNING: CPU: 2 PID: 1543 at fs/btrfs/ctree.c:1322 btrfs_search_old_slot+0x338/0x81d [btrfs]()
On Tue, 22 Oct 2013 17:46:19 +0200 David Sterba <dsterba@suse.cz> wrote:> On Tue, Oct 22, 2013 at 02:50:25AM +0900, Tomasz Chmielewski wrote: > > I''m getting these every few seconds when running scrub - expected? > > The system probably locked up again due to the corrupted data. Looking > at the stacktraces again, this is due to a BUG_ON in > btrfs_merge_bio_hook, and it does not look like an easy one to fix. > > The filesystem is damaged, but we don''t know to what extent. It''s > possible that ''btrfs restore'' will be able to get to the data.Actually it''s still working. If I run this: # btrfs scrub status /mnt/lxc1 scrub status for 8d08ad6d-4543-4fe5-8b1b-640dc1423d41 scrub started at Tue Oct 22 09:53:34 2013, running for 22101 seconds total bytes scrubbed: 2.61TB with 0 errors the number of bytes scrubbed is growing every now and then (still, 0 errors). However, it''s not possible to work with this system via SSH, because these keep popping up few times every minute: Message from syslogd@bkp010 at Oct 22 16:02:10 ... kernel:[22219.117012] BUG: soft lockup - CPU#2 stuck for 23s! [btrfs:5673] Message from syslogd@bkp010 at Oct 22 16:02:38 ... kernel:[22247.100515] BUG: soft lockup - CPU#0 stuck for 23s! [btrfs:5674] Message from syslogd@bkp010 at Oct 22 16:02:38 ... kernel:[22247.100519] BUG: soft lockup - CPU#2 stuck for 23s! [btrfs:5673] Will wait to see if it finishes or if it locks up at some point. -- Tomasz Chmielewski http://wpkg.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
David Sterba
2013-Oct-22 16:20 UTC
Re: WARNING: CPU: 2 PID: 1543 at fs/btrfs/ctree.c:1322 btrfs_search_old_slot+0x338/0x81d [btrfs]()
On Wed, Oct 23, 2013 at 01:04:11AM +0900, Tomasz Chmielewski wrote:> Actually it''s still working. > > If I run this: > > # btrfs scrub status /mnt/lxc1 > scrub status for 8d08ad6d-4543-4fe5-8b1b-640dc1423d41 > scrub started at Tue Oct 22 09:53:34 2013, running for 22101 seconds > total bytes scrubbed: 2.61TB with 0 errorsThis looks promising.> However, it''s not possible to work with this system via SSH, because > these keep popping up few times every minute:Probably only some of the files that get accessed during ssh login is corrupted, but the scrub does not get far enough to let you know the filenames. You can try to look into ''lsof'' output which files are open by sshd or it''s children.> kernel:[22219.117012] BUG: soft lockup - CPU#2 stuck for 23s! [btrfs:5673] > kernel:[22247.100515] BUG: soft lockup - CPU#0 stuck for 23s! [btrfs:5674] > kernel:[22247.100519] BUG: soft lockup - CPU#2 stuck for 23s! [btrfs:5673]Cpus 0 and 2 are stuck and every other attempt to access the broken files will pin another cpu. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Tomasz Chmielewski
2013-Oct-23 04:28 UTC
Re: WARNING: CPU: 2 PID: 1543 at fs/btrfs/ctree.c:1322 btrfs_search_old_slot+0x338/0x81d [btrfs]()
On Tue, 22 Oct 2013 18:20:19 +0200 David Sterba <dsterba@suse.cz> wrote:> > However, it''s not possible to work with this system via SSH, because > > these keep popping up few times every minute: > > Probably only some of the files that get accessed during ssh login is > corrupted, but the scrub does not get far enough to let you know the > filenames. You can try to look into ''lsof'' output which files are open > by sshd or it''s children.It''s not the case. Files in btrfs mount are not accessed in any way - the scrub was started after restart, and there is no way anything on this system can accidentally access data there. This is further confirmed by lsof output.> > kernel:[22219.117012] BUG: soft lockup - CPU#2 stuck for 23s! > > [btrfs:5673] kernel:[22247.100515] BUG: soft lockup - CPU#0 stuck > > for 23s! [btrfs:5674] kernel:[22247.100519] BUG: soft lockup - > > CPU#2 stuck for 23s! [btrfs:5673] > > Cpus 0 and 2 are stuck and every other attempt to access the broken > files will pin another cpu.Scrub was running for some time; after scrubbung about 2.8 TB, the system was so slow, that it was barely possible to launch any command. "reboot" issued via ssh took ~8 hours to execute. A new scrub started after the reboot immediately begins to show "BUG: soft lockup - CPU#2 stuck for 23s!". Anyway, it is RAID-1 - I would expect the scrub to either correct corrupt data (from a copy on the other disk), or mark it as invalid (both copies corrupt), but not "nearly hang" the server, or? -- Tomasz Chmielewski http://wpkg.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Tomasz Chmielewski
2013-Oct-24 05:54 UTC
Re: WARNING: CPU: 2 PID: 1543 at fs/btrfs/ctree.c:1322 btrfs_search_old_slot+0x338/0x81d [btrfs]()
On Wed, 23 Oct 2013 13:28:44 +0900
Tomasz Chmielewski <tch@virtall.com> wrote:
OK, putting high load aside - it finished, with no errors:
# btrfs scrub status /mnt/lxc1
scrub status for 8d08ad6d-4543-4fe5-8b1b-640dc1423d41
scrub started at Wed Oct 23 04:19:55 2013 and finished after 68479
seconds
total bytes scrubbed: 3.50TB with 0 errors
--
Tomasz Chmielewski
http://wpkg.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Tomasz Chmielewski
2013-Oct-24 08:41 UTC
Re: WARNING: CPU: 2 PID: 1543 at fs/btrfs/ctree.c:1322 btrfs_search_old_slot+0x338/0x81d [btrfs]()
On Thu, 24 Oct 2013 14:54:15 +0900 Tomasz Chmielewski <tch@virtall.com> wrote:> On Wed, 23 Oct 2013 13:28:44 +0900 > Tomasz Chmielewski <tch@virtall.com> wrote: > > OK, putting high load aside - it finished, with no errors: > > # btrfs scrub status /mnt/lxc1 > scrub status for 8d08ad6d-4543-4fe5-8b1b-640dc1423d41 > scrub started at Wed Oct 23 04:19:55 2013 and finished after > 68479 seconds total bytes scrubbed: 3.50TB with 0 errorsAnd right some time after using it again - no "unable to find logical" this time, and didn''t "hang" (yet?). [102428.040859] ------------[ cut here ]------------ [102428.040939] WARNING: CPU: 4 PID: 26530 at fs/btrfs/ctree.c:1322 btrfs_search_old_slot+0x338/0x81d [btrfs]() [102428.041055] Modules linked in: veth ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop i2c_i801 i2c_core lpc_ich mfd_core pcspkr acpi_cpufreq freq_table ehci_pci button video ehci_hcd ext4 crc16 jbd2 mbcache raid1 sg sd_mod ahci libahci r8169 mii libata scsi_mod [102428.041474] CPU: 4 PID: 26530 Comm: btrfs-endio-wri Not tainted 3.12.0-rc6 #1 [102428.041585] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013 [102428.041699] 0000000000000009 ffff8801fa1d37e8 ffffffff8138145b 0000000000000006 [102428.041816] 0000000000000000 ffff8801fa1d3828 ffffffff810366a9 0000023be09b3000 [102428.041930] ffffffffa0236e23 ffff8807eb8b9360 ffff88060beb1800 ffff8804e36e9040 [102428.042046] Call Trace: [102428.042106] [<ffffffff8138145b>] dump_stack+0x46/0x58 [102428.042169] [<ffffffff810366a9>] warn_slowpath_common+0x77/0x91 [102428.042240] [<ffffffffa0236e23>] ? btrfs_search_old_slot+0x338/0x81d [btrfs] [102428.042353] [<ffffffff810366d8>] warn_slowpath_null+0x15/0x17 [102428.042423] [<ffffffffa0236e23>] btrfs_search_old_slot+0x338/0x81d [btrfs] [102428.042503] [<ffffffffa029a6e8>] __resolve_indirect_refs+0x10f/0x481 [btrfs] [102428.042631] [<ffffffffa026bdf3>] ? free_extent_buffer+0x77/0x7c [btrfs] [102428.042708] [<ffffffffa029b0dd>] find_parent_nodes+0x33b/0x5d6 [btrfs] [102428.042784] [<ffffffffa029b40b>] btrfs_find_all_roots+0x7c/0xd7 [btrfs] [102428.042858] [<ffffffffa029e562>] btrfs_qgroup_account_ref+0x109/0x40d [btrfs] [102428.042981] [<ffffffffa029e41f>] ? btrfs_qgroup_record_ref+0x2d/0x67 [btrfs] [102428.043103] [<ffffffffa023e805>] btrfs_delayed_refs_qgroup_accounting+0xac/0xe0 [btrfs] [102428.043229] [<ffffffffa0252475>] __btrfs_end_transaction+0x51/0x2bd [btrfs] [102428.043303] [<ffffffffa025270c>] btrfs_end_transaction+0xb/0xd [btrfs] [102428.043379] [<ffffffffa025b95d>] relink_extent_backref+0x68a/0x6cf [btrfs] [102428.043455] [<ffffffffa025c185>] btrfs_finish_ordered_io+0x7e3/0x8e4 [btrfs] [102428.043569] [<ffffffff810e316a>] ? kmem_cache_free+0x164/0x17a [102428.043633] [<ffffffff810ae673>] ? mempool_free_slab+0x12/0x14 [102428.043705] [<ffffffffa025c296>] finish_ordered_fn+0x10/0x12 [btrfs] [102428.043781] [<ffffffffa0277322>] worker_loop+0x15e/0x494 [btrfs] [102428.043855] [<ffffffffa02771c4>] ? btrfs_queue_worker+0x26d/0x26d [btrfs] [102428.043921] [<ffffffff8104e6be>] kthread+0xb5/0xbd [102428.043982] [<ffffffff8104e609>] ? kthread_freezable_should_stop+0x43/0x43 [102428.044046] [<ffffffff813861fc>] ret_from_fork+0x7c/0xb0 [102428.044109] [<ffffffff8104e609>] ? kthread_freezable_should_stop+0x43/0x43 [102428.044193] ---[ end trace d0bf6f5b2d22f3d8 ]--- [102428.044269] ------------[ cut here ]------------ [102428.044344] WARNING: CPU: 4 PID: 26530 at fs/btrfs/ctree.c:1322 btrfs_search_old_slot+0x338/0x81d [btrfs]() [102428.044466] Modules linked in: veth ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop i2c_i801 i2c_core lpc_ich mfd_core pcspkr acpi_cpufreq freq_table ehci_pci button video ehci_hcd ext4 crc16 jbd2 mbcache raid1 sg sd_mod ahci libahci r8169 mii libata scsi_mod [102428.044881] CPU: 4 PID: 26530 Comm: btrfs-endio-wri Tainted: G W 3.12.0-rc6 #1 [102428.044994] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013 [102428.045109] 0000000000000009 ffff8801fa1d3758 ffffffff8138145b 0000000000000006 [102428.045225] 0000000000000000 ffff8801fa1d3798 ffffffff810366a9 0000023be09b3000 [102428.045340] ffffffffa0236e23 ffff8807eb8b9360 ffff88060beb1800 ffff8804e36e9040 [102428.045455] Call Trace: [102428.045513] [<ffffffff8138145b>] dump_stack+0x46/0x58 [102428.045574] [<ffffffff810366a9>] warn_slowpath_common+0x77/0x91 [102428.045644] [<ffffffffa0236e23>] ? btrfs_search_old_slot+0x338/0x81d [btrfs] [102428.045757] [<ffffffff810366d8>] warn_slowpath_null+0x15/0x17 [102428.045826] [<ffffffffa0236e23>] btrfs_search_old_slot+0x338/0x81d [btrfs] [102428.045891] [<ffffffff8109b72e>] ? call_rcu_sched+0x12/0x14 [102428.045967] [<ffffffffa026bd01>] ? alloc_extent_buffer+0x2dc/0x30f [btrfs] [102428.046038] [<ffffffffa0239053>] btrfs_next_old_leaf+0xc9/0x3a6 [btrfs] [102428.046115] [<ffffffffa029a92a>] __resolve_indirect_refs+0x351/0x481 [btrfs] [102428.046240] [<ffffffffa026bdf3>] ? free_extent_buffer+0x77/0x7c [btrfs] [102428.046315] [<ffffffffa029b0dd>] find_parent_nodes+0x33b/0x5d6 [btrfs] [102428.046389] [<ffffffffa029b40b>] btrfs_find_all_roots+0x7c/0xd7 [btrfs] [102428.046464] [<ffffffffa029e562>] btrfs_qgroup_account_ref+0x109/0x40d [btrfs] [102428.046586] [<ffffffffa029e41f>] ? btrfs_qgroup_record_ref+0x2d/0x67 [btrfs] [102428.046707] [<ffffffffa023e805>] btrfs_delayed_refs_qgroup_accounting+0xac/0xe0 [btrfs] [102428.046832] [<ffffffffa0252475>] __btrfs_end_transaction+0x51/0x2bd [btrfs] [102428.046906] [<ffffffffa025270c>] btrfs_end_transaction+0xb/0xd [btrfs] [102428.046981] [<ffffffffa025b95d>] relink_extent_backref+0x68a/0x6cf [btrfs] [102428.047055] [<ffffffffa025c185>] btrfs_finish_ordered_io+0x7e3/0x8e4 [btrfs] [102428.047168] [<ffffffff810e316a>] ? kmem_cache_free+0x164/0x17a [102428.047230] [<ffffffff810ae673>] ? mempool_free_slab+0x12/0x14 [102428.047303] [<ffffffffa025c296>] finish_ordered_fn+0x10/0x12 [btrfs] [102428.047378] [<ffffffffa0277322>] worker_loop+0x15e/0x494 [btrfs] [102428.047451] [<ffffffffa02771c4>] ? btrfs_queue_worker+0x26d/0x26d [btrfs] [102428.047516] [<ffffffff8104e6be>] kthread+0xb5/0xbd [102428.047577] [<ffffffff8104e609>] ? kthread_freezable_should_stop+0x43/0x43 [102428.047642] [<ffffffff813861fc>] ret_from_fork+0x7c/0xb0 [102428.047703] [<ffffffff8104e609>] ? kthread_freezable_should_stop+0x43/0x43 [102428.047767] ---[ end trace d0bf6f5b2d22f3d9 ]--- -- Tomasz Chmielewski http://wpkg.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html