Jim Schutt
2012-Sep-25 19:08 UTC
various processes blocked in btrfs_tree_read_lock for more than 120 secs
Hi, I''m hitting the following on a btrfs filesystem used as a Ceph OSD data store, under a heavy write load. My kernel is current Linus master (commit 56d27adcb536) merged with Josef Bacik''s btrfs-next master (commit d5b04fb3bbb6). What can I do to help resolve this? [ 1558.754105] INFO: task btrfs-endio-wri:28012 blocked for more than 120 seconds. [ 1558.761569] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1558.769575] btrfs-endio-wri D ffffffff8160b9c0 0 28012 2 0x00000000 [ 1558.777063] ffff8806230b39c0 0000000000000046 0000000000000002 ffff8806230b3fd8 [ 1558.785725] ffff8806230b2010 ffff8806230b2000 ffff8806230b2000 ffff8806230b2000 [ 1558.793776] ffff8806230b3fd8 ffff8806230b2000 ffff8806245a9740 ffff880615825d00 [ 1558.802396] Call Trace: [ 1558.805018] [<ffffffff8148b51d>] schedule+0x5d/0x60 [ 1558.810189] [<ffffffffa05c7423>] btrfs_tree_read_lock+0xf3/0x140 [btrfs] [ 1558.817223] [<ffffffff810660a0>] ? wake_up_bit+0x40/0x40 [ 1558.822851] [<ffffffffa056cf53>] btrfs_read_lock_root_node+0x23/0x50 [btrfs] [ 1558.830150] [<ffffffffa0575000>] btrfs_search_slot+0x1f0/0x730 [btrfs] [ 1558.837722] [<ffffffffa0588dbd>] btrfs_lookup_file_extent+0x3d/0x40 [btrfs] [ 1558.845099] [<ffffffffa05a80e1>] __btrfs_drop_extents+0x171/0xad0 [btrfs] [ 1558.852506] [<ffffffff8114b3eb>] ? kmem_cache_alloc+0xcb/0x160 [ 1558.858602] [<ffffffffa05a9443>] btrfs_drop_extents+0x73/0xa0 [btrfs] [ 1558.865457] [<ffffffffa059a1c0>] insert_reserved_file_extent.clone.0+0x80/0x2a0 [btrfs] [ 1558.873999] [<ffffffffa059640b>] ? start_transaction+0x3cb/0x450 [btrfs] [ 1558.881239] [<ffffffffa05a4bc9>] btrfs_finish_ordered_io+0x339/0x4d0 [btrfs] [ 1558.888608] [<ffffffffa05a4d75>] finish_ordered_fn+0x15/0x20 [btrfs] [ 1558.895352] [<ffffffffa05bfbd2>] worker_loop+0x1a2/0x400 [btrfs] [ 1558.901753] [<ffffffffa05bfa30>] ? check_pending_worker_creates+0xe0/0xe0 [btrfs] [ 1558.910128] [<ffffffffa05bfa30>] ? check_pending_worker_creates+0xe0/0xe0 [btrfs] [ 1558.918228] [<ffffffff81065b66>] kthread+0x96/0xa0 [ 1558.923339] [<ffffffff81495c24>] kernel_thread_helper+0x4/0x10 [ 1558.929821] [<ffffffff8148ca46>] ? retint_restore_args+0xe/0xe [ 1558.935920] [<ffffffff81065ad0>] ? __init_kthread_worker+0x40/0x40 [ 1558.942694] [<ffffffff81495c20>] ? gs_change+0xb/0xb [ 1558.948170] INFO: task flush-btrfs-21:31685 blocked for more than 120 seconds. [ 1558.955408] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1558.963274] flush-btrfs-21 D ffffffff8160b9c0 0 31685 2 0x00000000 [ 1558.970602] ffff8806074d3470 0000000000000046 0000000000000002 ffff8806074d3fd8 [ 1558.978333] ffff8806074d2010 ffff8806074d2000 ffff8806074d2000 ffff8806074d2000 [ 1558.986052] ffff8806074d3fd8 ffff8806074d2000 ffff8806245aae80 ffff8804cf048000 [ 1558.993744] Call Trace: [ 1558.996238] [<ffffffff8148b51d>] schedule+0x5d/0x60 [ 1559.001243] [<ffffffffa05c7423>] btrfs_tree_read_lock+0xf3/0x140 [btrfs] [ 1559.008057] [<ffffffff810660a0>] ? wake_up_bit+0x40/0x40 [ 1559.013554] [<ffffffffa056cf53>] btrfs_read_lock_root_node+0x23/0x50 [btrfs] [ 1559.020788] [<ffffffffa0575000>] btrfs_search_slot+0x1f0/0x730 [btrfs] [ 1559.027451] [<ffffffffa05aa4c7>] ? free_extent_map+0x87/0x90 [btrfs] [ 1559.033957] [<ffffffffa0588dbd>] btrfs_lookup_file_extent+0x3d/0x40 [btrfs] [ 1559.041062] [<ffffffffa05a80e1>] __btrfs_drop_extents+0x171/0xad0 [btrfs] [ 1559.047977] [<ffffffff8114b3eb>] ? kmem_cache_alloc+0xcb/0x160 [ 1559.053948] [<ffffffffa05a9443>] btrfs_drop_extents+0x73/0xa0 [btrfs] [ 1559.060698] [<ffffffffa059cc85>] cow_file_range_inline+0xe5/0x1c0 [btrfs] [ 1559.067807] [<ffffffffa059ced2>] cow_file_range+0x172/0x4a0 [btrfs] [ 1559.074296] [<ffffffffa059f27b>] run_delalloc_range+0x7b/0xa0 [btrfs] [ 1559.080967] [<ffffffffa05b406d>] __extent_writepage+0x22d/0x740 [btrfs] [ 1559.087868] [<ffffffffa0597757>] ? btrfs_add_delayed_iput+0x77/0xe0 [btrfs] [ 1559.095034] [<ffffffff810fcad8>] ? find_get_pages_tag+0x148/0x190 [ 1559.101329] [<ffffffffa05b4972>] extent_write_cache_pages.clone.3+0x242/0x3d0 [btrfs] [ 1559.109465] [<ffffffffa05b4b47>] extent_writepages+0x47/0x60 [btrfs] [ 1559.116008] [<ffffffffa05a2790>] ? btrfs_update_time+0xb0/0xb0 [btrfs] [ 1559.122716] [<ffffffffa0599987>] btrfs_writepages+0x27/0x30 [btrfs] [ 1559.129211] [<ffffffff81108a03>] do_writepages+0x23/0x40 [ 1559.134701] [<ffffffff8117f94b>] __writeback_single_inode+0x4b/0x180 [ 1559.141198] [<ffffffff81065f77>] ? bit_waitqueue+0x17/0xc0 [ 1559.147201] [<ffffffff811820f6>] writeback_sb_inodes+0x286/0x390 [ 1559.153365] [<ffffffff81182286>] __writeback_inodes_wb+0x86/0xd0 [ 1559.159509] [<ffffffff811824fb>] wb_writeback+0x18b/0x320 [ 1559.165082] [<ffffffff81182884>] wb_do_writeback+0x1f4/0x290 [ 1559.170988] [<ffffffff814897b2>] ? schedule_timeout+0x1d2/0x240 [ 1559.177249] [<ffffffff81182a03>] bdi_writeback_thread+0xe3/0x2c0 [ 1559.183685] [<ffffffff81182920>] ? wb_do_writeback+0x290/0x290 [ 1559.189740] [<ffffffff81182920>] ? wb_do_writeback+0x290/0x290 [ 1559.195878] [<ffffffff81065b66>] kthread+0x96/0xa0 [ 1559.200874] [<ffffffff81495c24>] kernel_thread_helper+0x4/0x10 [ 1559.207540] [<ffffffff8148ca46>] ? retint_restore_args+0xe/0xe [ 1559.213718] [<ffffffff81065ad0>] ? __init_kthread_worker+0x40/0x40 [ 1559.220087] [<ffffffff81495c20>] ? gs_change+0xb/0xb [ 1559.226637] INFO: task ceph-osd:38318 blocked for more than 120 seconds. [ 1559.233338] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1559.241157] ceph-osd D 000000000000000e 0 38318 1 0x00000000 [ 1559.248729] ffff880320e499c8 0000000000000086 0000000000000002 ffff880320e49fd8 [ 1559.256282] ffff880320e48010 ffff880320e48000 ffff880320e48000 ffff880320e48000 [ 1559.263784] ffff880320e49fd8 ffff880320e48000 ffff88032a7fc5c0 ffff880320e40000 [ 1559.271254] Call Trace: [ 1559.273783] [<ffffffff8148b51d>] schedule+0x5d/0x60 [ 1559.278805] [<ffffffffa05c7423>] btrfs_tree_read_lock+0xf3/0x140 [btrfs] [ 1559.285636] [<ffffffff810660a0>] ? wake_up_bit+0x40/0x40 [ 1559.291082] [<ffffffffa056cf53>] btrfs_read_lock_root_node+0x23/0x50 [btrfs] [ 1559.298278] [<ffffffffa0575000>] btrfs_search_slot+0x1f0/0x730 [btrfs] [ 1559.304910] [<ffffffffa05886cd>] btrfs_lookup_dir_item+0x7d/0xd0 [btrfs] [ 1559.311720] [<ffffffffa059fd34>] btrfs_lookup_dentry+0xe4/0x360 [btrfs] [ 1559.318439] [<ffffffffa059ffcd>] btrfs_lookup+0x1d/0x70 [btrfs] [ 1559.324543] [<ffffffff811622db>] ? lookup_dcache+0x9b/0xd0 [ 1559.330117] [<ffffffff81161ce6>] lookup_real+0x36/0x60 [ 1559.335352] [<ffffffff81162348>] __lookup_hash+0x38/0x50 [ 1559.340761] [<ffffffff811623c2>] lookup_slow+0x62/0xe0 [ 1559.345990] [<ffffffff81165cf7>] path_lookupat+0xf7/0x6c0 [ 1559.351494] [<ffffffff8114b36f>] ? kmem_cache_alloc+0x4f/0x160 [ 1559.357465] [<ffffffff811662f1>] do_path_lookup+0x31/0xd0 [ 1559.362990] [<ffffffff811671ac>] user_path_at_empty+0x6c/0xb0 [ 1559.368871] [<ffffffff8115c399>] ? cp_new_stat+0x129/0x140 [ 1559.374551] [<ffffffff81167201>] user_path_at+0x11/0x20 [ 1559.379855] [<ffffffff8115c0b7>] vfs_fstatat+0x47/0x80 [ 1559.385077] [<ffffffff8115c17b>] vfs_stat+0x1b/0x20 [ 1559.390086] [<ffffffff8115c474>] sys_newstat+0x24/0x50 [ 1559.395415] [<ffffffff8125014e>] ? trace_hardirqs_on_thunk+0x3a/0x3c [ 1559.401871] [<ffffffff81162ea3>] ? putname+0x33/0x50 [ 1559.406939] [<ffffffff81494b12>] system_call_fastpath+0x16/0x1b [ 1559.412949] INFO: task ceph-osd:38320 blocked for more than 120 seconds. [ 1559.419648] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1559.427471] ceph-osd D ffffffff8160b9c0 0 38320 1 0x00000000 [ 1559.434574] ffff880320e4d578 0000000000000086 ffff880320e4d538 ffff880320e4dfd8 [ 1559.442059] ffff880320e4c010 ffff880320e4c000 ffff880320e4c000 ffff880320e4c000 [ 1559.449571] ffff880320e4dfd8 ffff880320e4c000 ffff8806244d5d00 ffff880320e42e80 [ 1559.457108] Call Trace: [ 1559.459591] [<ffffffff8148b51d>] schedule+0x5d/0x60 [ 1559.464589] [<ffffffff8148b817>] schedule_preempt_disabled+0x27/0x40 [ 1559.471055] [<ffffffff81489ce5>] __mutex_lock_common+0x205/0x360 [ 1559.477175] [<ffffffff81489e5b>] __mutex_lock_slowpath+0x1b/0x20 [ 1559.483291] [<ffffffff81489e91>] mutex_lock+0x31/0x50 [ 1559.488489] [<ffffffffa057c6f4>] do_chunk_alloc+0x1c4/0x300 [btrfs] [ 1559.494872] [<ffffffffa05855cc>] find_free_extent+0x9fc/0xb30 [btrfs] [ 1559.501439] [<ffffffffa0585782>] btrfs_reserve_extent+0x82/0x190 [btrfs] [ 1559.508279] [<ffffffffa0585915>] btrfs_alloc_free_block+0x85/0x230 [btrfs] [ 1559.515272] [<ffffffffa05b1a82>] ? clear_extent_bit+0x352/0x3d0 [btrfs] [ 1559.522000] [<ffffffffa057188a>] __btrfs_cow_block+0x14a/0x4b0 [btrfs] [ 1559.528611] [<ffffffff8148c70e>] ? _raw_spin_unlock+0xe/0x20 [ 1559.534376] [<ffffffffa057228d>] btrfs_cow_block+0x18d/0x230 [btrfs] [ 1559.540852] [<ffffffffa0575170>] btrfs_search_slot+0x360/0x730 [btrfs] [ 1559.547493] [<ffffffffa0575f7d>] btrfs_insert_empty_items+0x8d/0x100 [btrfs] [ 1559.554637] [<ffffffffa0578e2b>] ? can_overcommit+0x8b/0xf0 [btrfs] [ 1559.560995] [<ffffffffa05888c3>] insert_with_overflow+0x43/0x120 [btrfs] [ 1559.567782] [<ffffffffa0588c66>] btrfs_insert_xattr_item+0xb6/0x1d0 [btrfs] [ 1559.574821] [<ffffffff8114b3eb>] ? kmem_cache_alloc+0xcb/0x160 [ 1559.580754] [<ffffffffa05ab8aa>] do_setxattr+0x17a/0x240 [btrfs] [ 1559.586857] [<ffffffffa05ab9fb>] __btrfs_setxattr+0x8b/0x110 [btrfs] [ 1559.593309] [<ffffffffa05abc97>] btrfs_setxattr+0xa7/0xc0 [btrfs] [ 1559.599485] [<ffffffff8117b18b>] __vfs_setxattr_noperm+0x7b/0x150 [ 1559.605665] [<ffffffff8117b2ec>] vfs_setxattr+0x8c/0xc0 [ 1559.610975] [<ffffffff8117b4be>] setxattr+0x19e/0x210 [ 1559.616118] [<ffffffff81148510>] ? kmem_cache_free+0x20/0x120 [ 1559.621941] [<ffffffff81162ea3>] ? putname+0x33/0x50 [ 1559.626985] [<ffffffff811590f3>] ? __sb_start_write+0xe3/0x130 [ 1559.632903] [<ffffffff81158c40>] ? __fput+0x280/0x290 [ 1559.638039] [<ffffffff8117b72f>] sys_setxattr+0x6f/0xb0 [ 1559.643341] [<ffffffff8125014e>] ? trace_hardirqs_on_thunk+0x3a/0x3c [ 1559.649778] [<ffffffff81494b12>] system_call_fastpath+0x16/0x1b [ 1559.655773] INFO: task ceph-osd:38324 blocked for more than 120 seconds. [ 1559.662459] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1559.670273] ceph-osd D ffffffff8160b9c0 0 38324 1 0x00000000 [ 1559.677362] ffff880320e6f258 0000000000000086 0000000000000000 ffff880320e6ffd8 [ 1559.684838] ffff880320e6e010 ffff880320e6e000 ffff880320e6e000 ffff880320e6e000 [ 1559.692325] ffff880320e6ffd8 ffff880320e6e000 ffff8806245f1740 ffff880320e71740 [ 1559.699807] Call Trace: [ 1559.702267] [<ffffffff8148b51d>] schedule+0x5d/0x60 [ 1559.707233] [<ffffffff8148b817>] schedule_preempt_disabled+0x27/0x40 [ 1559.713661] [<ffffffff81489ce5>] __mutex_lock_common+0x205/0x360 [ 1559.719745] [<ffffffff81489e5b>] __mutex_lock_slowpath+0x1b/0x20 [ 1559.725831] [<ffffffff81489e91>] mutex_lock+0x31/0x50 [ 1559.730979] [<ffffffffa057c6b1>] do_chunk_alloc+0x181/0x300 [btrfs] [ 1559.737351] [<ffffffffa05855cc>] find_free_extent+0x9fc/0xb30 [btrfs] [ 1559.743896] [<ffffffffa0585782>] btrfs_reserve_extent+0x82/0x190 [btrfs] [ 1559.750686] [<ffffffffa0585915>] btrfs_alloc_free_block+0x85/0x230 [btrfs] [ 1559.757663] [<ffffffffa056c8ac>] ? generic_bin_search+0xac/0x170 [btrfs] [ 1559.764477] [<ffffffffa057188a>] __btrfs_cow_block+0x14a/0x4b0 [btrfs] [ 1559.771148] [<ffffffffa057228d>] btrfs_cow_block+0x18d/0x230 [btrfs] [ 1559.777604] [<ffffffffa0575170>] btrfs_search_slot+0x360/0x730 [btrfs] [ 1559.784229] [<ffffffffa0575f7d>] btrfs_insert_empty_items+0x8d/0x100 [btrfs] [ 1559.791351] [<ffffffff8114b3eb>] ? kmem_cache_alloc+0xcb/0x160 [ 1559.797288] [<ffffffffa05b9735>] btrfs_alloc_dev_extent+0xb5/0x190 [btrfs] [ 1559.804267] [<ffffffffa05bd807>] __btrfs_alloc_chunk+0x6a7/0x750 [btrfs] [ 1559.811076] [<ffffffffa05b0265>] ? free_extent_buffer+0x75/0x90 [btrfs] [ 1559.817842] [<ffffffffa05bd90e>] btrfs_alloc_chunk+0x5e/0x90 [btrfs] [ 1559.824324] [<ffffffffa057aa01>] ? check_system_chunk+0x71/0x130 [btrfs] [ 1559.831192] [<ffffffffa057c7be>] do_chunk_alloc+0x28e/0x300 [btrfs] [ 1559.837588] [<ffffffffa05855cc>] find_free_extent+0x9fc/0xb30 [btrfs] [ 1559.844132] [<ffffffffa0585782>] btrfs_reserve_extent+0x82/0x190 [btrfs] [ 1559.850950] [<ffffffffa059cff6>] cow_file_range+0x296/0x4a0 [btrfs] [ 1559.857350] [<ffffffffa059f27b>] run_delalloc_range+0x7b/0xa0 [btrfs] [ 1559.863935] [<ffffffffa05b406d>] __extent_writepage+0x22d/0x740 [btrfs] [ 1559.870642] [<ffffffff810fcad8>] ? find_get_pages_tag+0x148/0x190 [ 1559.876870] [<ffffffffa05b4972>] extent_write_cache_pages.clone.3+0x242/0x3d0 [btrfs] [ 1559.884865] [<ffffffffa05b4b47>] extent_writepages+0x47/0x60 [btrfs] [ 1559.891351] [<ffffffffa05a2790>] ? btrfs_update_time+0xb0/0xb0 [btrfs] [ 1559.897991] [<ffffffffa0599987>] btrfs_writepages+0x27/0x30 [btrfs] [ 1559.904350] [<ffffffff81108a03>] do_writepages+0x23/0x40 [ 1559.909793] [<ffffffff810fd79e>] __filemap_fdatawrite_range+0x4e/0x50 [ 1559.916340] [<ffffffff810fda53>] filemap_fdatawrite_range+0x13/0x20 [ 1559.922701] [<ffffffff811863fc>] sys_sync_file_range+0x10c/0x170 [ 1559.928825] [<ffffffff81494b12>] system_call_fastpath+0x16/0x1b [ 1559.934837] INFO: task ceph-osd:38834 blocked for more than 120 seconds. [ 1559.941541] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1559.949360] ceph-osd D 0000000000000013 0 38834 1 0x00000000 [ 1559.956459] ffff880332339bc8 0000000000000086 ffff880c1ae97878 ffff880332339fd8 [ 1559.963954] ffff880332338010 ffff880332338000 ffff880332338000 ffff880332338000 [ 1559.971478] ffff880332339fd8 ffff880332338000 ffff88051c1c0000 ffff88038f535d00 [ 1559.979005] Call Trace: [ 1559.981465] [<ffffffff8148b51d>] schedule+0x5d/0x60 [ 1559.986421] [<ffffffff8148b817>] schedule_preempt_disabled+0x27/0x40 [ 1559.992869] [<ffffffff81489ce5>] __mutex_lock_common+0x205/0x360 [ 1559.998963] [<ffffffff81489e5b>] __mutex_lock_slowpath+0x1b/0x20 [ 1560.005071] [<ffffffff81489e91>] mutex_lock+0x31/0x50 [ 1560.010211] [<ffffffff811623b4>] lookup_slow+0x54/0xe0 [ 1560.015443] [<ffffffff81165cf7>] path_lookupat+0xf7/0x6c0 [ 1560.020956] [<ffffffff8114b36f>] ? kmem_cache_alloc+0x4f/0x160 [ 1560.026904] [<ffffffff811662f1>] do_path_lookup+0x31/0xd0 [ 1560.032400] [<ffffffff810076f9>] ? emulate_vsyscall+0x1d9/0x360 [ 1560.038403] [<ffffffff811671ac>] user_path_at_empty+0x6c/0xb0 [ 1560.044240] [<ffffffff81034843>] ? bad_area_nosemaphore+0x13/0x20 [ 1560.050414] [<ffffffff81490338>] ? do_page_fault+0x228/0x4a0 [ 1560.056156] [<ffffffff81167201>] user_path_at+0x11/0x20 [ 1560.061482] [<ffffffff8115c0b7>] vfs_fstatat+0x47/0x80 [ 1560.066719] [<ffffffff8115c17b>] vfs_stat+0x1b/0x20 [ 1560.071690] [<ffffffff8115c474>] sys_newstat+0x24/0x50 [ 1560.076948] [<ffffffff8125018a>] ? trace_hardirqs_off_thunk+0x3a/0x6c [ 1560.083470] [<ffffffff8125014e>] ? trace_hardirqs_on_thunk+0x3a/0x3c [ 1560.089898] [<ffffffff81494b12>] system_call_fastpath+0x16/0x1b [ 1560.095893] INFO: task ceph-osd:38835 blocked for more than 120 seconds. [ 1560.102581] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1560.110393] ceph-osd D 000000000000000b 0 38835 1 0x00000000 [ 1560.117482] ffff8803672d5bc8 0000000000000086 ffff880c1ae97878 ffff8803672d5fd8 [ 1560.124969] ffff8803672d4010 ffff8803672d4000 ffff8803672d4000 ffff8803672d4000 [ 1560.132457] ffff8803672d5fd8 ffff8803672d4000 ffff88060f092e80 ffff8803672d8000 [ 1560.139940] Call Trace: [ 1560.142401] [<ffffffff8148b51d>] schedule+0x5d/0x60 [ 1560.147368] [<ffffffff8148b817>] schedule_preempt_disabled+0x27/0x40 [ 1560.153810] [<ffffffff81489ce5>] __mutex_lock_common+0x205/0x360 [ 1560.159929] [<ffffffff81489e5b>] __mutex_lock_slowpath+0x1b/0x20 [ 1560.166015] [<ffffffff81489e91>] mutex_lock+0x31/0x50 [ 1560.171153] [<ffffffff811623b4>] lookup_slow+0x54/0xe0 [ 1560.176374] [<ffffffff81165cf7>] path_lookupat+0xf7/0x6c0 [ 1560.181851] [<ffffffff8114b36f>] ? kmem_cache_alloc+0x4f/0x160 [ 1560.187767] [<ffffffff811662f1>] do_path_lookup+0x31/0xd0 [ 1560.193266] [<ffffffff811671ac>] user_path_at_empty+0x6c/0xb0 [ 1560.199099] [<ffffffff81167201>] user_path_at+0x11/0x20 [ 1560.204415] [<ffffffff8115c0b7>] vfs_fstatat+0x47/0x80 [ 1560.209640] [<ffffffff8115c17b>] vfs_stat+0x1b/0x20 [ 1560.214602] [<ffffffff8115c474>] sys_newstat+0x24/0x50 [ 1560.219819] [<ffffffff8125014e>] ? trace_hardirqs_on_thunk+0x3a/0x3c [ 1560.226255] [<ffffffff81494b12>] system_call_fastpath+0x16/0x1b [ 1560.232257] INFO: task ceph-osd:38841 blocked for more than 120 seconds. [ 1560.238945] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1560.246783] ceph-osd D ffffffff8160b9c0 0 38841 1 0x00000000 [ 1560.253897] ffff880610037bc8 0000000000000086 ffff880c1ae97878 ffff880610037fd8 [ 1560.261372] ffff880610036010 ffff880610036000 ffff880610036000 ffff880610036000 [ 1560.268866] ffff880610037fd8 ffff880610036000 ffff8806245cae80 ffff8803672dc5c0 [ 1560.276335] Call Trace: [ 1560.278793] [<ffffffff8148b51d>] schedule+0x5d/0x60 [ 1560.283763] [<ffffffff8148b817>] schedule_preempt_disabled+0x27/0x40 [ 1560.290204] [<ffffffff81489ce5>] __mutex_lock_common+0x205/0x360 [ 1560.296293] [<ffffffff81489e5b>] __mutex_lock_slowpath+0x1b/0x20 [ 1560.302384] [<ffffffff81489e91>] mutex_lock+0x31/0x50 [ 1560.307518] [<ffffffff811623b4>] lookup_slow+0x54/0xe0 [ 1560.312732] [<ffffffff81165cf7>] path_lookupat+0xf7/0x6c0 [ 1560.318226] [<ffffffff81162827>] ? getname_flags+0x37/0x100 [ 1560.323886] [<ffffffff8114b3bc>] ? kmem_cache_alloc+0x9c/0x160 [ 1560.329815] [<ffffffff811662f1>] do_path_lookup+0x31/0xd0 [ 1560.335304] [<ffffffff811671ac>] user_path_at_empty+0x6c/0xb0 [ 1560.341136] [<ffffffff81167201>] user_path_at+0x11/0x20 [ 1560.346456] [<ffffffff8115c0b7>] vfs_fstatat+0x47/0x80 [ 1560.351694] [<ffffffff8115c17b>] vfs_stat+0x1b/0x20 [ 1560.356686] [<ffffffff8115c474>] sys_newstat+0x24/0x50 [ 1560.361918] [<ffffffff8125014e>] ? trace_hardirqs_on_thunk+0x3a/0x3c [ 1560.368371] [<ffffffff81494b12>] system_call_fastpath+0x16/0x1b [ 1560.374385] INFO: task ceph-osd:38844 blocked for more than 120 seconds. [ 1560.381082] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1560.388911] ceph-osd D ffffffff8160b9c0 0 38844 1 0x00000000 [ 1560.396023] ffff88061003dbc8 0000000000000086 ffff880c1ae97878 ffff88061003dfd8 [ 1560.403509] ffff88061003c010 ffff88061003c000 ffff88061003c000 ffff88061003c000 [ 1560.411003] ffff88061003dfd8 ffff88061003c000 ffff8806245c8000 ffff8803672ddd00 [ 1560.418522] Call Trace: [ 1560.420995] [<ffffffff8148b51d>] schedule+0x5d/0x60 [ 1560.425972] [<ffffffff8148b817>] schedule_preempt_disabled+0x27/0x40 [ 1560.432421] [<ffffffff81489ce5>] __mutex_lock_common+0x205/0x360 [ 1560.438545] [<ffffffff81489e5b>] __mutex_lock_slowpath+0x1b/0x20 [ 1560.444647] [<ffffffff81489e91>] mutex_lock+0x31/0x50 [ 1560.449801] [<ffffffff811623b4>] lookup_slow+0x54/0xe0 [ 1560.455039] [<ffffffff81165cf7>] path_lookupat+0xf7/0x6c0 [ 1560.460542] [<ffffffff8114b36f>] ? kmem_cache_alloc+0x4f/0x160 [ 1560.466463] [<ffffffff811662f1>] do_path_lookup+0x31/0xd0 [ 1560.471953] [<ffffffff811671ac>] user_path_at_empty+0x6c/0xb0 [ 1560.477788] [<ffffffff81167201>] user_path_at+0x11/0x20 [ 1560.483098] [<ffffffff8115c0b7>] vfs_fstatat+0x47/0x80 [ 1560.488330] [<ffffffff8115c17b>] vfs_stat+0x1b/0x20 [ 1560.493296] [<ffffffff8115c474>] sys_newstat+0x24/0x50 [ 1560.498528] [<ffffffff8125014e>] ? trace_hardirqs_on_thunk+0x3a/0x3c [ 1560.504962] [<ffffffff81494b12>] system_call_fastpath+0x16/0x1b [ 1560.510961] INFO: task ceph-osd:38847 blocked for more than 120 seconds. [ 1560.517647] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1560.525458] ceph-osd D 000000000000000b 0 38847 1 0x00000000 [ 1560.532555] ffff88061003fbc8 0000000000000086 ffff880c1ae97878 ffff88061003ffd8 [ 1560.540041] ffff88061003e010 ffff88061003e000 ffff88061003e000 ffff88061003e000 [ 1560.547516] ffff88061003ffd8 ffff88061003e000 ffff880347818000 ffff8803672e0000 [ 1560.554994] Call Trace: [ 1560.557455] [<ffffffff8148b51d>] schedule+0x5d/0x60 [ 1560.562423] [<ffffffff8148b817>] schedule_preempt_disabled+0x27/0x40 [ 1560.568861] [<ffffffff81489ce5>] __mutex_lock_common+0x205/0x360 [ 1560.574962] [<ffffffff81489e5b>] __mutex_lock_slowpath+0x1b/0x20 [ 1560.581066] [<ffffffff81489e91>] mutex_lock+0x31/0x50 [ 1560.586229] [<ffffffff811623b4>] lookup_slow+0x54/0xe0 [ 1560.591482] [<ffffffff81165cf7>] path_lookupat+0xf7/0x6c0 [ 1560.596971] [<ffffffff8114b36f>] ? kmem_cache_alloc+0x4f/0x160 [ 1560.602889] [<ffffffff811662f1>] do_path_lookup+0x31/0xd0 [ 1560.608379] [<ffffffff811671ac>] user_path_at_empty+0x6c/0xb0 [ 1560.614217] [<ffffffff81167201>] user_path_at+0x11/0x20 [ 1560.619536] [<ffffffff8115c0b7>] vfs_fstatat+0x47/0x80 [ 1560.624772] [<ffffffff8115c17b>] vfs_stat+0x1b/0x20 [ 1560.629749] [<ffffffff8115c474>] sys_newstat+0x24/0x50 [ 1560.634973] [<ffffffff8125014e>] ? trace_hardirqs_on_thunk+0x3a/0x3c [ 1560.641412] [<ffffffff81494b12>] system_call_fastpath+0x16/0x1b Thanks -- Jim -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html