Roman
2014-Sep-16 06:56 UTC
[Gluster-users] INFO: task "####" blocked for more than 120 seconds.
Hi This morning we had a VM crash and these logs (see attc) No logs on gluster servers nor virtual host where this is glusterfs mount is attached. Any ideas? -- Best regards, Roman. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140916/9c95b30b/attachment.html> -------------- next part -------------- Sep 16 06:36:58 munin kernel: [1025760.825197] jbd2/vda1-8 D ffff88003fc93780 0 179 2 0x00000000 Sep 16 06:36:58 munin kernel: [1025760.825204] ffff88003bb30740 0000000000000046 0000000000000000 ffff88003e216180 Sep 16 06:36:58 munin kernel: [1025760.825210] 0000000000013780 ffff880036d77fd8 ffff880036d77fd8 ffff88003bb30740 Sep 16 06:36:58 munin kernel: [1025760.825214] ffff880036d76000 0000000181066311 ffff88003e1d5c20 ffff88003fc93fd0 Sep 16 06:36:58 munin kernel: [1025760.825218] Call Trace: Sep 16 06:36:58 munin kernel: [1025760.825262] [<ffffffff8111da09>] ? wait_on_buffer+0x28/0x28 Sep 16 06:36:58 munin kernel: [1025760.825282] [<ffffffff8134f399>] ? io_schedule+0x59/0x71 Sep 16 06:36:58 munin kernel: [1025760.825286] [<ffffffff8111da0f>] ? sleep_on_buffer+0x6/0xa Sep 16 06:36:58 munin kernel: [1025760.825289] [<ffffffff8134f7dc>] ? __wait_on_bit+0x3e/0x71 Sep 16 06:36:58 munin kernel: [1025760.825292] [<ffffffff8134f87e>] ? out_of_line_wait_on_bit+0x6f/0x78 Sep 16 06:36:58 munin kernel: [1025760.825295] [<ffffffff8111da09>] ? wait_on_buffer+0x28/0x28 Sep 16 06:36:58 munin kernel: [1025760.825311] [<ffffffff8105fd7d>] ? autoremove_wake_function+0x2a/0x2a Sep 16 06:36:58 munin kernel: [1025760.825389] [<ffffffffa0115a4c>] ? jbd2_journal_commit_transaction+0xbeb/0x10bf [jbd2] Sep 16 06:36:58 munin kernel: [1025760.825403] [<ffffffff8100d02f>] ? load_TLS+0x7/0xa Sep 16 06:36:58 munin kernel: [1025760.825407] [<ffffffff8100d69e>] ? __switch_to+0x133/0x258 Sep 16 06:36:58 munin kernel: [1025760.825413] [<ffffffffa0119156>] ? kjournald2+0xc0/0x20a [jbd2] Sep 16 06:36:58 munin kernel: [1025760.825417] [<ffffffff8105fd53>] ? add_wait_queue+0x3c/0x3c Sep 16 06:36:58 munin kernel: [1025760.825422] [<ffffffffa0119096>] ? commit_timeout+0x5/0x5 [jbd2] Sep 16 06:36:58 munin kernel: [1025760.825426] [<ffffffff8105f701>] ? kthread+0x76/0x7e Sep 16 06:36:58 munin kernel: [1025760.825436] [<ffffffff813575b4>] ? kernel_thread_helper+0x4/0x10 Sep 16 06:36:58 munin kernel: [1025760.825440] [<ffffffff8105f68b>] ? kthread_worker_fn+0x139/0x139 Sep 16 06:36:58 munin kernel: [1025760.825444] [<ffffffff813575b0>] ? gs_change+0x13/0x13 Sep 16 06:36:58 munin kernel: [1025760.826398] flush-254:0 D ffff88003fc13780 0 2672 2 0x00000000 Sep 16 06:36:58 munin kernel: [1025760.826410] ffff880037075740 0000000000000046 0000000000000000 ffffffff8160d020 Sep 16 06:36:58 munin kernel: [1025760.826421] 0000000000013780 ffff88003b955fd8 ffff88003b955fd8 ffff880037075740 Sep 16 06:36:58 munin kernel: [1025760.826424] ffff88003b955770 000000013b955770 ffff88003d424c90 ffff88003fc13fd0 Sep 16 06:36:58 munin kernel: [1025760.826428] Call Trace: Sep 16 06:36:58 munin kernel: [1025760.826432] [<ffffffff8134f399>] ? io_schedule+0x59/0x71 Sep 16 06:36:58 munin kernel: [1025760.826447] [<ffffffff8119a2fc>] ? get_request_wait+0x105/0x18f Sep 16 06:36:58 munin kernel: [1025760.826452] [<ffffffff8105fd53>] ? add_wait_queue+0x3c/0x3c Sep 16 06:36:58 munin kernel: [1025760.826456] [<ffffffff8119b25b>] ? blk_queue_bio+0x17f/0x28c Sep 16 06:36:58 munin kernel: [1025760.826459] [<ffffffff81199d88>] ? generic_make_request+0x90/0xcf Sep 16 06:36:58 munin kernel: [1025760.826463] [<ffffffff81199e9a>] ? submit_bio+0xd3/0xf1 Sep 16 06:36:58 munin kernel: [1025760.826473] [<ffffffffa012dfad>] ? ext4_io_submit+0x21/0x4a [ext4] Sep 16 06:36:58 munin kernel: [1025760.826481] [<ffffffffa012ae52>] ? mpage_da_submit_io+0x359/0x36f [ext4] Sep 16 06:36:58 munin kernel: [1025760.826488] [<ffffffffa012cac7>] ? mpage_da_map_and_submit+0x2aa/0x2f9 [ext4] Sep 16 06:36:58 munin kernel: [1025760.826495] [<ffffffffa012bd80>] ? ext4_mark_inode_dirty+0x1af/0x1da [ext4] Sep 16 06:36:58 munin kernel: [1025760.826502] [<ffffffffa012cb00>] ? mpage_da_map_and_submit+0x2e3/0x2f9 [ext4] Sep 16 06:36:58 munin kernel: [1025760.826509] [<ffffffffa012d0e7>] ? ext4_da_writepages+0x239/0x45d [ext4] Sep 16 06:36:58 munin kernel: [1025760.826519] [<ffffffffa0142724>] ? ext4_journal_start_sb+0x139/0x14f [ext4] Sep 16 06:36:58 munin kernel: [1025760.826526] [<ffffffffa012d172>] ? ext4_da_writepages+0x2c4/0x45d [ext4] Sep 16 06:36:58 munin kernel: [1025760.826533] [<ffffffff81118ad3>] ? writeback_single_inode+0x11d/0x2cc Sep 16 06:36:58 munin kernel: [1025760.826537] [<ffffffff81118f53>] ? writeback_sb_inodes+0x16b/0x204 Sep 16 06:36:58 munin kernel: [1025760.826541] [<ffffffff81119059>] ? __writeback_inodes_wb+0x6d/0xab Sep 16 06:36:58 munin kernel: [1025760.826545] [<ffffffff811191bf>] ? wb_writeback+0x128/0x21f Sep 16 06:36:58 munin kernel: [1025760.826553] [<ffffffff8107116d>] ? arch_local_irq_save+0x11/0x17 Sep 16 06:36:58 munin kernel: [1025760.826558] [<ffffffff81119676>] ? wb_do_writeback+0x146/0x1a8 Sep 16 06:36:58 munin kernel: [1025760.826562] [<ffffffff8111975d>] ? bdi_writeback_thread+0x85/0x1e6 Sep 16 06:36:58 munin kernel: [1025760.826565] [<ffffffff811196d8>] ? wb_do_writeback+0x1a8/0x1a8 Sep 16 06:36:58 munin kernel: [1025760.826569] [<ffffffff8105f701>] ? kthread+0x76/0x7e Sep 16 06:36:58 munin kernel: [1025760.826573] [<ffffffff813575b4>] ? kernel_thread_helper+0x4/0x10 Sep 16 06:36:58 munin kernel: [1025760.826576] [<ffffffff8105f68b>] ? kthread_worker_fn+0x139/0x139 Sep 16 06:36:58 munin kernel: [1025760.826579] [<ffffffff813575b0>] ? gs_change+0x13/0x13 Sep 16 06:36:58 munin kernel: [1025760.827533] munin-graph D ffff88003fc13780 0 29913 29580 0x00000000 Sep 16 06:36:58 munin kernel: [1025760.827537] ffff88003d3a8140 0000000000000086 0000000000000000 ffffffff8160d020 Sep 16 06:36:58 munin kernel: [1025760.827540] 0000000000013780 ffff880036d2bfd8 ffff880036d2bfd8 ffff88003d3a8140 Sep 16 06:36:58 munin kernel: [1025760.827547] 0000000000000246 0000000181350461 ffffffffa016f508 ffff88002c9a4558 Sep 16 06:36:58 munin kernel: [1025760.827550] Call Trace: Sep 16 06:36:58 munin kernel: [1025760.827559] [<ffffffffa012dabc>] ? ext4_ioend_wait+0x82/0x98 [ext4] Sep 16 06:36:58 munin kernel: [1025760.827563] [<ffffffff8105fd53>] ? add_wait_queue+0x3c/0x3c Sep 16 06:36:58 munin kernel: [1025760.827571] [<ffffffffa012d344>] ? ext4_evict_inode+0x39/0x2af [ext4] Sep 16 06:36:58 munin kernel: [1025760.827577] [<ffffffffa012d30b>] ? ext4_da_writepages+0x45d/0x45d [ext4] Sep 16 06:36:58 munin kernel: [1025760.827589] [<ffffffff8110d748>] ? evict+0x96/0x148 Sep 16 06:36:58 munin kernel: [1025760.827593] [<ffffffff8110ab10>] ? dentry_kill+0x112/0x12e Sep 16 06:36:58 munin kernel: [1025760.827595] [<ffffffff8110ad62>] ? dput+0xe2/0xee Sep 16 06:36:58 munin kernel: [1025760.827602] [<ffffffff81106c71>] ? sys_renameat+0x1af/0x206 Sep 16 06:36:58 munin kernel: [1025760.827608] [<ffffffff81126b2a>] ? fsnotify+0x1e6/0x20c Sep 16 06:36:58 munin kernel: [1025760.827619] [<ffffffff8102b2d2>] ? kvm_clock_read+0x17/0x1a Sep 16 06:36:58 munin kernel: [1025760.827625] [<ffffffff81066311>] ? timekeeping_get_ns+0xd/0x2a Sep 16 06:36:58 munin kernel: [1025760.827628] [<ffffffff810668f5>] ? getnstimeofday+0x4a/0x7b Sep 16 06:36:58 munin kernel: [1025760.827631] [<ffffffff81066955>] ? do_gettimeofday+0x10/0x30 Sep 16 06:36:58 munin kernel: [1025760.827635] [<ffffffff81355452>] ? system_call_fastpath+0x16/0x1b Sep 16 06:38:58 munin kernel: [1025880.825070] jbd2/vda1-8 D ffff88003fc93780 0 179 2 0x00000000 Sep 16 06:38:58 munin kernel: [1025880.825076] ffff88003bb30740 0000000000000046 0000000000000000 ffff88003e216180 Sep 16 06:38:58 munin kernel: [1025880.825081] 0000000000013780 ffff880036d77fd8 ffff880036d77fd8 ffff88003bb30740 Sep 16 06:38:58 munin kernel: [1025880.825084] ffff880036d76000 0000000181066311 ffff88003e1d5c20 ffff88003fc93fd0 Sep 16 06:38:58 munin kernel: [1025880.825089] Call Trace: Sep 16 06:38:58 munin kernel: [1025880.825102] [<ffffffff8111da09>] ? wait_on_buffer+0x28/0x28 Sep 16 06:38:58 munin kernel: [1025880.825117] [<ffffffff8134f399>] ? io_schedule+0x59/0x71 Sep 16 06:38:58 munin kernel: [1025880.825120] [<ffffffff8111da0f>] ? sleep_on_buffer+0x6/0xa Sep 16 06:38:58 munin kernel: [1025880.825123] [<ffffffff8134f7dc>] ? __wait_on_bit+0x3e/0x71 Sep 16 06:38:58 munin kernel: [1025880.825126] [<ffffffff8134f87e>] ? out_of_line_wait_on_bit+0x6f/0x78 Sep 16 06:38:58 munin kernel: [1025880.825130] [<ffffffff8111da09>] ? wait_on_buffer+0x28/0x28 Sep 16 06:38:58 munin kernel: [1025880.825137] [<ffffffff8105fd7d>] ? autoremove_wake_function+0x2a/0x2a Sep 16 06:38:58 munin kernel: [1025880.825158] [<ffffffffa0115a4c>] ? jbd2_journal_commit_transaction+0xbeb/0x10bf [jbd2] Sep 16 06:38:58 munin kernel: [1025880.825165] [<ffffffff8100d02f>] ? load_TLS+0x7/0xa Sep 16 06:38:58 munin kernel: [1025880.825168] [<ffffffff8100d69e>] ? __switch_to+0x133/0x258 Sep 16 06:38:58 munin kernel: [1025880.825174] [<ffffffffa0119156>] ? kjournald2+0xc0/0x20a [jbd2] Sep 16 06:38:58 munin kernel: [1025880.825178] [<ffffffff8105fd53>] ? add_wait_queue+0x3c/0x3c Sep 16 06:38:58 munin kernel: [1025880.825183] [<ffffffffa0119096>] ? commit_timeout+0x5/0x5 [jbd2] Sep 16 06:38:58 munin kernel: [1025880.825187] [<ffffffff8105f701>] ? kthread+0x76/0x7e Sep 16 06:38:58 munin kernel: [1025880.825193] [<ffffffff813575b4>] ? kernel_thread_helper+0x4/0x10 Sep 16 06:38:58 munin kernel: [1025880.825197] [<ffffffff8105f68b>] ? kthread_worker_fn+0x139/0x139 Sep 16 06:38:58 munin kernel: [1025880.825200] [<ffffffff813575b0>] ? gs_change+0x13/0x13 Sep 16 06:38:58 munin kernel: [1025880.826146] flush-254:0 D ffff88003fc13780 0 2672 2 0x00000000 Sep 16 06:38:58 munin kernel: [1025880.826150] ffff880037075740 0000000000000046 0000000000000000 ffffffff8160d020 Sep 16 06:38:58 munin kernel: [1025880.826154] 0000000000013780 ffff88003b955fd8 ffff88003b955fd8 ffff880037075740 Sep 16 06:38:58 munin kernel: [1025880.826157] ffff88003b955770 000000013b955770 ffff88003d424c90 ffff88003fc13fd0 Sep 16 06:38:58 munin kernel: [1025880.826161] Call Trace: Sep 16 06:38:58 munin kernel: [1025880.826165] [<ffffffff8134f399>] ? io_schedule+0x59/0x71 Sep 16 06:38:58 munin kernel: [1025880.826171] [<ffffffff8119a2fc>] ? get_request_wait+0x105/0x18f Sep 16 06:38:58 munin kernel: [1025880.826175] [<ffffffff8105fd53>] ? add_wait_queue+0x3c/0x3c Sep 16 06:38:58 munin kernel: [1025880.826179] [<ffffffff8119b25b>] ? blk_queue_bio+0x17f/0x28c Sep 16 06:38:58 munin kernel: [1025880.826182] [<ffffffff81199d88>] ? generic_make_request+0x90/0xcf Sep 16 06:38:58 munin kernel: [1025880.826186] [<ffffffff81199e9a>] ? submit_bio+0xd3/0xf1 Sep 16 06:38:58 munin kernel: [1025880.826196] [<ffffffffa012dfad>] ? ext4_io_submit+0x21/0x4a [ext4] Sep 16 06:38:58 munin kernel: [1025880.826204] [<ffffffffa012ae52>] ? mpage_da_submit_io+0x359/0x36f [ext4] Sep 16 06:38:58 munin kernel: [1025880.826212] [<ffffffffa012cac7>] ? mpage_da_map_and_submit+0x2aa/0x2f9 [ext4] Sep 16 06:38:58 munin kernel: [1025880.826219] [<ffffffffa012bd80>] ? ext4_mark_inode_dirty+0x1af/0x1da [ext4] Sep 16 06:38:58 munin kernel: [1025880.826226] [<ffffffffa012cb00>] ? mpage_da_map_and_submit+0x2e3/0x2f9 [ext4] Sep 16 06:38:58 munin kernel: [1025880.826232] [<ffffffffa012d0e7>] ? ext4_da_writepages+0x239/0x45d [ext4] Sep 16 06:38:58 munin kernel: [1025880.826242] [<ffffffffa0142724>] ? ext4_journal_start_sb+0x139/0x14f [ext4] Sep 16 06:38:58 munin kernel: [1025880.826249] [<ffffffffa012d172>] ? ext4_da_writepages+0x2c4/0x45d [ext4] Sep 16 06:38:58 munin kernel: [1025880.826266] [<ffffffff81118ad3>] ? writeback_single_inode+0x11d/0x2cc Sep 16 06:38:58 munin kernel: [1025880.826270] [<ffffffff81118f53>] ? writeback_sb_inodes+0x16b/0x204 Sep 16 06:38:58 munin kernel: [1025880.826274] [<ffffffff81119059>] ? __writeback_inodes_wb+0x6d/0xab Sep 16 06:38:58 munin kernel: [1025880.826278] [<ffffffff811191bf>] ? wb_writeback+0x128/0x21f Sep 16 06:38:58 munin kernel: [1025880.826283] [<ffffffff8107116d>] ? arch_local_irq_save+0x11/0x17 Sep 16 06:38:58 munin kernel: [1025880.826287] [<ffffffff81119676>] ? wb_do_writeback+0x146/0x1a8 Sep 16 06:38:58 munin kernel: [1025880.826291] [<ffffffff8111975d>] ? bdi_writeback_thread+0x85/0x1e6 Sep 16 06:38:58 munin kernel: [1025880.826295] [<ffffffff811196d8>] ? wb_do_writeback+0x1a8/0x1a8 Sep 16 06:38:58 munin kernel: [1025880.826298] [<ffffffff8105f701>] ? kthread+0x76/0x7e Sep 16 06:38:58 munin kernel: [1025880.826302] [<ffffffff813575b4>] ? kernel_thread_helper+0x4/0x10 Sep 16 06:38:58 munin kernel: [1025880.826305] [<ffffffff8105f68b>] ? kthread_worker_fn+0x139/0x139 Sep 16 06:38:58 munin kernel: [1025880.826308] [<ffffffff813575b0>] ? gs_change+0x13/0x13 Sep 16 06:38:58 munin kernel: [1025880.827272] munin-graph D ffff88003fc13780 0 29913 29580 0x00000000 Sep 16 06:38:58 munin kernel: [1025880.827276] ffff88003d3a8140 0000000000000086 0000000000000000 ffffffff8160d020 Sep 16 06:38:58 munin kernel: [1025880.827279] 0000000000013780 ffff880036d2bfd8 ffff880036d2bfd8 ffff88003d3a8140 Sep 16 06:38:58 munin kernel: [1025880.827283] 0000000000000246 0000000181350461 ffffffffa016f508 ffff88002c9a4558 Sep 16 06:38:58 munin kernel: [1025880.827287] Call Trace: Sep 16 06:38:58 munin kernel: [1025880.827295] [<ffffffffa012dabc>] ? ext4_ioend_wait+0x82/0x98 [ext4] Sep 16 06:38:58 munin kernel: [1025880.827299] [<ffffffff8105fd53>] ? add_wait_queue+0x3c/0x3c Sep 16 06:38:58 munin kernel: [1025880.827307] [<ffffffffa012d344>] ? ext4_evict_inode+0x39/0x2af [ext4] Sep 16 06:38:58 munin kernel: [1025880.827313] [<ffffffffa012d30b>] ? ext4_da_writepages+0x45d/0x45d [ext4] Sep 16 06:38:58 munin kernel: [1025880.827319] [<ffffffff8110d748>] ? evict+0x96/0x148 Sep 16 06:38:58 munin kernel: [1025880.827322] [<ffffffff8110ab10>] ? dentry_kill+0x112/0x12e Sep 16 06:38:58 munin kernel: [1025880.827325] [<ffffffff8110ad62>] ? dput+0xe2/0xee Sep 16 06:38:58 munin kernel: [1025880.827330] [<ffffffff81106c71>] ? sys_renameat+0x1af/0x206 Sep 16 06:38:58 munin kernel: [1025880.827335] [<ffffffff81126b2a>] ? fsnotify+0x1e6/0x20c Sep 16 06:38:58 munin kernel: [1025880.827340] [<ffffffff8102b2d2>] ? kvm_clock_read+0x17/0x1a Sep 16 06:38:58 munin kernel: [1025880.827344] [<ffffffff81066311>] ? timekeeping_get_ns+0xd/0x2a Sep 16 06:38:58 munin kernel: [1025880.827347] [<ffffffff810668f5>] ? getnstimeofday+0x4a/0x7b Sep 16 06:38:58 munin kernel: [1025880.827350] [<ffffffff81066955>] ? do_gettimeofday+0x10/0x30 Sep 16 06:38:58 munin kernel: [1025880.827354] [<ffffffff81355452>] ? system_call_fastpath+0x16/0x1b Sep 16 06:38:58 munin kernel: [1025880.829214] munin-update D ffff88003fc93780 0 29920 29919 0x00000000 Sep 16 06:38:58 munin kernel: [1025880.829220] ffff8800370cf800 0000000000000082 0000000000000000 ffff88003e216180 Sep 16 06:38:58 munin kernel: [1025880.829227] 0000000000013780 ffff88003712ffd8 ffff88003712ffd8 ffff8800370cf800 Sep 16 06:38:58 munin kernel: [1025880.829233] 0000000000000005 000000013712fe68 ffff88003712fd68 ffff88003dd068e0 Sep 16 06:38:58 munin kernel: [1025880.829239] Call Trace: Sep 16 06:38:58 munin kernel: [1025880.829245] [<ffffffff8134fa44>] ? __mutex_lock_common.isra.5+0xff/0x164 Sep 16 06:38:58 munin kernel: [1025880.829250] [<ffffffff8134f932>] ? mutex_lock+0x1a/0x2d Sep 16 06:38:58 munin kernel: [1025880.829256] [<ffffffff8110576b>] ? do_last+0x1b1/0x58d Sep 16 06:38:58 munin kernel: [1025880.829261] [<ffffffff8110613f>] ? path_openat+0xce/0x33a Sep 16 06:38:58 munin kernel: [1025880.829277] [<ffffffff810fa734>] ? do_sync_write+0xb4/0xec Sep 16 06:38:58 munin kernel: [1025880.829282] [<ffffffff8110646d>] ? do_filp_open+0x2a/0x6e Sep 16 06:38:58 munin kernel: [1025880.829287] [<ffffffff8134f144>] ? _cond_resched+0x7/0x1c Sep 16 06:38:58 munin kernel: [1025880.829307] [<ffffffff811b4f49>] ? __strncpy_from_user+0x18/0x48 Sep 16 06:38:58 munin kernel: [1025880.829314] [<ffffffff8110f1f3>] ? alloc_fd+0x64/0x109 Sep 16 06:38:58 munin kernel: [1025880.829318] [<ffffffff810fa439>] ? do_sys_open+0x5e/0xe5 Sep 16 06:38:58 munin kernel: [1025880.829323] [<ffffffff81355452>] ? system_call_fastpath+0x16/0x1b Sep 16 06:40:58 munin kernel: [1026000.830447] jbd2/vda1-8 D ffff88003fc93780 0 179 2 0x00000000 Sep 16 06:40:58 munin kernel: [1026000.830454] ffff88003bb30740 0000000000000046 0000000000000000 ffff88003e216180 Sep 16 06:40:58 munin kernel: [1026000.830467] 0000000000013780 ffff880036d77fd8 ffff880036d77fd8 ffff88003bb30740 Sep 16 06:40:58 munin kernel: [1026000.830471] ffff880036d76000 0000000181066311 ffff88003e1d5c20 ffff88003fc93fd0 Sep 16 06:40:58 munin kernel: [1026000.830476] Call Trace: Sep 16 06:40:58 munin kernel: [1026000.830491] [<ffffffff8111da09>] ? wait_on_buffer+0x28/0x28 Sep 16 06:40:58 munin kernel: [1026000.830498] [<ffffffff8134f399>] ? io_schedule+0x59/0x71 Sep 16 06:40:58 munin kernel: [1026000.830501] [<ffffffff8111da0f>] ? sleep_on_buffer+0x6/0xa Sep 16 06:40:58 munin kernel: [1026000.830504] [<ffffffff8134f7dc>] ? __wait_on_bit+0x3e/0x71 Sep 16 06:40:58 munin kernel: [1026000.830508] [<ffffffff8134f87e>] ? out_of_line_wait_on_bit+0x6f/0x78 Sep 16 06:40:58 munin kernel: [1026000.830511] [<ffffffff8111da09>] ? wait_on_buffer+0x28/0x28 Sep 16 06:40:58 munin kernel: [1026000.830518] [<ffffffff8105fd7d>] ? autoremove_wake_function+0x2a/0x2a Sep 16 06:40:58 munin kernel: [1026000.830539] [<ffffffffa0115a4c>] ? jbd2_journal_commit_transaction+0xbeb/0x10bf [jbd2] Sep 16 06:40:58 munin kernel: [1026000.830546] [<ffffffff8100d02f>] ? load_TLS+0x7/0xa Sep 16 06:40:58 munin kernel: [1026000.830550] [<ffffffff8100d69e>] ? __switch_to+0x133/0x258 Sep 16 06:40:58 munin kernel: [1026000.830556] [<ffffffffa0119156>] ? kjournald2+0xc0/0x20a [jbd2] Sep 16 06:40:58 munin kernel: [1026000.830560] [<ffffffff8105fd53>] ? add_wait_queue+0x3c/0x3c Sep 16 06:40:58 munin kernel: [1026000.830565] [<ffffffffa0119096>] ? commit_timeout+0x5/0x5 [jbd2] Sep 16 06:40:58 munin kernel: [1026000.830569] [<ffffffff8105f701>] ? kthread+0x76/0x7e Sep 16 06:40:58 munin kernel: [1026000.830575] [<ffffffff813575b4>] ? kernel_thread_helper+0x4/0x10 Sep 16 06:40:58 munin kernel: [1026000.830579] [<ffffffff8105f68b>] ? kthread_worker_fn+0x139/0x139 Sep 16 06:40:58 munin kernel: [1026000.830582] [<ffffffff813575b0>] ? gs_change+0x13/0x13 Sep 16 06:40:58 munin kernel: [1026000.832915] flush-254:0 D ffff88003fc13780 0 2672 2 0x00000000 Sep 16 06:40:58 munin kernel: [1026000.832915] ffff880037075740 0000000000000046 0000000000000000 ffffffff8160d020 Sep 16 06:40:58 munin kernel: [1026000.832919] 0000000000013780 ffff88003b955fd8 ffff88003b955fd8 ffff880037075740 Sep 16 06:40:58 munin kernel: [1026000.832925] ffff88003b955770 000000013b955770 ffff88003d424c90 ffff88003fc13fd0 Sep 16 06:40:58 munin kernel: [1026000.832930] Call Trace: Sep 16 06:40:58 munin kernel: [1026000.832936] [<ffffffff8134f399>] ? io_schedule+0x59/0x71 Sep 16 06:40:58 munin kernel: [1026000.832944] [<ffffffff8119a2fc>] ? get_request_wait+0x105/0x18f Sep 16 06:40:58 munin kernel: [1026000.832950] [<ffffffff8105fd53>] ? add_wait_queue+0x3c/0x3c Sep 16 06:40:58 munin kernel: [1026000.832956] [<ffffffff8119b25b>] ? blk_queue_bio+0x17f/0x28c Sep 16 06:40:58 munin kernel: [1026000.832961] [<ffffffff81199d88>] ? generic_make_request+0x90/0xcf Sep 16 06:40:58 munin kernel: [1026000.832966] [<ffffffff81199e9a>] ? submit_bio+0xd3/0xf1 Sep 16 06:40:58 munin kernel: [1026000.832981] [<ffffffffa012dfad>] ? ext4_io_submit+0x21/0x4a [ext4] Sep 16 06:40:58 munin kernel: [1026000.832991] [<ffffffffa012ae52>] ? mpage_da_submit_io+0x359/0x36f [ext4] Sep 16 06:40:58 munin kernel: [1026000.833002] [<ffffffffa012cac7>] ? mpage_da_map_and_submit+0x2aa/0x2f9 [ext4] Sep 16 06:40:58 munin kernel: [1026000.833012] [<ffffffffa012bd80>] ? ext4_mark_inode_dirty+0x1af/0x1da [ext4] Sep 16 06:40:58 munin kernel: [1026000.833022] [<ffffffffa012cb00>] ? mpage_da_map_and_submit+0x2e3/0x2f9 [ext4] Sep 16 06:40:58 munin kernel: [1026000.833031] [<ffffffffa012d0e7>] ? ext4_da_writepages+0x239/0x45d [ext4] Sep 16 06:40:58 munin kernel: [1026000.833045] [<ffffffffa0142724>] ? ext4_journal_start_sb+0x139/0x14f [ext4] Sep 16 06:40:58 munin kernel: [1026000.833055] [<ffffffffa012d172>] ? ext4_da_writepages+0x2c4/0x45d [ext4] Sep 16 06:40:58 munin kernel: [1026000.833064] [<ffffffff81118ad3>] ? writeback_single_inode+0x11d/0x2cc Sep 16 06:40:58 munin kernel: [1026000.833070] [<ffffffff81118f53>] ? writeback_sb_inodes+0x16b/0x204 Sep 16 06:40:58 munin kernel: [1026000.833076] [<ffffffff81119059>] ? __writeback_inodes_wb+0x6d/0xab Sep 16 06:40:58 munin kernel: [1026000.833081] [<ffffffff811191bf>] ? wb_writeback+0x128/0x21f Sep 16 06:40:58 munin kernel: [1026000.833088] [<ffffffff8107116d>] ? arch_local_irq_save+0x11/0x17 Sep 16 06:40:58 munin kernel: [1026000.833093] [<ffffffff81119676>] ? wb_do_writeback+0x146/0x1a8 Sep 16 06:40:58 munin kernel: [1026000.833098] [<ffffffff8111975d>] ? bdi_writeback_thread+0x85/0x1e6 Sep 16 06:40:58 munin kernel: [1026000.833104] [<ffffffff811196d8>] ? wb_do_writeback+0x1a8/0x1a8 Sep 16 06:40:58 munin kernel: [1026000.833109] [<ffffffff8105f701>] ? kthread+0x76/0x7e Sep 16 06:40:58 munin kernel: [1026000.833114] [<ffffffff813575b4>] ? kernel_thread_helper+0x4/0x10 Sep 16 06:40:58 munin kernel: [1026000.833119] [<ffffffff8105f68b>] ? kthread_worker_fn+0x139/0x139 Sep 16 06:40:58 munin kernel: [1026000.833124] [<ffffffff813575b0>] ? gs_change+0x13/0x13 Sep 16 06:40:58 munin kernel: [1026000.835590] munin-graph D ffff88003fc13780 0 29913 29580 0x00000000 Sep 16 06:40:58 munin kernel: [1026000.835597] ffff88003d3a8140 0000000000000086 0000000000000000 ffffffff8160d020 Sep 16 06:40:58 munin kernel: [1026000.835601] 0000000000013780 ffff880036d2bfd8 ffff880036d2bfd8 ffff88003d3a8140 Sep 16 06:40:58 munin kernel: [1026000.835604] 0000000000000246 0000000181350461 ffffffffa016f508 ffff88002c9a4558 Sep 16 06:40:58 munin kernel: [1026000.835608] Call Trace: Sep 16 06:40:58 munin kernel: [1026000.835618] [<ffffffffa012dabc>] ? ext4_ioend_wait+0x82/0x98 [ext4] Sep 16 06:40:58 munin kernel: [1026000.835628] [<ffffffff8105fd53>] ? add_wait_queue+0x3c/0x3c Sep 16 06:40:58 munin kernel: [1026000.835636] [<ffffffffa012d344>] ? ext4_evict_inode+0x39/0x2af [ext4] Sep 16 06:40:58 munin kernel: [1026000.835643] [<ffffffffa012d30b>] ? ext4_da_writepages+0x45d/0x45d [ext4] Sep 16 06:40:58 munin kernel: [1026000.835649] [<ffffffff8110d748>] ? evict+0x96/0x148 Sep 16 06:40:58 munin kernel: [1026000.835652] [<ffffffff8110ab10>] ? dentry_kill+0x112/0x12e Sep 16 06:40:58 munin kernel: [1026000.835655] [<ffffffff8110ad62>] ? dput+0xe2/0xee Sep 16 06:40:58 munin kernel: [1026000.835660] [<ffffffff81106c71>] ? sys_renameat+0x1af/0x206 Sep 16 06:40:58 munin kernel: [1026000.835665] [<ffffffff81126b2a>] ? fsnotify+0x1e6/0x20c Sep 16 06:40:58 munin kernel: [1026000.835670] [<ffffffff8102b2d2>] ? kvm_clock_read+0x17/0x1a Sep 16 06:40:58 munin kernel: [1026000.835674] [<ffffffff81066311>] ? timekeeping_get_ns+0xd/0x2a Sep 16 06:40:58 munin kernel: [1026000.835677] [<ffffffff810668f5>] ? getnstimeofday+0x4a/0x7b Sep 16 06:40:58 munin kernel: [1026000.835680] [<ffffffff81066955>] ? do_gettimeofday+0x10/0x30 Sep 16 06:40:58 munin kernel: [1026000.835684] [<ffffffff81355452>] ? system_call_fastpath+0x16/0x1b -------------- next part -------------- Sep 16 06:36:58 munin kernel: [1025760.824197] INFO: task jbd2/vda1-8:179 blocked for more than 120 seconds. Sep 16 06:36:58 munin kernel: [1025760.824699] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 16 06:36:58 munin kernel: [1025760.825461] INFO: task flush-254:0:2672 blocked for more than 120 seconds. Sep 16 06:36:58 munin kernel: [1025760.825909] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 16 06:36:58 munin kernel: [1025760.826597] INFO: task munin-graph:29913 blocked for more than 120 seconds. Sep 16 06:36:58 munin kernel: [1025760.827045] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 16 06:38:58 munin kernel: [1025880.824119] INFO: task jbd2/vda1-8:179 blocked for more than 120 seconds. Sep 16 06:38:58 munin kernel: [1025880.824580] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.