Hervé Moris
2019-Apr-04 09:16 UTC
[Pkg-xen-devel] DomU freeze with linux-image-4.9.0-8-amd64 and gnttab_max_frames=256
Subject: xen-hypervisor-4.11-amd64: DomU freeze with linux-image-4.9.0-8-amd64 and gnttab_max_frames=256 Package: xen-hypervisor-4.11-amd64 Version: 4.11.1-1 Justification: breaks the whole system Severity: critical Tags: upstream Dear Maintainers, The issue I have looks a lot to the bug reported here : https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=880554 except that putting gnttab_max_frames=256 didn't solved my issue and the environement may be a bit different. So here is the situation, I've downloaded Xen sources from the repo, compiled them with XSM/Flask support, made deb packages and installed them on a fresh install of Debian 9. Dom0 4-6 vcpu with 4G ram Dom0 and DomU uname -a : Linux 4.9.0-8-amd64 #1 SMP Debian 4.9.144-3.1 (2019-02-19) x86_64 GNU/Linux DomUs are launched with PvGrub and seclabel='system_u:system_r:isolated_domU_t' The problematic DomU is connected to a LUN target (iscsi) for some storage and run some docker containers. (just in case ...) I thought it might be related to the iscsi drive so I completely uninstalled it but the problem is still occuring. DomU.conf : kernel = '/usr/lib/grub-xen/grub-x86_64-xen.bin' extra = '(hd0)/boot/grub/grub.cfg' seclabel='system_u:system_r:isolated_domU_t' vcpus = '4' memory = '4048' root = '/dev/xvda2 ro' disk = [ 'phy:/dev/vg/domu-root,xvda2,w', 'phy:/dev/vg/domu-var-lib-docker,xvda10,w', 'phy:/dev/vg/domu-var-log-audit,xvda9,w', 'phy:/dev/vg/domu-var-log,xvda8,w', 'phy:/dev/vg/domu-home,xvda7,w', 'phy:/dev/vg/domu-var,xvda6,w', 'phy:/dev/vg/domu-usr,xvda5,w', 'phy:/dev/vg/domu-tmp,xvda4,w', 'phy:/dev/vg/domu-srv,xvda3,w', 'phy:/dev/vg/domu-swap,xvda1,w', 'phy:/dev/vg/domu-storage,xvda11,w', ] name = 'domu' vif = [ 'mac=xx:xx:xx:xx:xx:xx, bridge=xenbr0' ] on_poweroff = 'destroy' on_reboot = 'restart' on_crash = 'restart' After some "heavy" IO loads, the DomU freeze and the console show messages like following : [ 484.316142] INFO: task kworker/u8:1:48 blocked for more than 120 seconds. [ 484.316183] Not tainted 4.9.0-8-amd64 #1 Debian 4.9.144-3.1 [ 484.316208] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 484.316240] kworker/u8:1 D 0 48 2 0x00000080 [ 484.316276] Workqueue: writeback wb_workfn (flush-202:4) [ 484.316313] ffff8800e8252540 0000000000000000 ffff8800f2c822c0 ffff8800f7f98980 [ 484.316330] ffff8800f6c06380 ffffc900408d35d0 ffffffff816144b9 0000000000000001 [ 484.316347] 00ff8800ebbf60c0 ffff8800f7f98980 ffffffff813135d9 ffff8800f2c822c0 [ 484.316367] Call Trace: [ 484.316378] [<ffffffff816144b9>] ? __schedule+0x239/0x6f0 [ 484.316389] [<ffffffff813135d9>] ? blk_mq_flush_plug_list+0x139/0x160 [ 484.316401] [<ffffffff816149a2>] ? schedule+0x32/0x80 [ 484.316410] [<ffffffff81617d4d>] ? schedule_timeout+0x1dd/0x380 [ 484.316421] [<ffffffff813086b3>] ? blk_flush_plug_list+0xc3/0x230 [ 484.316433] [<ffffffff811861b9>] ? mempool_alloc+0x69/0x190 [ 484.316444] [<ffffffff8101bf91>] ? xen_clocksource_get_cycles+0x11/0x20 [ 484.316455] [<ffffffff8161421d>] ? io_schedule_timeout+0x9d/0x100 [ 484.316467] [<ffffffff81363964>] ? __sbitmap_queue_get+0x24/0x90 [ 484.316477] [<ffffffff81313b80>] ? bt_get.isra.6+0x160/0x220 [ 484.316489] [<ffffffff810bd350>] ? prepare_to_wait_event+0xf0/0xf0 [ 484.316500] [<ffffffff81313ef3>] ? blk_mq_get_tag+0x23/0x90 [ 484.316510] [<ffffffff8130f77a>] ? __blk_mq_alloc_request+0x1a/0x220 [ 484.316520] [<ffffffff81310649>] ? blk_mq_map_request+0xd9/0x170 [ 484.316531] [<ffffffff81312e8b>] ? blk_mq_make_request+0xbb/0x580 [ 484.316541] [<ffffffff81306d35>] ? generic_make_request+0x115/0x2d0 [ 484.316551] [<ffffffff81306f66>] ? submit_bio+0x76/0x140 [ 484.316573] [<ffffffffc01be358>] ? ext4_io_submit+0x48/0x60 [ext4] [ 484.316593] [<ffffffffc01be5c6>] ? ext4_bio_write_page+0x236/0x4d0 [ext4] [ 484.316605] [<ffffffff81190f98>] ? clear_page_dirty_for_io+0x148/0x1b0 [ 484.316621] [<ffffffffc01b3a32>] ? mpage_submit_page+0x52/0x70 [ext4] [ 484.316636] [<ffffffffc01b3c83>] ? mpage_map_and_submit_buffers+0x133/0x230 [ext4] [ 484.316658] [<ffffffffc01b8fb0>] ? ext4_writepages+0x7c0/0xd60 [ext4] [ 484.316669] [<ffffffff81618df4>] ? _raw_spin_unlock_irqrestore+0x14/0x20 [ 484.316680] [<ffffffff810e9c57>] ? mod_timer+0x177/0x3b0 [ 484.316690] [<ffffffff8123a7bd>] ? __writeback_single_inode+0x3d/0x320 [ 484.316700] [<ffffffff8123af61>] ? writeback_sb_inodes+0x221/0x4f0 [ 484.316714] [<ffffffff8123b2b7>] ? __writeback_inodes_wb+0x87/0xb0 [ 484.316724] [<ffffffff8123b62e>] ? wb_writeback+0x27e/0x310 [ 484.316734] [<ffffffff8123be5e>] ? wb_workfn+0x17e/0x380 [ 484.316745] [<ffffffff8109427a>] ? process_one_work+0x18a/0x430 [ 484.316755] [<ffffffff8109456d>] ? worker_thread+0x4d/0x490 [ 484.316765] [<ffffffff81094520>] ? process_one_work+0x430/0x430 [ 484.316775] [<ffffffff8109a5d9>] ? kthread+0xd9/0xf0 [ 484.316786] [<ffffffff8109a500>] ? kthread_park+0x60/0x60 [ 484.316796] [<ffffffff816193f7>] ? ret_from_fork+0x57/0x70 [ 484.316808] INFO: task jbd2/xvda5-8:194 blocked for more than 120 seconds. [ 484.316819] Not tainted 4.9.0-8-amd64 #1 Debian 4.9.144-3.1 [ 484.316828] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 484.316841] jbd2/xvda5-8 D 0 194 2 0x00000080 [ 484.316851] ffff8800f2dac500 0000000000000000 ffff88000298c380 ffff8800f7e18980 [ 484.316867] ffff8800f2dfa7c0 ffffc9004093bb30 ffffffff816144b9 ffffc9004093bbe8 [ 484.316885] 00ffffff81307eaf ffff8800f7e18980 0000000000000000 ffff88000298c380 [ 484.316903] Call Trace: [ 484.316910] [<ffffffff816144b9>] ? __schedule+0x239/0x6f0 [ 484.316920] [<ffffffff81615170>] ? bit_wait+0x50/0x50 [ 484.316930] [<ffffffff816149a2>] ? schedule+0x32/0x80 [ 484.316938] [<ffffffff81617d4d>] ? schedule_timeout+0x1dd/0x380 [ 484.316952] [<ffffffff810f107e>] ? ktime_get+0x3e/0xb0 [ 484.316961] [<ffffffff81615170>] ? bit_wait+0x50/0x50 [ 484.316971] [<ffffffff8161421d>] ? io_schedule_timeout+0x9d/0x100 [ 484.316980] [<ffffffff810bcf87>] ? prepare_to_wait+0x57/0x80 [ 484.316991] [<ffffffff81615187>] ? bit_wait_io+0x17/0x60 [ 484.317000] [<ffffffff81614d45>] ? __wait_on_bit+0x55/0x80 [ 484.317009] [<ffffffff81615170>] ? bit_wait+0x50/0x50 [ 484.317018] [<ffffffff81614eae>] ? out_of_line_wait_on_bit+0x7e/0xa0 [ 484.317029] [<ffffffff810bd3f0>] ? wake_atomic_t_function+0x60/0x60 [ 484.317043] [<ffffffffc0076e6e>] ? jbd2_journal_commit_transaction+0xdde/0x17b0 [jbd2] [ 484.317059] [<ffffffff81619364>] ? __switch_to_asm+0x34/0x70 [ 484.317069] [<ffffffff81619370>] ? __switch_to_asm+0x40/0x70 [ 484.317080] [<ffffffff810a1998>] ? finish_task_switch+0x78/0x210 [ 484.317090] [<ffffffff81618df4>] ? _raw_spin_unlock_irqrestore+0x14/0x20 [ 484.317102] [<ffffffffc007bc62>] ? kjournald2+0xc2/0x260 [jbd2] [ 484.317114] [<ffffffff810bd350>] ? prepare_to_wait_event+0xf0/0xf0 [ 484.317125] [<ffffffffc007bba0>] ? commit_timeout+0x10/0x10 [jbd2] [ 484.317135] [<ffffffff8109a5d9>] ? kthread+0xd9/0xf0 [ 484.317146] [<ffffffff8109a500>] ? kthread_park+0x60/0x60 [ 484.317154] [<ffffffff816193f7>] ? ret_from_fork+0x57/0x70 [ 484.317164] INFO: task jbd2/xvda4-8:328 blocked for more than 120 seconds. [ 484.317176] Not tainted 4.9.0-8-amd64 #1 Debian 4.9.144-3.1 [ 484.317185] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 484.317196] jbd2/xvda4-8 D 0 328 2 0x00000080 [ 484.317205] ffff8800f5c3b200 0000000000000000 ffff8800f2d56800 ffff8800f7f18980 [ 484.317221] ffff880002cb0480 ffffc900409c7a70 ffffffff816144b9 ffffffff81092a28 [ 484.317237] 00ffe8ffffd8b540 ffff8800f7f18980 ffffffff813135d9 ffff8800f2d56800 [ 484.317253] Call Trace: [ 484.317260] [<ffffffff816144b9>] ? __schedule+0x239/0x6f0 [ 484.317268] [<ffffffff81092a28>] ? queue_work_on+0x28/0x40 [ 484.317278] [<ffffffff813135d9>] ? blk_mq_flush_plug_list+0x139/0x160 [ 484.317287] [<ffffffff81615170>] ? bit_wait+0x50/0x50 [ 484.317296] [<ffffffff816149a2>] ? schedule+0x32/0x80 [ 484.317304] [<ffffffff81617d4d>] ? schedule_timeout+0x1dd/0x380 [ 484.317315] [<ffffffff8101bf91>] ? xen_clocksource_get_cycles+0x11/0x20 [ 484.317325] [<ffffffff81615170>] ? bit_wait+0x50/0x50 [ 484.317334] [<ffffffff8161421d>] ? io_schedule_timeout+0x9d/0x100 [ 484.317343] [<ffffffff810bcf87>] ? prepare_to_wait+0x57/0x80 [ 484.317353] [<ffffffff81615187>] ? bit_wait_io+0x17/0x60 [ 484.317362] [<ffffffff81614d45>] ? __wait_on_bit+0x55/0x80 [ 484.317373] [<ffffffff811828c8>] ? find_get_pages_tag+0x158/0x2e0 [ 484.317383] [<ffffffff81181b2f>] ? wait_on_page_bit+0x7f/0xa0 [ 484.317393] [<ffffffff810bd3f0>] ? wake_atomic_t_function+0x60/0x60 [ 484.317403] [<ffffffff81181c30>] ? __filemap_fdatawait_range+0xe0/0x140 [ 484.317413] [<ffffffff81181c9f>] ? filemap_fdatawait_range+0xf/0x30 [ 484.317425] [<ffffffffc00767cd>] ? jbd2_journal_commit_transaction+0x73d/0x17b0 [jbd2] [ 484.317437] [<ffffffff81619364>] ? __switch_to_asm+0x34/0x70 [ 484.317447] [<ffffffff81619370>] ? __switch_to_asm+0x40/0x70 [ 484.317456] [<ffffffff810158e2>] ? xen_load_sp0+0x82/0x170 [ 484.317465] [<ffffffff810a1998>] ? finish_task_switch+0x78/0x210 [ 484.317475] [<ffffffff81618df4>] ? _raw_spin_unlock_irqrestore+0x14/0x20 [ 484.317486] [<ffffffffc007bc62>] ? kjournald2+0xc2/0x260 [jbd2] [ 484.317496] [<ffffffff810bd350>] ? prepare_to_wait_event+0xf0/0xf0 [ 484.516588] [<ffffffffc007bba0>] ? commit_timeout+0x10/0x10 [jbd2] [ 484.516603] [<ffffffff8109a5d9>] ? kthread+0xd9/0xf0 [ 484.516612] [<ffffffff8109a500>] ? kthread_park+0x60/0x60 [ 484.516624] [<ffffffff816193f7>] ? ret_from_fork+0x57/0x70 [ 484.516640] INFO: task jbd2/xvda8-8:367 blocked for more than 120 seconds. [ 484.516656] Not tainted 4.9.0-8-amd64 #1 Debian 4.9.144-3.1 [ 484.516667] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 484.516684] jbd2/xvda8-8 D 0 367 2 0x00000080 [ 484.516704] ffff8800f5c3b200 0000000000000000 ffff8800f5cac740 ffff8800f7f18980 [ 484.516741] ffff880002cb0480 ffffc90040ae7b30 ffffffff816144b9 ffff8800ee926000 [ 484.516776] 00ff8800ee943200 ffff8800f7f18980 ffff88008e403f00 ffff8800f5cac740 [ 484.516794] Call Trace: [ 484.516802] [<ffffffff816144b9>] ? __schedule+0x239/0x6f0 [ 484.516811] [<ffffffff81615170>] ? bit_wait+0x50/0x50 [ 484.516823] [<ffffffff816149a2>] ? schedule+0x32/0x80 [ 484.516831] [<ffffffff81617d4d>] ? schedule_timeout+0x1dd/0x380 [ 484.516841] [<ffffffff81619370>] ? __switch_to_asm+0x40/0x70 [ 484.516850] [<ffffffff81619364>] ? __switch_to_asm+0x34/0x70 [ 484.516861] [<ffffffff8101bf91>] ? xen_clocksource_get_cycles+0x11/0x20 [ 484.516871] [<ffffffff810f107e>] ? ktime_get+0x3e/0xb0 [ 484.516880] [<ffffffff81615170>] ? bit_wait+0x50/0x50 [ 484.516889] [<ffffffff8161421d>] ? io_schedule_timeout+0x9d/0x100 [ 484.516899] [<ffffffff810bcf87>] ? prepare_to_wait+0x57/0x80 [ 484.516910] [<ffffffff81615187>] ? bit_wait_io+0x17/0x60 [ 484.516921] [<ffffffff81614d45>] ? __wait_on_bit+0x55/0x80 [ 484.516930] [<ffffffff810a1998>] ? finish_task_switch+0x78/0x210 [ 484.516940] [<ffffffff81615170>] ? bit_wait+0x50/0x50 [ 484.516950] [<ffffffff81614eae>] ? out_of_line_wait_on_bit+0x7e/0xa0 [ 484.516961] [<ffffffff810bd3f0>] ? wake_atomic_t_function+0x60/0x60 [ 484.516972] [<ffffffffc0076fe5>] ? jbd2_journal_commit_transaction+0xf55/0x17b0 [jbd2] [ 484.516985] [<ffffffff81619364>] ? __switch_to_asm+0x34/0x70 [ 484.516994] [<ffffffff81619370>] ? __switch_to_asm+0x40/0x70 [ 484.517005] [<ffffffff810a1998>] ? finish_task_switch+0x78/0x210 [ 484.517016] [<ffffffff81618df4>] ? _raw_spin_unlock_irqrestore+0x14/0x20 [ 484.517028] [<ffffffffc007bc62>] ? kjournald2+0xc2/0x260 [jbd2] [ 484.517038] [<ffffffff810bd350>] ? prepare_to_wait_event+0xf0/0xf0 [ 484.517051] [<ffffffffc007bba0>] ? commit_timeout+0x10/0x10 [jbd2] [ 484.517062] [<ffffffff8109a5d9>] ? kthread+0xd9/0xf0 [ 484.517071] [<ffffffff8109a500>] ? kthread_park+0x60/0x60 [ 484.517079] [<ffffffff816193f7>] ? ret_from_fork+0x57/0x70 [ 605.148081] INFO: task kworker/u8:1:48 blocked for more than 120 seconds. [ 605.148121] Not tainted 4.9.0-8-amd64 #1 Debian 4.9.144-3.1 [ 605.148146] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 605.148178] kworker/u8:1 D 0 48 2 0x00000080 [ 605.148214] Workqueue: writeback wb_workfn (flush-202:4) [ 605.148242] ffff8800e8252540 0000000000000000 ffff8800f2c822c0 ffff8800f7f98980 [ 605.148287] ffff8800f6c06380 ffffc900408d35d0 ffffffff816144b9 0000000000000001 [ 605.148336] 00ff8800ebbf60c0 ffff8800f7f98980 ffffffff813135d9 ffff8800f2c822c0 [ 605.148381] Call Trace: [ 605.148410] [<ffffffff816144b9>] ? __schedule+0x239/0x6f0 [ 605.148438] [<ffffffff813135d9>] ? blk_mq_flush_plug_list+0x139/0x160 [ 605.148476] [<ffffffff816149a2>] ? schedule+0x32/0x80 [ 605.148493] [<ffffffff81617d4d>] ? schedule_timeout+0x1dd/0x380 [ 605.148510] [<ffffffff813086b3>] ? blk_flush_plug_list+0xc3/0x230 [ 605.148527] [<ffffffff811861b9>] ? mempool_alloc+0x69/0x190 [ 605.148541] [<ffffffff8101bf91>] ? xen_clocksource_get_cycles+0x11/0x20 [ 605.148555] [<ffffffff8161421d>] ? io_schedule_timeout+0x9d/0x100 [ 605.148571] [<ffffffff81363964>] ? __sbitmap_queue_get+0x24/0x90 [ 605.148586] [<ffffffff81313b80>] ? bt_get.isra.6+0x160/0x220 [ 605.148602] [<ffffffff810bd350>] ? prepare_to_wait_event+0xf0/0xf0 [ 605.148618] [<ffffffff81313ef3>] ? blk_mq_get_tag+0x23/0x90 [ 605.148629] [<ffffffff8130f77a>] ? __blk_mq_alloc_request+0x1a/0x220 [ 605.148639] [<ffffffff81310649>] ? blk_mq_map_request+0xd9/0x170 [ 605.148649] [<ffffffff81312e8b>] ? blk_mq_make_request+0xbb/0x580 [ 605.148660] [<ffffffff81306d35>] ? generic_make_request+0x115/0x2d0 [ 605.148670] [<ffffffff81306f66>] ? submit_bio+0x76/0x140 [ 605.148694] [<ffffffffc01be358>] ? ext4_io_submit+0x48/0x60 [ext4] [ 605.148714] [<ffffffffc01be5c6>] ? ext4_bio_write_page+0x236/0x4d0 [ext4] [ 605.148725] [<ffffffff81190f98>] ? clear_page_dirty_for_io+0x148/0x1b0 [ 605.148742] [<ffffffffc01b3a32>] ? mpage_submit_page+0x52/0x70 [ext4] [ 605.148757] [<ffffffffc01b3c83>] ? mpage_map_and_submit_buffers+0x133/0x230 [ext4] [ 605.148777] [<ffffffffc01b8fb0>] ? ext4_writepages+0x7c0/0xd60 [ext4] [ 605.148788] [<ffffffff81618df4>] ? _raw_spin_unlock_irqrestore+0x14/0x20 [ 605.148799] [<ffffffff810e9c57>] ? mod_timer+0x177/0x3b0 [ 605.148809] [<ffffffff8123a7bd>] ? __writeback_single_inode+0x3d/0x320 [ 605.148819] [<ffffffff8123af61>] ? writeback_sb_inodes+0x221/0x4f0 [ 605.148829] [<ffffffff8123b2b7>] ? __writeback_inodes_wb+0x87/0xb0 [ 605.148840] [<ffffffff8123b62e>] ? wb_writeback+0x27e/0x310 [ 605.148850] [<ffffffff8123be5e>] ? wb_workfn+0x17e/0x380 [ 605.148859] [<ffffffff8109427a>] ? process_one_work+0x18a/0x430 [ 605.148871] [<ffffffff8109456d>] ? worker_thread+0x4d/0x490 [ 605.148881] [<ffffffff81094520>] ? process_one_work+0x430/0x430 [ 605.148891] [<ffffffff8109a5d9>] ? kthread+0xd9/0xf0 [ 605.148900] [<ffffffff8109a500>] ? kthread_park+0x60/0x60 [ 605.148909] [<ffffffff816193f7>] ? ret_from_fork+0x57/0x70 [ 605.148920] INFO: task jbd2/xvda5-8:194 blocked for more than 120 seconds. [ 605.148929] Not tainted 4.9.0-8-amd64 #1 Debian 4.9.144-3.1 [ 605.148941] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 605.148952] jbd2/xvda5-8 D 0 194 2 0x00000080 [ 605.148961] ffff8800f2dac500 0000000000000000 ffff88000298c380 ffff8800f7e18980 [ 605.148979] ffff8800f2dfa7c0 ffffc9004093bb30 ffffffff816144b9 ffffc9004093bbe8 [ 605.148995] 00ffffff81307eaf ffff8800f7e18980 0000000000000000 ffff88000298c380 [ 605.149011] Call Trace: [ 605.149019] [<ffffffff816144b9>] ? __schedule+0x239/0x6f0 [ 605.149028] [<ffffffff81615170>] ? bit_wait+0x50/0x50 [ 605.149038] [<ffffffff816149a2>] ? schedule+0x32/0x80 [ 605.149046] [<ffffffff81617d4d>] ? schedule_timeout+0x1dd/0x380 [ 605.149057] [<ffffffff810f107e>] ? ktime_get+0x3e/0xb0 [ 605.149066] [<ffffffff81615170>] ? bit_wait+0x50/0x50 [ 605.149075] [<ffffffff8161421d>] ? io_schedule_timeout+0x9d/0x100 [ 605.149085] [<ffffffff810bcf87>] ? prepare_to_wait+0x57/0x80 [ 605.149096] [<ffffffff81615187>] ? bit_wait_io+0x17/0x60 [ 605.149106] [<ffffffff81614d45>] ? __wait_on_bit+0x55/0x80 [ 605.149115] [<ffffffff81615170>] ? bit_wait+0x50/0x50 [ 605.149124] [<ffffffff81614eae>] ? out_of_line_wait_on_bit+0x7e/0xa0 [ 605.149134] [<ffffffff810bd3f0>] ? wake_atomic_t_function+0x60/0x60 [ 605.149150] [<ffffffffc0076e6e>] ? jbd2_journal_commit_transaction+0xdde/0x17b0 [jbd2] [ 605.149162] [<ffffffff81619364>] ? __switch_to_asm+0x34/0x70 [ 605.149172] [<ffffffff81619370>] ? __switch_to_asm+0x40/0x70 [ 605.149184] [<ffffffff810a1998>] ? finish_task_switch+0x78/0x210 [ 605.149194] [<ffffffff81618df4>] ? _raw_spin_unlock_irqrestore+0x14/0x20 [ 605.149206] [<ffffffffc007bc62>] ? kjournald2+0xc2/0x260 [jbd2] [ 605.149216] [<ffffffff810bd350>] ? prepare_to_wait_event+0xf0/0xf0 [ 605.149227] [<ffffffffc007bba0>] ? commit_timeout+0x10/0x10 [jbd2] [ 605.149237] [<ffffffff8109a5d9>] ? kthread+0xd9/0xf0 [ 605.149246] [<ffffffff8109a500>] ? kthread_park+0x60/0x60 [ 605.149254] [<ffffffff816193f7>] ? ret_from_fork+0x57/0x70 [ 605.149264] INFO: task jbd2/xvda4-8:328 blocked for more than 120 seconds. [ 605.149273] Not tainted 4.9.0-8-amd64 #1 Debian 4.9.144-3.1 [ 605.149283] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 605.149294] jbd2/xvda4-8 D 0 328 2 0x00000080 [ 605.149303] ffff8800f5c3b200 0000000000000000 ffff8800f2d56800 ffff8800f7f18980 [ 605.149320] ffff880002cb0480 ffffc900409c7a70 ffffffff816144b9 ffffffff81092a28 [ 605.149337] 00ffe8ffffd8b540 ffff8800f7f18980 ffffffff813135d9 ffff8800f2d56800 [ 605.149354] Call Trace: [ 605.149361] [<ffffffff816144b9>] ? __schedule+0x239/0x6f0 [ 605.149370] [<ffffffff81092a28>] ? queue_work_on+0x28/0x40 [ 605.149379] [<ffffffff813135d9>] ? blk_mq_flush_plug_list+0x139/0x160 [ 605.149389] [<ffffffff81615170>] ? bit_wait+0x50/0x50 [ 605.149397] [<ffffffff816149a2>] ? schedule+0x32/0x80 [ 605.149406] [<ffffffff81617d4d>] ? schedule_timeout+0x1dd/0x380 [ 605.149416] [<ffffffff8101bf91>] ? xen_clocksource_get_cycles+0x11/0x20 [ 605.149426] [<ffffffff81615170>] ? bit_wait+0x50/0x50 [ 605.149434] [<ffffffff8161421d>] ? io_schedule_timeout+0x9d/0x100 [ 605.149444] [<ffffffff810bcf87>] ? prepare_to_wait+0x57/0x80 [ 605.149454] [<ffffffff81615187>] ? bit_wait_io+0x17/0x60 [ 605.149463] [<ffffffff81614d45>] ? __wait_on_bit+0x55/0x80 [ 605.149473] [<ffffffff811828c8>] ? find_get_pages_tag+0x158/0x2e0 [ 605.149483] [<ffffffff81181b2f>] ? wait_on_page_bit+0x7f/0xa0 [ 605.149493] [<ffffffff810bd3f0>] ? wake_atomic_t_function+0x60/0x60 [ 605.149503] [<ffffffff81181c30>] ? __filemap_fdatawait_range+0xe0/0x140 [ 605.149513] [<ffffffff81181c9f>] ? filemap_fdatawait_range+0xf/0x30 [ 605.149525] [<ffffffffc00767cd>] ? jbd2_journal_commit_transaction+0x73d/0x17b0 [jbd2] [ 605.149537] [<ffffffff81619364>] ? __switch_to_asm+0x34/0x70 [ 605.149546] [<ffffffff81619370>] ? __switch_to_asm+0x40/0x70 [ 605.149555] [<ffffffff810158e2>] ? xen_load_sp0+0x82/0x170 [ 605.149565] [<ffffffff810a1998>] ? finish_task_switch+0x78/0x210 [ 605.149574] [<ffffffff81618df4>] ? _raw_spin_unlock_irqrestore+0x14/0x20 [ 605.149585] [<ffffffffc007bc62>] ? kjournald2+0xc2/0x260 [jbd2] [ 605.149595] [<ffffffff810bd350>] ? prepare_to_wait_event+0xf0/0xf0 [ 605.149606] [<ffffffffc007bba0>] ? commit_timeout+0x10/0x10 [jbd2] [ 605.149616] [<ffffffff8109a5d9>] ? kthread+0xd9/0xf0 [ 605.149624] [<ffffffff8109a500>] ? kthread_park+0x60/0x60 [ 605.149633] [<ffffffff816193f7>] ? ret_from_fork+0x57/0x70 [ 605.149642] INFO: task jbd2/xvda8-8:367 blocked for more than 120 seconds. [ 605.348704] Not tainted 4.9.0-8-amd64 #1 Debian 4.9.144-3.1 [ 605.348715] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 605.348727] jbd2/xvda8-8 D 0 367 2 0x00000080 [ 605.348740] ffff8800f5c3b200 0000000000000000 ffff8800f5cac740 ffff8800f7f18980 [ 605.348779] ffff880002cb0480 ffffc90040ae7b30 ffffffff816144b9 ffff8800ee926000 [ 605.348820] 00ff8800ee943200 ffff8800f7f18980 ffff88008e403f00 ffff8800f5cac740 [ 605.348854] Call Trace: [ 605.348866] [<ffffffff816144b9>] ? __schedule+0x239/0x6f0 [ 605.348879] [<ffffffff81615170>] ? bit_wait+0x50/0x50 [ 605.348900] [<ffffffff816149a2>] ? schedule+0x32/0x80 [ 605.348915] [<ffffffff81617d4d>] ? schedule_timeout+0x1dd/0x380 [ 605.348929] [<ffffffff81619370>] ? __switch_to_asm+0x40/0x70 [ 605.348939] [<ffffffff81619364>] ? __switch_to_asm+0x34/0x70 [ 605.348950] [<ffffffff8101bf91>] ? xen_clocksource_get_cycles+0x11/0x20 [ 605.348961] [<ffffffff810f107e>] ? ktime_get+0x3e/0xb0 [ 605.348971] [<ffffffff81615170>] ? bit_wait+0x50/0x50 [ 605.348980] [<ffffffff8161421d>] ? io_schedule_timeout+0x9d/0x100 [ 605.348991] [<ffffffff810bcf87>] ? prepare_to_wait+0x57/0x80 [ 605.349002] [<ffffffff81615187>] ? bit_wait_io+0x17/0x60 [ 605.349013] [<ffffffff81614d45>] ? __wait_on_bit+0x55/0x80 [ 605.349023] [<ffffffff810a1998>] ? finish_task_switch+0x78/0x210 [ 605.349037] [<ffffffff81615170>] ? bit_wait+0x50/0x50 [ 605.349046] [<ffffffff81614eae>] ? out_of_line_wait_on_bit+0x7e/0xa0 [ 605.349058] [<ffffffff810bd3f0>] ? wake_atomic_t_function+0x60/0x60 [ 605.349070] [<ffffffffc0076fe5>] ? jbd2_journal_commit_transaction+0xf55/0x17b0 [jbd2] [ 605.349083] [<ffffffff81619364>] ? __switch_to_asm+0x34/0x70 [ 605.349093] [<ffffffff81619370>] ? __switch_to_asm+0x40/0x70 [ 605.349104] [<ffffffff810a1998>] ? finish_task_switch+0x78/0x210 [ 605.349115] [<ffffffff81618df4>] ? _raw_spin_unlock_irqrestore+0x14/0x20 [ 605.349127] [<ffffffffc007bc62>] ? kjournald2+0xc2/0x260 [jbd2] [ 605.349139] [<ffffffff810bd350>] ? prepare_to_wait_event+0xf0/0xf0 [ 605.349151] [<ffffffffc007bba0>] ? commit_timeout+0x10/0x10 [jbd2] [ 605.349162] [<ffffffff8109a5d9>] ? kthread+0xd9/0xf0 [ 605.349172] [<ffffffff8109a500>] ? kthread_park+0x60/0x60 [ 605.349181] [<ffffffff816193f7>] ? ret_from_fork+0x57/0x70 [ 725.980078] INFO: task kworker/u8:1:48 blocked for more than 120 seconds. [ 725.980098] Not tainted 4.9.0-8-amd64 #1 Debian 4.9.144-3.1 [ 725.980107] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 725.980119] kworker/u8:1 D 0 48 2 0x00000080 [ 725.980135] Workqueue: writeback wb_workfn (flush-202:4) [ 725.980146] ffff8800e8252540 0000000000000000 ffff8800f2c822c0 ffff8800f7f98980 [ 725.980164] ffff8800f6c06380 ffffc900408d35d0 ffffffff816144b9 0000000000000001 [ 725.980181] 00ff8800ebbf60c0 ffff8800f7f98980 ffffffff813135d9 ffff8800f2c822c0 [ 725.980199] Call Trace: [ 725.980210] [<ffffffff816144b9>] ? __schedule+0x239/0x6f0 [ 725.980221] [<ffffffff813135d9>] ? blk_mq_flush_plug_list+0x139/0x160 [ 725.980232] [<ffffffff816149a2>] ? schedule+0x32/0x80 [ 725.980243] [<ffffffff81617d4d>] ? schedule_timeout+0x1dd/0x380 [ 725.980254] [<ffffffff813086b3>] ? blk_flush_plug_list+0xc3/0x230 [ 725.980266] [<ffffffff811861b9>] ? mempool_alloc+0x69/0x190 [ 725.980277] [<ffffffff8101bf91>] ? xen_clocksource_get_cycles+0x11/0x20 [ 725.980290] [<ffffffff8161421d>] ? io_schedule_timeout+0x9d/0x100 [ 725.980300] [<ffffffff81363964>] ? __sbitmap_queue_get+0x24/0x90 [ 725.980312] [<ffffffff81313b80>] ? bt_get.isra.6+0x160/0x220 [ 725.980323] [<ffffffff810bd350>] ? prepare_to_wait_event+0xf0/0xf0 [ 725.980335] [<ffffffff81313ef3>] ? blk_mq_get_tag+0x23/0x90 [ 725.980346] [<ffffffff8130f77a>] ? __blk_mq_alloc_request+0x1a/0x220 [ 725.980359] [<ffffffff81310649>] ? blk_mq_map_request+0xd9/0x170 [ 725.980369] [<ffffffff81312e8b>] ? blk_mq_make_request+0xbb/0x580 [ 725.980382] [<ffffffff81306d35>] ? generic_make_request+0x115/0x2d0 [ 725.980392] [<ffffffff81306f66>] ? submit_bio+0x76/0x140 [ 725.980415] [<ffffffffc01be358>] ? ext4_io_submit+0x48/0x60 [ext4] [ 725.980435] [<ffffffffc01be5c6>] ? ext4_bio_write_page+0x236/0x4d0 [ext4] [ 725.980448] [<ffffffff81190f98>] ? clear_page_dirty_for_io+0x148/0x1b0 [ 725.980465] [<ffffffffc01b3a32>] ? mpage_submit_page+0x52/0x70 [ext4] [ 725.980480] [<ffffffffc01b3c83>] ? mpage_map_and_submit_buffers+0x133/0x230 [ext4] [ 725.980502] [<ffffffffc01b8fb0>] ? ext4_writepages+0x7c0/0xd60 [ext4] [ 725.980513] [<ffffffff81618df4>] ? _raw_spin_unlock_irqrestore+0x14/0x20 [ 725.980523] [<ffffffff810e9c57>] ? mod_timer+0x177/0x3b0 [ 725.980534] [<ffffffff8123a7bd>] ? __writeback_single_inode+0x3d/0x320 [ 725.980545] [<ffffffff8123af61>] ? writeback_sb_inodes+0x221/0x4f0 [ 725.980556] [<ffffffff8123b2b7>] ? __writeback_inodes_wb+0x87/0xb0 [ 725.980567] [<ffffffff8123b62e>] ? wb_writeback+0x27e/0x310 [ 725.980579] [<ffffffff8123be5e>] ? wb_workfn+0x17e/0x380 [ 725.980588] [<ffffffff8109427a>] ? process_one_work+0x18a/0x430 [ 725.980599] [<ffffffff8109456d>] ? worker_thread+0x4d/0x490 [ 725.980609] [<ffffffff81094520>] ? process_one_work+0x430/0x430 [ 725.980620] [<ffffffff8109a5d9>] ? kthread+0xd9/0xf0 [ 725.980629] [<ffffffff8109a500>] ? kthread_park+0x60/0x60 [ 725.980639] [<ffffffff816193f7>] ? ret_from_fork+0x57/0x70 [ 725.980652] INFO: task jbd2/xvda5-8:194 blocked for more than 120 seconds. [ 725.980663] Not tainted 4.9.0-8-amd64 #1 Debian 4.9.144-3.1 [ 725.980672] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 725.980684] jbd2/xvda5-8 D 0 194 2 0x00000080 [ 725.980693] ffff8800f2dac500 0000000000000000 ffff88000298c380 ffff8800f7e18980 [ 725.980710] ffff8800f2dfa7c0 ffffc9004093bb30 ffffffff816144b9 ffffc9004093bbe8 [ 725.980729] 00ffffff81307eaf ffff8800f7e18980 0000000000000000 ffff88000298c380 [ 725.980746] Call Trace: [ 725.980756] [<ffffffff816144b9>] ? __schedule+0x239/0x6f0 [ 725.980765] [<ffffffff81615170>] ? bit_wait+0x50/0x50 [ 725.980774] [<ffffffff816149a2>] ? schedule+0x32/0x80 [ 725.980783] [<ffffffff81617d4d>] ? schedule_timeout+0x1dd/0x380 [ 725.980795] [<ffffffff810f107e>] ? ktime_get+0x3e/0xb0 [ 725.980804] [<ffffffff81615170>] ? bit_wait+0x50/0x50 [ 725.980814] [<ffffffff8161421d>] ? io_schedule_timeout+0x9d/0x100 [ 725.980824] [<ffffffff810bcf87>] ? prepare_to_wait+0x57/0x80 [ 725.980835] [<ffffffff81615187>] ? bit_wait_io+0x17/0x60 [ 725.980844] [<ffffffff81614d45>] ? __wait_on_bit+0x55/0x80 [ 725.980853] [<ffffffff81615170>] ? bit_wait+0x50/0x50 [ 725.980862] [<ffffffff81614eae>] ? out_of_line_wait_on_bit+0x7e/0xa0 [ 725.980872] [<ffffffff810bd3f0>] ? wake_atomic_t_function+0x60/0x60 [ 725.980887] [<ffffffffc0076e6e>] ? jbd2_journal_commit_transaction+0xdde/0x17b0 [jbd2] [ 725.980899] [<ffffffff81619364>] ? __switch_to_asm+0x34/0x70 [ 725.980909] [<ffffffff81619370>] ? __switch_to_asm+0x40/0x70 [ 725.980919] [<ffffffff810a1998>] ? finish_task_switch+0x78/0x210 [ 725.980929] [<ffffffff81618df4>] ? _raw_spin_unlock_irqrestore+0x14/0x20 [ 725.980941] [<ffffffffc007bc62>] ? kjournald2+0xc2/0x260 [jbd2] [ 725.980951] [<ffffffff810bd350>] ? prepare_to_wait_event+0xf0/0xf0 [ 725.980962] [<ffffffffc007bba0>] ? commit_timeout+0x10/0x10 [jbd2] [ 725.980973] [<ffffffff8109a5d9>] ? kthread+0xd9/0xf0 [ 725.980981] [<ffffffff8109a500>] ? kthread_park+0x60/0x60 [ 725.980990] [<ffffffff816193f7>] ? ret_from_fork+0x57/0x70 $ xen-diag gnttab_query_size 0 domid=0: nr_frames=1, max_nr_frames=256 $ xen-diag gnttab_query_size 51 xc: error: Could not query dom 20's grant size: Internal error $ xl sched-credit Cpupool Pool-0: tslice=30ms ratelimit=1000us migration-delay=0us Name ID Weight Cap Domain-0 0 512 0 domu1 51 256 0 If you need more information don't hesitate :) Hervé -- *Hervé Moris* System Administrator / Backend developer *Loyaltek* Cantersteen 47 1000 - Brussels Tel : +32 2 808.07.58 Fax : +32 2 706.56.84 www.loyaltek.com <http://www.loyaltek.com> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20190404/31190a36/attachment-0001.html>