thr3ads.net - Linux Virtualization - virtio balloon: do not call blocking ops when !TASK

If this information is useful, please help other people find it:
Share via:

Thomas Huth

2015-Feb-25 10:13 UTC

virtio balloon: do not call blocking ops when !TASK_RUNNING

Hi all,

with the recent kernel 3.19, I get a kernel warning when I start my
KVM guest on s390 with virtio balloon enabled:

[    0.839687] do not call blocking ops when !TASK_RUNNING; state=1 set at
               [<0000000000174a1e>] prepare_to_wait_event+0x7e/0x108
[    0.839694] ------------[ cut here ]------------
[    0.839697] WARNING: at kernel/sched/core.c:7326
[    0.839698] Modules linked in:
[    0.839702] CPU: 0 PID: 46 Comm: vballoon Not tainted 3.19.0 #233
[    0.839705] task: 00000000021d0000 ti: 00000000021d8000 task.ti:
00000000021d8000
[    0.839707] Krnl PSW : 0704c00180000000 000000000015bf8e
(__might_sleep+0x8e/0x98)
[    0.839713]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0
EA:3
Krnl GPRS: 000000000000000d 00000000021d0000 0000000000000071 0000000000000001
[    0.839718]            0000000000675ace 0000000001998c50 cccccccccccccccc
cccccccccccccccc
[    0.839720]            0000000000982134 000000000058f824 0000000000a008a8
0000000000000000
[    0.839722]            00000000000004d9 00000000007ea992 000000000015bf8a
00000000021dbc28
[    0.839731] Krnl Code: 000000000015bf7e: c0200033e838	larl	%r2,7d8fee
           000000000015bf84: c0e50028cd62	brasl	%r14,675a48
          #000000000015bf8a: a7f40001		brc	15,15bf8c
          >000000000015bf8e: 9201a000		mvi	0(%r10),1
           000000000015bf92: a7f4ffe2		brc	15,15bf56
           000000000015bf96: 0707		bcr	0,%r7
           000000000015bf98: ebdff0800024	stmg	%r13,%r15,128(%r15)
           000000000015bf9e: a7f13fe0		tmll	%r15,16352
[    0.839749] Call Trace:
[    0.839751] ([<000000000015bf8a>] __might_sleep+0x8a/0x98)
[    0.839756]  [<000000000028a562>] __kmalloc+0x272/0x350
[    0.839759]  [<000000000058f824>] virtio_ccw_get_config+0x3c/0x100
[    0.839762]  [<000000000049fcb0>] balloon+0x1b8/0x330
[    0.839765]  [<00000000001529c8>] kthread+0x120/0x138
[    0.839767]  [<0000000000683c22>] kernel_thread_starter+0x6/0xc
[    0.839770]  [<0000000000683c1c>] kernel_thread_starter+0x0/0xc
[    0.839772] no locks held by vballoon/46.
[    0.839773] Last Breaking-Event-Address:
[    0.839776]  [<000000000015bf8a>] __might_sleep+0x8a/0x98
[    0.839778] ---[ end trace d27fcdfa27273d7c ]---

The problem seems to be this code in balloon() in
drivers/virtio/virtio_balloon.c:

	wait_event_interruptible(vb->config_change,
				 (diff = towards_target(vb)) != 0
				 || vb->need_stats_update
				 || kthread_should_stop()
				 || freezing(current));

wait_event_interruptible() sets the state of the current task to
TASK_INTERRUPTIBLE, then checks the condition. The condition contains
towards_target() which reads the virtio config space via virtio_cread().
On s390, this then triggers virtio_ccw_get_config() - and this function
calls some other functions again that might sleep (e.g. kzalloc or
wait_event in ccw_io_helper) ... and this causes the new kernel warning
message with kernel 3.19.

I think it would be quite difficult or at least ugly to rewrite
virtio_ccw_get_config() so that it does not call sleepable functions
anymore. So would it be feasible to rewrite the balloon() function that
it does not call the towards_target() in its wait_event condition
anymore? I am unfortunately not that familiar with the balloon code
semantics, so any help is very appreciated here!

 Thanks,
  Thomas

Cornelia Huck

2015-Feb-25 11:09 UTC

head link

virtio balloon: do not call blocking ops when !TASK_RUNNING

On Wed, 25 Feb 2015 11:13:18 +0100
Thomas Huth <thuth at linux.vnet.ibm.com> wrote:
> 
>  Hi all,
> 
> with the recent kernel 3.19, I get a kernel warning when I start my
> KVM guest on s390 with virtio balloon enabled:
> 
> [    0.839687] do not call blocking ops when !TASK_RUNNING; state=1 set at
>                [<0000000000174a1e>] prepare_to_wait_event+0x7e/0x108
> [    0.839694] ------------[ cut here ]------------
> [    0.839697] WARNING: at kernel/sched/core.c:7326
> [    0.839698] Modules linked in:
> [    0.839702] CPU: 0 PID: 46 Comm: vballoon Not tainted 3.19.0 #233
> [    0.839705] task: 00000000021d0000 ti: 00000000021d8000 task.ti:
00000000021d8000
> [    0.839707] Krnl PSW : 0704c00180000000 000000000015bf8e
(__might_sleep+0x8e/0x98)
> [    0.839713]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0
PM:0 EA:3
> Krnl GPRS: 000000000000000d 00000000021d0000 0000000000000071
0000000000000001
> [    0.839718]            0000000000675ace 0000000001998c50
cccccccccccccccc cccccccccccccccc
> [    0.839720]            0000000000982134 000000000058f824
0000000000a008a8 0000000000000000
> [    0.839722]            00000000000004d9 00000000007ea992
000000000015bf8a 00000000021dbc28
> [    0.839731] Krnl Code: 000000000015bf7e: c0200033e838	larl	%r2,7d8fee
>            000000000015bf84: c0e50028cd62	brasl	%r14,675a48
>           #000000000015bf8a: a7f40001		brc	15,15bf8c
>           >000000000015bf8e: 9201a000		mvi	0(%r10),1
>            000000000015bf92: a7f4ffe2		brc	15,15bf56
>            000000000015bf96: 0707		bcr	0,%r7
>            000000000015bf98: ebdff0800024	stmg	%r13,%r15,128(%r15)
>            000000000015bf9e: a7f13fe0		tmll	%r15,16352
> [    0.839749] Call Trace:
> [    0.839751] ([<000000000015bf8a>] __might_sleep+0x8a/0x98)
> [    0.839756]  [<000000000028a562>] __kmalloc+0x272/0x350
> [    0.839759]  [<000000000058f824>] virtio_ccw_get_config+0x3c/0x100
> [    0.839762]  [<000000000049fcb0>] balloon+0x1b8/0x330
> [    0.839765]  [<00000000001529c8>] kthread+0x120/0x138
> [    0.839767]  [<0000000000683c22>] kernel_thread_starter+0x6/0xc
> [    0.839770]  [<0000000000683c1c>] kernel_thread_starter+0x0/0xc
> [    0.839772] no locks held by vballoon/46.
> [    0.839773] Last Breaking-Event-Address:
> [    0.839776]  [<000000000015bf8a>] __might_sleep+0x8a/0x98
> [    0.839778] ---[ end trace d27fcdfa27273d7c ]---
> 
> The problem seems to be this code in balloon() in
> drivers/virtio/virtio_balloon.c:
> 
> 	wait_event_interruptible(vb->config_change,
> 				 (diff = towards_target(vb)) != 0
> 				 || vb->need_stats_update
> 				 || kthread_should_stop()
> 				 || freezing(current));
> 
> wait_event_interruptible() sets the state of the current task to
> TASK_INTERRUPTIBLE, then checks the condition. The condition contains
> towards_target() which reads the virtio config space via virtio_cread().
> On s390, this then triggers virtio_ccw_get_config() - and this function
> calls some other functions again that might sleep (e.g. kzalloc or
> wait_event in ccw_io_helper) ... and this causes the new kernel warning
> message with kernel 3.19.
> 
> I think it would be quite difficult or at least ugly to rewrite
> virtio_ccw_get_config() so that it does not call sleepable functions
> anymore.
Yes: The config-space interacting functions for virtio-ccw trigger
channel I/O, which is by nature asynchronous. No way to get this
non-sleeping without really ugly hacks.
> So would it be feasible to rewrite the balloon() function that
> it does not call the towards_target() in its wait_event condition
> anymore? I am unfortunately not that familiar with the balloon code
> semantics, so any help is very appreciated here!
It might be possible to use nested wait event functions like
wake_woken(), but I haven't looked into that deeply.

Michael S. Tsirkin

2015-Feb-25 14:17 UTC

head link

virtio balloon: do not call blocking ops when !TASK_RUNNING

On Wed, Feb 25, 2015 at 11:13:18AM +0100, Thomas Huth
wrote:> 
>  Hi all,
> 
> with the recent kernel 3.19, I get a kernel warning when I start my
> KVM guest on s390 with virtio balloon enabled:
> 
> [    0.839687] do not call blocking ops when !TASK_RUNNING; state=1 set at
>                [<0000000000174a1e>] prepare_to_wait_event+0x7e/0x108
> [    0.839694] ------------[ cut here ]------------
> [    0.839697] WARNING: at kernel/sched/core.c:7326
> [    0.839698] Modules linked in:
> [    0.839702] CPU: 0 PID: 46 Comm: vballoon Not tainted 3.19.0 #233
> [    0.839705] task: 00000000021d0000 ti: 00000000021d8000 task.ti:
00000000021d8000
> [    0.839707] Krnl PSW : 0704c00180000000 000000000015bf8e
(__might_sleep+0x8e/0x98)
> [    0.839713]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0
PM:0 EA:3
> Krnl GPRS: 000000000000000d 00000000021d0000 0000000000000071
0000000000000001
> [    0.839718]            0000000000675ace 0000000001998c50
cccccccccccccccc cccccccccccccccc
> [    0.839720]            0000000000982134 000000000058f824
0000000000a008a8 0000000000000000
> [    0.839722]            00000000000004d9 00000000007ea992
000000000015bf8a 00000000021dbc28
> [    0.839731] Krnl Code: 000000000015bf7e: c0200033e838	larl	%r2,7d8fee
>            000000000015bf84: c0e50028cd62	brasl	%r14,675a48
>           #000000000015bf8a: a7f40001		brc	15,15bf8c
>           >000000000015bf8e: 9201a000		mvi	0(%r10),1
>            000000000015bf92: a7f4ffe2		brc	15,15bf56
>            000000000015bf96: 0707		bcr	0,%r7
>            000000000015bf98: ebdff0800024	stmg	%r13,%r15,128(%r15)
>            000000000015bf9e: a7f13fe0		tmll	%r15,16352
> [    0.839749] Call Trace:
> [    0.839751] ([<000000000015bf8a>] __might_sleep+0x8a/0x98)
> [    0.839756]  [<000000000028a562>] __kmalloc+0x272/0x350
> [    0.839759]  [<000000000058f824>] virtio_ccw_get_config+0x3c/0x100
> [    0.839762]  [<000000000049fcb0>] balloon+0x1b8/0x330
> [    0.839765]  [<00000000001529c8>] kthread+0x120/0x138
> [    0.839767]  [<0000000000683c22>] kernel_thread_starter+0x6/0xc
> [    0.839770]  [<0000000000683c1c>] kernel_thread_starter+0x0/0xc
> [    0.839772] no locks held by vballoon/46.
> [    0.839773] Last Breaking-Event-Address:
> [    0.839776]  [<000000000015bf8a>] __might_sleep+0x8a/0x98
> [    0.839778] ---[ end trace d27fcdfa27273d7c ]---
> 
> The problem seems to be this code in balloon() in
> drivers/virtio/virtio_balloon.c:
> 
> 	wait_event_interruptible(vb->config_change,
> 				 (diff = towards_target(vb)) != 0
> 				 || vb->need_stats_update
> 				 || kthread_should_stop()
> 				 || freezing(current));
> 
> wait_event_interruptible() sets the state of the current task to
> TASK_INTERRUPTIBLE, then checks the condition. The condition contains
> towards_target() which reads the virtio config space via virtio_cread().
> On s390, this then triggers virtio_ccw_get_config() - and this function
> calls some other functions again that might sleep (e.g. kzalloc or
> wait_event in ccw_io_helper) ... and this causes the new kernel warning
> message with kernel 3.19.
> 
> I think it would be quite difficult or at least ugly to rewrite
> virtio_ccw_get_config() so that it does not call sleepable functions
> anymore. So would it be feasible to rewrite the balloon() function that
> it does not call the towards_target() in its wait_event condition
> anymore? I am unfortunately not that familiar with the balloon code
> semantics, so any help is very appreciated here!
> 
>  Thanks,
>   Thomas
Thanks for finding this!
I just sent a patch that should fix this problem:
	http://article.gmane.org/gmane.linux.kernel.virtualization/24851
Testing would be appreciated.

Thanks again!

-- 
MST

Rusty Russell

2015-Feb-26 01:20 UTC

head link

virtio balloon: do not call blocking ops when !TASK_RUNNING

Thomas Huth <thuth at linux.vnet.ibm.com> writes:>  Hi all,
>
> with the recent kernel 3.19, I get a kernel warning when I start my
> KVM guest on s390 with virtio balloon enabled:
The deeper problem is that virtio_ccw_get_config just silently fails on
OOM.

Neither get_config nor set_config are expected to fail.

Cornelia, I think ccw and config_area should be allocated inside vcdev.
You could either use pointers, or simply allocate vcdev with GDP_DMA.

This would avoid the kmalloc inside these calls.

Thanks,
Rusty.
>
> [    0.839687] do not call blocking ops when !TASK_RUNNING; state=1 set at
>                [<0000000000174a1e>] prepare_to_wait_event+0x7e/0x108
> [    0.839694] ------------[ cut here ]------------
> [    0.839697] WARNING: at kernel/sched/core.c:7326
> [    0.839698] Modules linked in:
> [    0.839702] CPU: 0 PID: 46 Comm: vballoon Not tainted 3.19.0 #233
> [    0.839705] task: 00000000021d0000 ti: 00000000021d8000 task.ti:
00000000021d8000
> [    0.839707] Krnl PSW : 0704c00180000000 000000000015bf8e
(__might_sleep+0x8e/0x98)
> [    0.839713]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0
PM:0 EA:3
> Krnl GPRS: 000000000000000d 00000000021d0000 0000000000000071
0000000000000001
> [    0.839718]            0000000000675ace 0000000001998c50
cccccccccccccccc cccccccccccccccc
> [    0.839720]            0000000000982134 000000000058f824
0000000000a008a8 0000000000000000
> [    0.839722]            00000000000004d9 00000000007ea992
000000000015bf8a 00000000021dbc28
> [    0.839731] Krnl Code: 000000000015bf7e: c0200033e838	larl	%r2,7d8fee
>            000000000015bf84: c0e50028cd62	brasl	%r14,675a48
>           #000000000015bf8a: a7f40001		brc	15,15bf8c
>           >000000000015bf8e: 9201a000		mvi	0(%r10),1
>            000000000015bf92: a7f4ffe2		brc	15,15bf56
>            000000000015bf96: 0707		bcr	0,%r7
>            000000000015bf98: ebdff0800024	stmg	%r13,%r15,128(%r15)
>            000000000015bf9e: a7f13fe0		tmll	%r15,16352
> [    0.839749] Call Trace:
> [    0.839751] ([<000000000015bf8a>] __might_sleep+0x8a/0x98)
> [    0.839756]  [<000000000028a562>] __kmalloc+0x272/0x350
> [    0.839759]  [<000000000058f824>] virtio_ccw_get_config+0x3c/0x100
> [    0.839762]  [<000000000049fcb0>] balloon+0x1b8/0x330
> [    0.839765]  [<00000000001529c8>] kthread+0x120/0x138
> [    0.839767]  [<0000000000683c22>] kernel_thread_starter+0x6/0xc
> [    0.839770]  [<0000000000683c1c>] kernel_thread_starter+0x0/0xc
> [    0.839772] no locks held by vballoon/46.
> [    0.839773] Last Breaking-Event-Address:
> [    0.839776]  [<000000000015bf8a>] __might_sleep+0x8a/0x98
> [    0.839778] ---[ end trace d27fcdfa27273d7c ]---
>
> The problem seems to be this code in balloon() in
> drivers/virtio/virtio_balloon.c:
>
> 	wait_event_interruptible(vb->config_change,
> 				 (diff = towards_target(vb)) != 0
> 				 || vb->need_stats_update
> 				 || kthread_should_stop()
> 				 || freezing(current));
>
> wait_event_interruptible() sets the state of the current task to
> TASK_INTERRUPTIBLE, then checks the condition. The condition contains
> towards_target() which reads the virtio config space via virtio_cread().
> On s390, this then triggers virtio_ccw_get_config() - and this function
> calls some other functions again that might sleep (e.g. kzalloc or
> wait_event in ccw_io_helper) ... and this causes the new kernel warning
> message with kernel 3.19.
>
> I think it would be quite difficult or at least ugly to rewrite
> virtio_ccw_get_config() so that it does not call sleepable functions
> anymore. So would it be feasible to rewrite the balloon() function that
> it does not call the towards_target() in its wait_event condition
> anymore? I am unfortunately not that familiar with the balloon code
> semantics, so any help is very appreciated here!
>
>  Thanks,
>   Thomas

Thomas Huth

2015-Feb-26 07:36 UTC

head link

virtio balloon: do not call blocking ops when !TASK_RUNNING

On Thu, 26 Feb 2015 11:50:42 +1030
Rusty Russell <rusty at rustcorp.com.au> wrote:
> Thomas Huth <thuth at linux.vnet.ibm.com> writes:
> >  Hi all,
> >
> > with the recent kernel 3.19, I get a kernel warning when I start my
> > KVM guest on s390 with virtio balloon enabled:
> 
> The deeper problem is that virtio_ccw_get_config just silently fails on
> OOM.
> 
> Neither get_config nor set_config are expected to fail.
AFAIK this is currently not a problem. According to
http://lwn.net/Articles/627419/ these kmalloc calls never
fail because they allocate less than a page.

 Thomas

Michael S. Tsirkin

2015-Feb-26 08:30 UTC

head link

virtio balloon: do not call blocking ops when !TASK_RUNNING

On Thu, Feb 26, 2015 at 11:50:42AM +1030, Rusty Russell
wrote:> Thomas Huth <thuth at linux.vnet.ibm.com> writes:
> >  Hi all,
> >
> > with the recent kernel 3.19, I get a kernel warning when I start my
> > KVM guest on s390 with virtio balloon enabled:
> 
> The deeper problem is that virtio_ccw_get_config just silently fails on
> OOM.
> 
> Neither get_config nor set_config are expected to fail.
> 
> Cornelia, I think ccw and config_area should be allocated inside vcdev.
> You could either use pointers, or simply allocate vcdev with GDP_DMA.
> 
> This would avoid the kmalloc inside these calls.
> 
> Thanks,
> Rusty.
But it won't solve the problem of nested sleepers
with ccw: ATM is invokes ccw_io_helper to execute
commands, and that one calls wait_event
to wait for an interrupt.

Might be fixable but I think my patch looks like a safer
solution for 4.0/3.19, no?

-- 
MST

Michael S. Tsirkin

2015-Feb-26 08:45 UTC

head link

virtio balloon: do not call blocking ops when !TASK_RUNNING

On Thu, Feb 26, 2015 at 11:50:42AM +1030, Rusty Russell
wrote:> Thomas Huth <thuth at linux.vnet.ibm.com> writes:
> >  Hi all,
> >
> > with the recent kernel 3.19, I get a kernel warning when I start my
> > KVM guest on s390 with virtio balloon enabled:
> 
> The deeper problem is that virtio_ccw_get_config just silently fails on
> OOM.
Same problem with virtio_ccw_reset.
But avoiding kmalloc calls in virtio_ccw_get_config isn't enough I think,
it might still sleep.

> 
> Neither get_config nor set_config are expected to fail.
> 
> Cornelia, I think ccw and config_area should be allocated inside vcdev.
> You could either use pointers, or simply allocate vcdev with GDP_DMA.
> 
> This would avoid the kmalloc inside these calls.
> 
> Thanks,
> Rusty.

Cornelia Huck

2015-Feb-26 08:47 UTC

head link

virtio balloon: do not call blocking ops when !TASK_RUNNING

On Thu, 26 Feb 2015 11:50:42 +1030
Rusty Russell <rusty at rustcorp.com.au> wrote:
> Thomas Huth <thuth at linux.vnet.ibm.com> writes:
> >  Hi all,
> >
> > with the recent kernel 3.19, I get a kernel warning when I start my
> > KVM guest on s390 with virtio balloon enabled:
> 
> The deeper problem is that virtio_ccw_get_config just silently fails on
> OOM.
> 
> Neither get_config nor set_config are expected to fail.
It is a problem that we cannot relay failures back to the caller: not
only for the memory allocations. We need to do channel I/O, and any
channel I/O can fail. For our virtio case, we don't have to deal with
the failures that may happen on real hardware (like path failures), but
what can happen is a hotunplug, which means we cannot talk to the
device anymore from one moment to the other.
> 
> Cornelia, I think ccw and config_area should be allocated inside vcdev.
> You could either use pointers, or simply allocate vcdev with GDP_DMA.
> 
> This would avoid the kmalloc inside these calls.
I can certainly look into that, but I'm not sure it's worth it. We still
have to deal with possible failures from doing channel I/O.

Seemingly Similar Threads

Search for more seemingly similar threads

Linux Virtualization - Feb 2015 - virtio balloon: do not call blocking ops when !TASK_RUNNING

virtio balloon: do not call blocking ops when !TASK_RUNNING

virtio balloon: do not call blocking ops when !TASK_RUNNING

virtio balloon: do not call blocking ops when !TASK_RUNNING

virtio balloon: do not call blocking ops when !TASK_RUNNING

virtio balloon: do not call blocking ops when !TASK_RUNNING

virtio balloon: do not call blocking ops when !TASK_RUNNING

virtio balloon: do not call blocking ops when !TASK_RUNNING

virtio balloon: do not call blocking ops when !TASK_RUNNING

Seemingly Similar Threads