thr3ads.net - Linux Virtualization - [PATCH net-next V4 0/6] switch to use tx skb array in tun [Jul 2016]

If this information is useful, please help other people find it:
Share via:

Craig Gallek

2016-Jul-06 17:45 UTC

[PATCH net-next V4 0/6] switch to use tx skb array in tun

On Thu, Jun 30, 2016 at 2:45 AM, Jason Wang <jasowang at redhat.com>
wrote:> Hi all:
>
> This series tries to switch to use skb array in tun. This is used to
> eliminate the spinlock contention between producer and consumer. The
> conversion was straightforward: just introdce a tx skb array and use
> it instead of sk_receive_queue.
I'm seeing the splat below after this series.  I'm still wrapping my
head around this code, but it appears to be happening because the
tun_struct passed into tun_queue_resize is uninitialized.
Specifically, iteration over the disabled list_head fails because prev
= next = NULL.  This seems to happen when a startup script on my test
machine changes the queue length.  I'll try to figure out what's
happening, but if it's obvious to someone else from the stack, please
let me know.

[   72.322236] BUG: unable to handle kernel NULL pointer dereference
at 0000000000000010
[   72.329993] IP: [<ffffffff8153c1a0>] tun_device_event+0x110/0x340
[   72.336032] PGD 7f054f1067 PUD 7ef6f3f067 PMD 0
[   72.340616] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
[   72.345498] gsmi: Log Shutdown Reason 0x03
[   72.349541] Modules linked in: w1_therm wire cdc_acm ehci_pci
ehci_hcd mlx4_en ib_uverbs mlx4_ib ib_core mlx4_core
[   72.359870] CPU: 12 PID: 7820 Comm: set.ixion-haswe Not tainted
4.7.0-dbx-DEV #10
[   72.360253] mlx4_en: eth0: Link Up
[   72.370618] Hardware name: Intel Grantley,Wellsburg/Ixion_IT_15,
BIOS 2.50.0 01/21/2016
[   72.378525] task: ffff883f2501e8c0 ti: ffff883f3ef08000 task.ti:
ffff883f3ef08000
[   72.385917] RIP: 0010:[<ffffffff8153c1a0>]  [<ffffffff8153c1a0>]
tun_device_event+0x110/0x340
[   72.394353] RSP: 0018:ffff883f3ef0bbe8  EFLAGS: 00010202
[   72.399599] RAX: fffffffffffffae8 RBX: ffff887ef9883378 RCX: 0000000000000000
[   72.406647] RDX: 0000000000000000 RSI: 0000000000000028 RDI: 0000000000000000
[   72.413694] RBP: ffff883f3ef0bc58 R08: 0000000000000000 R09: 0000000000000001
[   72.420742] R10: 0000000000000004 R11: 0000000000000000 R12: 0000000000000010
[   72.427789] R13: 0000000000000000 R14: 0000000000000001 R15: ffff883f3ef0bd10
[   72.434837] FS:  00007fac4e5dd700(0000) GS:ffff883f7f700000(0000)
knlGS:0000000000000000
[   72.442832] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   72.448507] CR2: 0000000000000010 CR3: 0000007ef66ac000 CR4: 00000000001406e0
[   72.455555] Stack:
[   72.457541]  ffff883f3ef0bc18 0000000000000246 000000000000001e
ffff887ef9880000
[   72.464880]  ffff883f3ef0bd10 0000000000000000 0000000000000000
ffffffff00000000
[   72.472219]  ffff883f3ef0bc58 ffffffff81d24070 00000000fffffff9
ffffffff81cec7a0
[   72.479559] Call Trace:
[   72.481986]  [<ffffffff810eeb0d>] notifier_call_chain+0x5d/0x80
[   72.487844]  [<ffffffff816365c0>] ? show_tx_maxrate+0x30/0x30
[   72.493520]  [<ffffffff810eeb3e>] __raw_notifier_call_chain+0xe/0x10
[   72.499801]  [<ffffffff810eeb56>] raw_notifier_call_chain+0x16/0x20
[   72.506001]  [<ffffffff8160eb30>]
call_netdevice_notifiers_info+0x40/0x70
[   72.512706]  [<ffffffff8160ec36>] call_netdevice_notifiers+0x16/0x20
[   72.518986]  [<ffffffff816365f8>] change_tx_queue_len+0x38/0x80
[   72.524838]  [<ffffffff816381cf>] netdev_store.isra.5+0xbf/0xd0
[   72.530688]  [<ffffffff81638330>] tx_queue_len_store+0x50/0x60
[   72.536459]  [<ffffffff814a6798>] dev_attr_store+0x18/0x30
[   72.541888]  [<ffffffff812ea3ff>] sysfs_kf_write+0x4f/0x70
[   72.547306]  [<ffffffff812e9507>] kernfs_fop_write+0x147/0x1d0
[   72.553077]  [<ffffffff81134a4f>] ? rcu_read_lock_sched_held+0x8f/0xa0
[   72.559534]  [<ffffffff8125a108>] __vfs_write+0x28/0x120
[   72.564781]  [<ffffffff8111b137>] ? percpu_down_read+0x57/0x90
[   72.570542]  [<ffffffff8125d7d8>] ? __sb_start_write+0xc8/0xe0
[   72.576303]  [<ffffffff8125d7d8>] ? __sb_start_write+0xc8/0xe0
[   72.582063]  [<ffffffff8125bd5e>] vfs_write+0xbe/0x1b0
[   72.587138]  [<ffffffff8125c092>] SyS_write+0x52/0xa0
[   72.592135]  [<ffffffff817528a5>] entry_SYSCALL_64_fastpath+0x18/0xa8
[   72.598497] Code: 45 31 f6 48 8b 93 78 33 00 00 48 81 c3 78 33 00
00 48 39 d3 48 8d 82 e8 fa ff ff 74 25 48 8d b0 40 05 00 00 49 63 d6
41 83 c6 01 <49> 89 34 d4 48 8b 90 18 05 00 00 48 39 d3 48 8d 82 e8 fa
ff ff
[   72.617767] RIP  [<ffffffff8153c1a0>] tun_device_event+0x110/0x340
[   72.623883]  RSP <ffff883f3ef0bbe8>
[   72.627327] CR2: 0000000000000010
[   72.630638] ---[ end trace b0c54137cf861b91 ]---

Michael S. Tsirkin

2016-Jul-08 06:19 UTC

head link

[PATCH net-next V4 0/6] switch to use tx skb array in tun

On Wed, Jul 06, 2016 at 01:45:58PM -0400, Craig Gallek
wrote:> On Thu, Jun 30, 2016 at 2:45 AM, Jason Wang <jasowang at redhat.com>
wrote:
> > Hi all:
> >
> > This series tries to switch to use skb array in tun. This is used to
> > eliminate the spinlock contention between producer and consumer. The
> > conversion was straightforward: just introdce a tx skb array and use
> > it instead of sk_receive_queue.
> 
> I'm seeing the splat below after this series.  I'm still wrapping
my
> head around this code, but it appears to be happening because the
> tun_struct passed into tun_queue_resize is uninitialized.
> Specifically, iteration over the disabled list_head fails because prev
> = next = NULL.  This seems to happen when a startup script on my test
> machine changes the queue length.  I'll try to figure out what's
> happening, but if it's obvious to someone else from the stack, please
> let me know.

Don't see anything obvious. I'm traveling, will look at it when I'm
back
unless it's fixed by then. Jason, any idea?
> [   72.322236] BUG: unable to handle kernel NULL pointer dereference
> at 0000000000000010
> [   72.329993] IP: [<ffffffff8153c1a0>] tun_device_event+0x110/0x340
> [   72.336032] PGD 7f054f1067 PUD 7ef6f3f067 PMD 0
> [   72.340616] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
> [   72.345498] gsmi: Log Shutdown Reason 0x03
> [   72.349541] Modules linked in: w1_therm wire cdc_acm ehci_pci
> ehci_hcd mlx4_en ib_uverbs mlx4_ib ib_core mlx4_core
> [   72.359870] CPU: 12 PID: 7820 Comm: set.ixion-haswe Not tainted
> 4.7.0-dbx-DEV #10
> [   72.360253] mlx4_en: eth0: Link Up
> [   72.370618] Hardware name: Intel Grantley,Wellsburg/Ixion_IT_15,
> BIOS 2.50.0 01/21/2016
> [   72.378525] task: ffff883f2501e8c0 ti: ffff883f3ef08000 task.ti:
> ffff883f3ef08000
> [   72.385917] RIP: 0010:[<ffffffff8153c1a0>] 
[<ffffffff8153c1a0>]
> tun_device_event+0x110/0x340
> [   72.394353] RSP: 0018:ffff883f3ef0bbe8  EFLAGS: 00010202
> [   72.399599] RAX: fffffffffffffae8 RBX: ffff887ef9883378 RCX:
0000000000000000
> [   72.406647] RDX: 0000000000000000 RSI: 0000000000000028 RDI:
0000000000000000
> [   72.413694] RBP: ffff883f3ef0bc58 R08: 0000000000000000 R09:
0000000000000001
> [   72.420742] R10: 0000000000000004 R11: 0000000000000000 R12:
0000000000000010
> [   72.427789] R13: 0000000000000000 R14: 0000000000000001 R15:
ffff883f3ef0bd10
> [   72.434837] FS:  00007fac4e5dd700(0000) GS:ffff883f7f700000(0000)
> knlGS:0000000000000000
> [   72.442832] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   72.448507] CR2: 0000000000000010 CR3: 0000007ef66ac000 CR4:
00000000001406e0
> [   72.455555] Stack:
> [   72.457541]  ffff883f3ef0bc18 0000000000000246 000000000000001e
> ffff887ef9880000
> [   72.464880]  ffff883f3ef0bd10 0000000000000000 0000000000000000
> ffffffff00000000
> [   72.472219]  ffff883f3ef0bc58 ffffffff81d24070 00000000fffffff9
> ffffffff81cec7a0
> [   72.479559] Call Trace:
> [   72.481986]  [<ffffffff810eeb0d>] notifier_call_chain+0x5d/0x80
> [   72.487844]  [<ffffffff816365c0>] ? show_tx_maxrate+0x30/0x30
> [   72.493520]  [<ffffffff810eeb3e>]
__raw_notifier_call_chain+0xe/0x10
> [   72.499801]  [<ffffffff810eeb56>]
raw_notifier_call_chain+0x16/0x20
> [   72.506001]  [<ffffffff8160eb30>]
call_netdevice_notifiers_info+0x40/0x70
> [   72.512706]  [<ffffffff8160ec36>]
call_netdevice_notifiers+0x16/0x20
> [   72.518986]  [<ffffffff816365f8>] change_tx_queue_len+0x38/0x80
> [   72.524838]  [<ffffffff816381cf>] netdev_store.isra.5+0xbf/0xd0
> [   72.530688]  [<ffffffff81638330>] tx_queue_len_store+0x50/0x60
> [   72.536459]  [<ffffffff814a6798>] dev_attr_store+0x18/0x30
> [   72.541888]  [<ffffffff812ea3ff>] sysfs_kf_write+0x4f/0x70
> [   72.547306]  [<ffffffff812e9507>] kernfs_fop_write+0x147/0x1d0
> [   72.553077]  [<ffffffff81134a4f>] ?
rcu_read_lock_sched_held+0x8f/0xa0
> [   72.559534]  [<ffffffff8125a108>] __vfs_write+0x28/0x120
> [   72.564781]  [<ffffffff8111b137>] ? percpu_down_read+0x57/0x90
> [   72.570542]  [<ffffffff8125d7d8>] ? __sb_start_write+0xc8/0xe0
> [   72.576303]  [<ffffffff8125d7d8>] ? __sb_start_write+0xc8/0xe0
> [   72.582063]  [<ffffffff8125bd5e>] vfs_write+0xbe/0x1b0
> [   72.587138]  [<ffffffff8125c092>] SyS_write+0x52/0xa0
> [   72.592135]  [<ffffffff817528a5>]
entry_SYSCALL_64_fastpath+0x18/0xa8
> [   72.598497] Code: 45 31 f6 48 8b 93 78 33 00 00 48 81 c3 78 33 00
> 00 48 39 d3 48 8d 82 e8 fa ff ff 74 25 48 8d b0 40 05 00 00 49 63 d6
> 41 83 c6 01 <49> 89 34 d4 48 8b 90 18 05 00 00 48 39 d3 48 8d 82 e8
fa
> ff ff
> [   72.617767] RIP  [<ffffffff8153c1a0>] tun_device_event+0x110/0x340
> [   72.623883]  RSP <ffff883f3ef0bbe8>
> [   72.627327] CR2: 0000000000000010
> [   72.630638] ---[ end trace b0c54137cf861b91 ]---

Michael S. Tsirkin

2016-Jul-08 06:19 UTC

head link

[PATCH net-next V4 0/6] switch to use tx skb array in tun

On Wed, Jul 06, 2016 at 01:45:58PM -0400, Craig Gallek
wrote:> On Thu, Jun 30, 2016 at 2:45 AM, Jason Wang <jasowang at redhat.com>
wrote:
> > Hi all:
> >
> > This series tries to switch to use skb array in tun. This is used to
> > eliminate the spinlock contention between producer and consumer. The
> > conversion was straightforward: just introdce a tx skb array and use
> > it instead of sk_receive_queue.
> 
> I'm seeing the splat below after this series.  I'm still wrapping
my
> head around this code, but it appears to be happening because the
> tun_struct passed into tun_queue_resize is uninitialized.
> Specifically, iteration over the disabled list_head fails because prev
> = next = NULL.  This seems to happen when a startup script on my test
> machine changes the queue length.  I'll try to figure out what's
> happening, but if it's obvious to someone else from the stack, please
> let me know.

Uploading your .config might help BTW.
> [   72.322236] BUG: unable to handle kernel NULL pointer dereference
> at 0000000000000010
> [   72.329993] IP: [<ffffffff8153c1a0>] tun_device_event+0x110/0x340
> [   72.336032] PGD 7f054f1067 PUD 7ef6f3f067 PMD 0
> [   72.340616] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
> [   72.345498] gsmi: Log Shutdown Reason 0x03
> [   72.349541] Modules linked in: w1_therm wire cdc_acm ehci_pci
> ehci_hcd mlx4_en ib_uverbs mlx4_ib ib_core mlx4_core
> [   72.359870] CPU: 12 PID: 7820 Comm: set.ixion-haswe Not tainted
> 4.7.0-dbx-DEV #10
> [   72.360253] mlx4_en: eth0: Link Up
> [   72.370618] Hardware name: Intel Grantley,Wellsburg/Ixion_IT_15,
> BIOS 2.50.0 01/21/2016
> [   72.378525] task: ffff883f2501e8c0 ti: ffff883f3ef08000 task.ti:
> ffff883f3ef08000
> [   72.385917] RIP: 0010:[<ffffffff8153c1a0>] 
[<ffffffff8153c1a0>]
> tun_device_event+0x110/0x340
> [   72.394353] RSP: 0018:ffff883f3ef0bbe8  EFLAGS: 00010202
> [   72.399599] RAX: fffffffffffffae8 RBX: ffff887ef9883378 RCX:
0000000000000000
> [   72.406647] RDX: 0000000000000000 RSI: 0000000000000028 RDI:
0000000000000000
> [   72.413694] RBP: ffff883f3ef0bc58 R08: 0000000000000000 R09:
0000000000000001
> [   72.420742] R10: 0000000000000004 R11: 0000000000000000 R12:
0000000000000010
> [   72.427789] R13: 0000000000000000 R14: 0000000000000001 R15:
ffff883f3ef0bd10
> [   72.434837] FS:  00007fac4e5dd700(0000) GS:ffff883f7f700000(0000)
> knlGS:0000000000000000
> [   72.442832] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   72.448507] CR2: 0000000000000010 CR3: 0000007ef66ac000 CR4:
00000000001406e0
> [   72.455555] Stack:
> [   72.457541]  ffff883f3ef0bc18 0000000000000246 000000000000001e
> ffff887ef9880000
> [   72.464880]  ffff883f3ef0bd10 0000000000000000 0000000000000000
> ffffffff00000000
> [   72.472219]  ffff883f3ef0bc58 ffffffff81d24070 00000000fffffff9
> ffffffff81cec7a0
> [   72.479559] Call Trace:
> [   72.481986]  [<ffffffff810eeb0d>] notifier_call_chain+0x5d/0x80
> [   72.487844]  [<ffffffff816365c0>] ? show_tx_maxrate+0x30/0x30
> [   72.493520]  [<ffffffff810eeb3e>]
__raw_notifier_call_chain+0xe/0x10
> [   72.499801]  [<ffffffff810eeb56>]
raw_notifier_call_chain+0x16/0x20
> [   72.506001]  [<ffffffff8160eb30>]
call_netdevice_notifiers_info+0x40/0x70
> [   72.512706]  [<ffffffff8160ec36>]
call_netdevice_notifiers+0x16/0x20
> [   72.518986]  [<ffffffff816365f8>] change_tx_queue_len+0x38/0x80
> [   72.524838]  [<ffffffff816381cf>] netdev_store.isra.5+0xbf/0xd0
> [   72.530688]  [<ffffffff81638330>] tx_queue_len_store+0x50/0x60
> [   72.536459]  [<ffffffff814a6798>] dev_attr_store+0x18/0x30
> [   72.541888]  [<ffffffff812ea3ff>] sysfs_kf_write+0x4f/0x70
> [   72.547306]  [<ffffffff812e9507>] kernfs_fop_write+0x147/0x1d0
> [   72.553077]  [<ffffffff81134a4f>] ?
rcu_read_lock_sched_held+0x8f/0xa0
> [   72.559534]  [<ffffffff8125a108>] __vfs_write+0x28/0x120
> [   72.564781]  [<ffffffff8111b137>] ? percpu_down_read+0x57/0x90
> [   72.570542]  [<ffffffff8125d7d8>] ? __sb_start_write+0xc8/0xe0
> [   72.576303]  [<ffffffff8125d7d8>] ? __sb_start_write+0xc8/0xe0
> [   72.582063]  [<ffffffff8125bd5e>] vfs_write+0xbe/0x1b0
> [   72.587138]  [<ffffffff8125c092>] SyS_write+0x52/0xa0
> [   72.592135]  [<ffffffff817528a5>]
entry_SYSCALL_64_fastpath+0x18/0xa8
> [   72.598497] Code: 45 31 f6 48 8b 93 78 33 00 00 48 81 c3 78 33 00
> 00 48 39 d3 48 8d 82 e8 fa ff ff 74 25 48 8d b0 40 05 00 00 49 63 d6
> 41 83 c6 01 <49> 89 34 d4 48 8b 90 18 05 00 00 48 39 d3 48 8d 82 e8
fa
> ff ff
> [   72.617767] RIP  [<ffffffff8153c1a0>] tun_device_event+0x110/0x340
> [   72.623883]  RSP <ffff883f3ef0bbe8>
> [   72.627327] CR2: 0000000000000010
> [   72.630638] ---[ end trace b0c54137cf861b91 ]---

Jason Wang

2016-Jul-08 09:14 UTC

head link

[PATCH net-next V4 0/6] switch to use tx skb array in tun

On 2016?07?08? 14:19, Michael S. Tsirkin wrote:> On Wed, Jul 06, 2016 at 01:45:58PM -0400, Craig Gallek wrote:
>> >On Thu, Jun 30, 2016 at 2:45 AM, Jason Wang<jasowang at
redhat.com>  wrote:
>>> > >Hi all:
>>> > >
>>> > >This series tries to switch to use skb array in tun. This
is used to
>>> > >eliminate the spinlock contention between producer and
consumer. The
>>> > >conversion was straightforward: just introdce a tx skb
array and use
>>> > >it instead of sk_receive_queue.
>> >
>> >I'm seeing the splat below after this series.  I'm still
wrapping my
>> >head around this code, but it appears to be happening because the
>> >tun_struct passed into tun_queue_resize is uninitialized.
>> >Specifically, iteration over the disabled list_head fails because
prev
>> >= next = NULL.  This seems to happen when a startup script on my
test
>> >machine changes the queue length.  I'll try to figure out
what's
>> >happening, but if it's obvious to someone else from the stack,
please
>> >let me know.
> Don't see anything obvious. I'm traveling, will look at it when
I'm back
> unless it's fixed by then. Jason, any idea?
>
Looks like Craig has posted a fix to this:

http://patchwork.ozlabs.org/patch/645645/

Possibly Parallel Threads

Search for more apparently analagous threads

Linux Virtualization - Jul 2016 - [PATCH net-next V4 0/6] switch to use tx skb array in tun

[PATCH net-next V4 0/6] switch to use tx skb array in tun

[PATCH net-next V4 0/6] switch to use tx skb array in tun

[PATCH net-next V4 0/6] switch to use tx skb array in tun

[PATCH net-next V4 0/6] switch to use tx skb array in tun

Possibly Parallel Threads