Christian Borntraeger
2022-Oct-19 10:59 UTC
6.1-rc1 regression: virtio-net cpumask and during reboot
Michael,
as a heads-up.
I have not looked into any details yet but we do get the following during reboot
of a system on s390.
It seems to be new with 6.1-rc1 (over 6.0)
[ 8.532461] ------------[ cut here ]------------
[ 8.532497] WARNING: CPU: 8 PID: 377 at include/linux/cpumask.h:110
__netif_set_xps_queue+0x3d8/0xca8
[ 8.532507] Modules linked in: sha1_s390(+) sha_common virtio_net(+)
net_failover failover pkey zcrypt rng_core autofs4
[ 8.532528] CPU: 8 PID: 377 Comm: systemd-udevd Not tainted
6.1.0-20221018.rc1.git15.0fd5f2557625.300.fc36.s390x+debug #1
[ 8.532533] Hardware name: IBM 8561 T01 701 (KVM/Linux)
[ 8.532537] Krnl PSW : 0704e00180000000 00000000b05ec33c
(__netif_set_xps_queue+0x3dc/0xca8)
[ 8.532546] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0
RI:0 EA:3
[ 8.532552] Krnl GPRS: 00000000e7fb8b3f 0000000080000001 00000000b1870700
00000000b0ca1d3c
[ 8.532557] 0000000000000100 0000000000000300 000000008b362500
00000000b133ba48
[ 8.532561] 000000000000000c 0000038000000100 000000000000000c
0000000000000070
[ 8.532566] 0000000084cd3200 0000000000000000 00000000b05ec0c2
00000380010b77c8
[ 8.532575] Krnl Code: 00000000b05ec32e: c0e500187331 brasl
%r14,00000000b08fa990
00000000b05ec334: a7f4ff0c brc
15,00000000b05ec14c
#00000000b05ec338: af000000 mc 0,0
>00000000b05ec33c: ec76fed8007c cgij
%r7,0,6,00000000b05ec0ec
00000000b05ec342: e310f0b00004 lg
%r1,176(%r15)
00000000b05ec348: ec16ffac007c cgij
%r1,0,6,00000000b05ec2a0
00000000b05ec34e: ec680388007c cgij
%r6,0,8,00000000b05eca5e
00000000b05ec354: e310f0b80004 lg
%r1,184(%r15)
[ 8.532600] Call Trace:
[ 8.532604] [<00000000b05ec33c>] __netif_set_xps_queue+0x3dc/0xca8
[ 8.532609] ([<00000000b05ec0c2>] __netif_set_xps_queue+0x162/0xca8)
[ 8.532614] [<000003ff7fbb81ce>] virtnet_set_affinity+0x1de/0x2a0
[virtio_net]
[ 8.532622] [<000003ff7fbbb674>] virtnet_probe+0x4d4/0xc08
[virtio_net]
[ 8.532630] [<00000000b04ec4e8>] virtio_dev_probe+0x1e8/0x418
[ 8.532638] [<00000000b05350ea>] really_probe+0xd2/0x480
[ 8.532644] [<00000000b0535648>] driver_probe_device+0x40/0xf0
[ 8.532649] [<00000000b0535fac>] __driver_attach+0x10c/0x208
[ 8.532655] [<00000000b0532542>] bus_for_each_dev+0x82/0xb8
[ 8.532662] [<00000000b053422e>] bus_add_driver+0x1d6/0x260
[ 8.532667] [<00000000b0536a70>] driver_register+0xa8/0x170
[ 8.532672] [<000003ff7fbc8088>] virtio_net_driver_init+0x88/0x1000
[virtio_net]
[ 8.532680] [<00000000afb50ab0>] do_one_initcall+0x78/0x388
[ 8.532685] [<00000000afc7b5b8>] do_init_module+0x60/0x248
[ 8.532692] [<00000000afc7ce96>] __do_sys_init_module+0xbe/0xd8
[ 8.532698] [<00000000b09123b2>] __do_syscall+0x1da/0x208
[ 8.532704] [<00000000b0925b12>] system_call+0x82/0xb0
[ 8.532710] 3 locks held by systemd-udevd/377:
[ 8.532715] #0: 0000000089af5188 (&dev->mutex){....}-{3:3}, at:
__driver_attach+0xfe/0x208
[ 8.532728] #1: 00000000b14668f0 (cpu_hotplug_lock){++++}-{0:0}, at:
virtnet_probe+0x4ca/0xc08 [virtio_net]
[ 8.532744] #2: 00000000b1509d40 (xps_map_mutex){+.+.}-{3:3}, at:
__netif_set_xps_queue+0x88/0xca8
[ 8.532757] Last Breaking-Event-Address:
[ 8.532760] [<00000000b05ec0e0>] __netif_set_xps_queue+0x180/0xca8
Christian Borntraeger
2022-Oct-19 11:12 UTC
6.1-rc1 regression: virtio-net cpumask and during reboot
Am 19.10.22 um 12:59 schrieb Christian Borntraeger:> Michael, > > as a heads-up. > I have not looked into any details yet but we do get the following during reboot of a system on s390. > It seems to be new with 6.1-rc1 (over 6.0)Its not the reboot, its the boot with a kernel with debugging enabled.> > ? [??? 8.532461] ------------[ cut here ]------------ > ? [??? 8.532497] WARNING: CPU: 8 PID: 377 at include/linux/cpumask.h:110 __netif_set_xps_queue+0x3d8/0xca8 > ? [??? 8.532507] Modules linked in: sha1_s390(+) sha_common virtio_net(+) net_failover failover pkey zcrypt rng_core autofs4 > ? [??? 8.532528] CPU: 8 PID: 377 Comm: systemd-udevd Not tainted 6.1.0-20221018.rc1.git15.0fd5f2557625.300.fc36.s390x+debug #1 > ? [??? 8.532533] Hardware name: IBM 8561 T01 701 (KVM/Linux) > ? [??? 8.532537] Krnl PSW : 0704e00180000000 00000000b05ec33c (__netif_set_xps_queue+0x3dc/0xca8) > ? [??? 8.532546]??????????? R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:0 EA:3 > ? [??? 8.532552] Krnl GPRS: 00000000e7fb8b3f 0000000080000001 00000000b1870700 00000000b0ca1d3c > ? [??? 8.532557]??????????? 0000000000000100 0000000000000300 000000008b362500 00000000b133ba48 > ? [??? 8.532561]??????????? 000000000000000c 0000038000000100 000000000000000c 0000000000000070 > ? [??? 8.532566]??????????? 0000000084cd3200 0000000000000000 00000000b05ec0c2 00000380010b77c8 > ? [??? 8.532575] Krnl Code: 00000000b05ec32e: c0e500187331????? brasl?? %r14,00000000b08fa990 > ??????????????????????????? 00000000b05ec334: a7f4ff0c????????? brc???? 15,00000000b05ec14c > ?????????????????????????? #00000000b05ec338: af000000????????? mc????? 0,0 > ?????????????????????????? >00000000b05ec33c: ec76fed8007c????? cgij??? %r7,0,6,00000000b05ec0ec > ??????????????????????????? 00000000b05ec342: e310f0b00004????? lg????? %r1,176(%r15) > ??????????????????????????? 00000000b05ec348: ec16ffac007c????? cgij??? %r1,0,6,00000000b05ec2a0 > ??????????????????????????? 00000000b05ec34e: ec680388007c????? cgij??? %r6,0,8,00000000b05eca5e > ??????????????????????????? 00000000b05ec354: e310f0b80004????? lg????? %r1,184(%r15) > ? [??? 8.532600] Call Trace: > ? [??? 8.532604]? [<00000000b05ec33c>] __netif_set_xps_queue+0x3dc/0xca8 > ? [??? 8.532609] ([<00000000b05ec0c2>] __netif_set_xps_queue+0x162/0xca8) > ? [??? 8.532614]? [<000003ff7fbb81ce>] virtnet_set_affinity+0x1de/0x2a0 [virtio_net] > ? [??? 8.532622]? [<000003ff7fbbb674>] virtnet_probe+0x4d4/0xc08 [virtio_net] > ? [??? 8.532630]? [<00000000b04ec4e8>] virtio_dev_probe+0x1e8/0x418 > ? [??? 8.532638]? [<00000000b05350ea>] really_probe+0xd2/0x480 > ? [??? 8.532644]? [<00000000b0535648>] driver_probe_device+0x40/0xf0 > ? [??? 8.532649]? [<00000000b0535fac>] __driver_attach+0x10c/0x208 > ? [??? 8.532655]? [<00000000b0532542>] bus_for_each_dev+0x82/0xb8 > ? [??? 8.532662]? [<00000000b053422e>] bus_add_driver+0x1d6/0x260 > ? [??? 8.532667]? [<00000000b0536a70>] driver_register+0xa8/0x170 > ? [??? 8.532672]? [<000003ff7fbc8088>] virtio_net_driver_init+0x88/0x1000 [virtio_net] > ? [??? 8.532680]? [<00000000afb50ab0>] do_one_initcall+0x78/0x388 > ? [??? 8.532685]? [<00000000afc7b5b8>] do_init_module+0x60/0x248 > ? [??? 8.532692]? [<00000000afc7ce96>] __do_sys_init_module+0xbe/0xd8 > ? [??? 8.532698]? [<00000000b09123b2>] __do_syscall+0x1da/0x208 > ? [??? 8.532704]? [<00000000b0925b12>] system_call+0x82/0xb0 > ? [??? 8.532710] 3 locks held by systemd-udevd/377: > ? [??? 8.532715]? #0: 0000000089af5188 (&dev->mutex){....}-{3:3}, at: __driver_attach+0xfe/0x208 > ? [??? 8.532728]? #1: 00000000b14668f0 (cpu_hotplug_lock){++++}-{0:0}, at: virtnet_probe+0x4ca/0xc08 [virtio_net] > ? [??? 8.532744]? #2: 00000000b1509d40 (xps_map_mutex){+.+.}-{3:3}, at: __netif_set_xps_queue+0x88/0xca8 > ? [??? 8.532757] Last Breaking-Event-Address: > ? [??? 8.532760]? [<00000000b05ec0e0>] __netif_set_xps_queue+0x180/0xca8
Michael S. Tsirkin
2022-Oct-19 11:50 UTC
6.1-rc1 regression: virtio-net cpumask and during reboot
On Wed, Oct 19, 2022 at 12:59:58PM +0200, Christian Borntraeger wrote:> Michael, > > as a heads-up. > I have not looked into any details yet but we do get the following during reboot of a system on s390. > It seems to be new with 6.1-rc1 (over 6.0) > > [ 8.532461] ------------[ cut here ]------------ > [ 8.532497] WARNING: CPU: 8 PID: 377 at include/linux/cpumask.h:110 __netif_set_xps_queue+0x3d8/0xca8 > [ 8.532507] Modules linked in: sha1_s390(+) sha_common virtio_net(+) net_failover failover pkey zcrypt rng_core autofs4 > [ 8.532528] CPU: 8 PID: 377 Comm: systemd-udevd Not tainted 6.1.0-20221018.rc1.git15.0fd5f2557625.300.fc36.s390x+debug #1 > [ 8.532533] Hardware name: IBM 8561 T01 701 (KVM/Linux) > [ 8.532537] Krnl PSW : 0704e00180000000 00000000b05ec33c (__netif_set_xps_queue+0x3dc/0xca8) > [ 8.532546] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:0 EA:3 > [ 8.532552] Krnl GPRS: 00000000e7fb8b3f 0000000080000001 00000000b1870700 00000000b0ca1d3c > [ 8.532557] 0000000000000100 0000000000000300 000000008b362500 00000000b133ba48 > [ 8.532561] 000000000000000c 0000038000000100 000000000000000c 0000000000000070 > [ 8.532566] 0000000084cd3200 0000000000000000 00000000b05ec0c2 00000380010b77c8 > [ 8.532575] Krnl Code: 00000000b05ec32e: c0e500187331 brasl %r14,00000000b08fa990 > 00000000b05ec334: a7f4ff0c brc 15,00000000b05ec14c > #00000000b05ec338: af000000 mc 0,0 > >00000000b05ec33c: ec76fed8007c cgij %r7,0,6,00000000b05ec0ec > 00000000b05ec342: e310f0b00004 lg %r1,176(%r15) > 00000000b05ec348: ec16ffac007c cgij %r1,0,6,00000000b05ec2a0 > 00000000b05ec34e: ec680388007c cgij %r6,0,8,00000000b05eca5e > 00000000b05ec354: e310f0b80004 lg %r1,184(%r15) > [ 8.532600] Call Trace: > [ 8.532604] [<00000000b05ec33c>] __netif_set_xps_queue+0x3dc/0xca8 > [ 8.532609] ([<00000000b05ec0c2>] __netif_set_xps_queue+0x162/0xca8) > [ 8.532614] [<000003ff7fbb81ce>] virtnet_set_affinity+0x1de/0x2a0 [virtio_net] > [ 8.532622] [<000003ff7fbbb674>] virtnet_probe+0x4d4/0xc08 [virtio_net] > [ 8.532630] [<00000000b04ec4e8>] virtio_dev_probe+0x1e8/0x418 > [ 8.532638] [<00000000b05350ea>] really_probe+0xd2/0x480 > [ 8.532644] [<00000000b0535648>] driver_probe_device+0x40/0xf0 > [ 8.532649] [<00000000b0535fac>] __driver_attach+0x10c/0x208 > [ 8.532655] [<00000000b0532542>] bus_for_each_dev+0x82/0xb8 > [ 8.532662] [<00000000b053422e>] bus_add_driver+0x1d6/0x260 > [ 8.532667] [<00000000b0536a70>] driver_register+0xa8/0x170 > [ 8.532672] [<000003ff7fbc8088>] virtio_net_driver_init+0x88/0x1000 [virtio_net] > [ 8.532680] [<00000000afb50ab0>] do_one_initcall+0x78/0x388 > [ 8.532685] [<00000000afc7b5b8>] do_init_module+0x60/0x248 > [ 8.532692] [<00000000afc7ce96>] __do_sys_init_module+0xbe/0xd8 > [ 8.532698] [<00000000b09123b2>] __do_syscall+0x1da/0x208 > [ 8.532704] [<00000000b0925b12>] system_call+0x82/0xb0 > [ 8.532710] 3 locks held by systemd-udevd/377: > [ 8.532715] #0: 0000000089af5188 (&dev->mutex){....}-{3:3}, at: __driver_attach+0xfe/0x208 > [ 8.532728] #1: 00000000b14668f0 (cpu_hotplug_lock){++++}-{0:0}, at: virtnet_probe+0x4ca/0xc08 [virtio_net] > [ 8.532744] #2: 00000000b1509d40 (xps_map_mutex){+.+.}-{3:3}, at: __netif_set_xps_queue+0x88/0xca8 > [ 8.532757] Last Breaking-Event-Address: > [ 8.532760] [<00000000b05ec0e0>] __netif_set_xps_queue+0x180/0xca8Does this fix it for you? https://lore.kernel.org/r/20221017030947.1295426-1-yury.norov%40gmail.com -- MST