CentOS virt - Apr 2017 - Testing kernel crash: 4.9.23-26.el6.x86_64

Hello CentOS Xen Heroes,

Yesterday, I have installed testing kernel 4.9.23-26.el6.x86_64 from
virt-xen-testing repo.
It crashed today morning.

Hardware is a pretty ancient, testing machine (CO6 PV guests only), but had no
problems yet. It was stable on 4.9*, including testing 4.9.15-22.el6.x86_64

Console output:

[59826.069427] general protection fault: 0000 [#1] SMP
[59826.069463] Modules linked in: xt_physdev br_netfilter xt_mac ebt_arp
xen_pciback xen_gntalloc ebtable_filter ebtables bridge 8021q mrp garp stp llc
xt_CT xt_addrtype iptable_raw nf_log_
ipv4 nf_log_common xt_LOG xt_limit xt_owner iptable_mangle iptable_nat
nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip6t_REJECT nf_reject_ipv6
nf_conntrack_ipv6 nf_defrag_ipv6 xt_sta
te nf_conntrack ip6table_filter ip6_tables blktap xen_netback xen_blkback
xen_gntdev xen_evtchn xenfs xen_privcmd ppdev parport_pc parport fjes 3c59x
asus_atk0110 pcspkr serio_raw k8temp vi
a_rhine mii i2c_viapro shpchp raid1 pata_via pata_acpi ata_generic sata_via
[59826.069842] CPU: 0 PID: 32443 Comm: sshd Not tainted 4.9.23-26.el6.x86_64 #1
[59826.069860] Hardware name: System manufacturer System Product Name/K8V-VM,
BIOS 0201    06/13/2006
[59826.069882] task: ffff880014c8c900 task.stack: ffffc90043820000
[59826.069896] RIP: e030:[<ffffffff812405d6>]  [<ffffffff812405d6>]
kmem_cache_alloc_node+0xa6/0x1c0
[59826.070019] RSP: e02b:ffffc90043823b08  EFLAGS: 00010246
[59826.070019] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000ec741e
[59826.070019] RDX: 0000000000ec741d RSI: 00000000024000c0 RDI: ffff0000ffff0000
[59826.070019] RBP: ffffc90043823b68 R08: 000000000001cd00 R09: 0000000000000000
[59826.070019] R10: 0000000000007ff0 R11: ffffc900407ebe88 R12: ffff88001a8013c0
[59826.070019] R13: ffff88001a8013c0 R14: 00000000ffffffff R15: 00000000024000c0
[59826.070019] FS:  00007f652071e7c0(0000) GS:ffff88001f800000(0000)
knlGS:0000000000000000
[59826.070019] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[59826.070019] CR2: 00007fff0870ad60 CR3: 000000000f940000 CR4: 0000000000000660
[59826.070019] Stack:
[59826.070019]  ffffc90043823b38 ffffffff818d5e36 ffff880016711758
ffffffff81792687
[59826.070019]  ffff880015d1ec10 ffff880015d1ec00 ffffc90043823c38
ffff880015d1ec00
[59826.070019]  00000000024000c0 00000000ffffffff ffff88001a8013c0
0000000000000024
[59826.070019] Call Trace:
[59826.070019]  [<ffffffff818d5e36>] ? mutex_lock+0x16/0x40
[59826.070019]  [<ffffffff81792687>] ? __alloc_skb+0x57/0x200
[59826.070019]  [<ffffffff81792687>] __alloc_skb+0x57/0x200
[59826.070019]  [<ffffffff817e2af0>] ? netlink_attachskb+0x1b0/0x1b0
[59826.070019]  [<ffffffff817df900>] ? netlink_overrun+0x50/0x50
[59826.070019]  [<ffffffff817e319c>] netlink_ack+0x6c/0x170
[59826.070019]  [<ffffffff8115526b>] audit_receive+0x8b/0xa0
[59826.070019]  [<ffffffff817e30aa>] netlink_unicast+0x19a/0x220
[59826.070019]  [<ffffffff817e3ed2>] netlink_sendmsg+0x302/0x380
[59826.070019]  [<ffffffff81787ba9>] sock_sendmsg+0x49/0x60
[59826.070019]  [<ffffffff8178a006>] SYSC_sendto+0x116/0x170
[59826.070019]  [<ffffffff81287efc>] ? __alloc_fd+0x4c/0x1b0
[59826.070019]  [<ffffffff810033c1>] ? syscall_trace_enter+0x201/0x2c0
[59826.070019]  [<ffffffff8115b199>] ? __audit_syscall_exit+0x229/0x2b0
[59826.070019]  [<ffffffff8178a06e>] SyS_sendto+0xe/0x10
[59826.070019]  [<ffffffff81003e7a>] do_syscall_64+0x7a/0x240
[59826.070019]  [<ffffffff8106f1a7>] ? do_page_fault+0x37/0x90
[59826.070019]  [<ffffffff818d83eb>] entry_SYSCALL64_slow_path+0x25/0x25
[59826.070019] Code: e8 36 41 39 c6 74 17 48 8b 4d b8 44 89 f2 44 89 fe 4c 89 e7
e8 2c f7 ff ff 48 89 c3 eb 2c 49 63 44 24 20 4d 8b 04 24 48 8d 4a 01 <48>
8b 1c 07 48 89 f8 49 8d 30 e8 eb b
4 1e 00 3c 01 75 95 49 63 
[59826.070019] RIP  [<ffffffff812405d6>] kmem_cache_alloc_node+0xa6/0x1c0
[59826.070019]  RSP <ffffc90043823b08>
[59826.076823] general protection fault: 0000 [#2] SMP
[59826.077009] Modules linked in: xt_physdev br_netfilter xt_mac ebt_arp
xen_pciback xen_gntalloc ebtable_filter ebtables bridge 8021q mrp garp stp llc
xt_CT xt_addrtype iptable_raw nf_log_
ipv4 nf_log_common xt_LOG xt_limit xt_owner iptable_mangle iptable_nat
nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip6t_REJECT nf_reject_ipv6
nf_conntrack_ipv6 nf_defrag_ipv6 xt_sta
te nf_conntrack ip6table_filter ip6_tables blktap xen_netback xen_blkback
xen_gntdev xen_evtchn xenfs xen_privcmd ppdev parport_pc parport fjes 3c59x
asus_atk0110 pcspkr serio_raw k8temp vi
a_rhine mii i2c_viapro shpchp raid1 pata_via pata_acpi ata_generic sata_via
[59826.077733] CPU: 0 PID: 32443 Comm: sshd Tainted: G      D        
4.9.23-26.el6.x86_64 #1
[59826.077733] Hardware name: System manufacturer System Product Name/K8V-VM,
BIOS 0201    06/13/2006
[59826.077733] task: ffff880014c8c900 task.stack: ffffc90043820000
[59826.077733] RIP: e030:[<ffffffff81241ad5>]  [<ffffffff81241ad5>]
kmem_cache_alloc+0x85/0x1b0
[59826.077733] RSP: e02b:ffff88001f803ba8  EFLAGS: 00010286
[59826.077733] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000ec741e
[59826.077733] RDX: 0000000000ec741d RSI: 0000000002080020 RDI: 000000000001cd00
[59826.077733] RBP: ffff88001f803bf8 R08: ffff88001f81cd00 R09: ffff0000ffff0000
[59826.077733] R10: 0000000000000003 R11: 0000000000000000 R12: ffff88001a8013c0
[59826.077733] R13: ffff88001a8013c0 R14: ffffffff8179240e R15: 0000000002080020
[59826.077733] FS:  00007f652071e7c0(0000) GS:ffff88001f800000(0000)
knlGS:0000000000000000
[59826.077733] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[59826.077733] CR2: 00007fff0870ad60 CR3: 000000000f940000 CR4: 0000000000000660
[59826.077733] Stack:
[59826.077733]  ffff88001f803bb8 ffffffff81025105 ffff88001f803bc8
ffff88001f816760
[59826.077733]  0000000000000780 0000000000000780 ffff8800144af880
0000000000000000
[59826.077733]  0000000000000200 ffff8800144af880 ffff88001f803c38
ffffffff8179240e
[59826.077733] Call Trace:
[59826.077733]  <IRQ> [59826.077733]  [<ffffffff81025105>] ?
xen_clocksource_read+0x15/0x20
[59826.077733]  [<ffffffff8179240e>] __build_skb+0x2e/0xe0
[59826.077733]  [<ffffffff817929fa>] __netdev_alloc_skb+0xda/0x120
[59826.077733]  [<ffffffff810d0a5a>] ? activate_task+0x7a/0x90
[59826.077733]  [<ffffffffc0027368>] rhine_rx+0x2b8/0x530 [via_rhine]
[59826.077733]  [<ffffffffc00276be>] rhine_napipoll+0xde/0x190 [via_rhine]
[59826.077733]  [<ffffffff817abfdb>] napi_poll+0xdb/0x1f0
[59826.077733]  [<ffffffff81119aae>] ? process_timeout+0xe/0x10
[59826.077733]  [<ffffffff81119732>] ? call_timer_fn+0x42/0x140
[59826.077733]  [<ffffffff817ac1b5>] net_rx_action+0xc5/0x240
[59826.077733]  [<ffffffff8111b1d6>] ? run_timer_softirq+0x276/0x2c0
[59826.077733]  [<ffffffff810d833c>] ? sched_clock_local+0x1c/0x90
[59826.077733]  [<ffffffff810d844a>] ? sched_clock_cpu+0x9a/0xb0
[59826.077733]  [<ffffffff818db0ce>] __do_softirq+0xce/0x2bf
[59826.077733]  [<ffffffff810d844a>] ? sched_clock_cpu+0x9a/0xb0
[59826.077733]  [<ffffffff810ab1d5>] irq_exit+0x105/0x110
[59826.077733]  [<ffffffff8150e3f9>] xen_evtchn_do_upcall+0x39/0x50
[59826.077733]  [<ffffffff818d93de>] xen_do_hypervisor_callback+0x1e/0x40
[59826.077733]  <EOI> [59826.077733]  [<ffffffff8100122a>] ?
xen_hypercall_xen_version+0xa/0x20
[59826.077733]  [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[59826.077733]  [<ffffffff81103fff>] ? console_unlock+0x2ff/0x3e0
[59826.077733]  [<ffffffff81024c5d>] ? xen_force_evtchn_callback+0xd/0x10
[59826.077733]  [<ffffffff810253b6>] ? check_events+0x16/0x30
[59826.077733]  [<ffffffff81025393>] ? xen_restore_fl_direct_reloc+0x4/0x4
[59826.077733]  [<ffffffff81032b3d>] ? oops_end+0x7d/0xe0
[59826.077733]  [<ffffffff81032cab>] ? die+0x5b/0x90
[59826.077733]  [<ffffffff81030084>] ? do_general_protection+0xe4/0x170
[59826.077733]  [<ffffffff818d97a8>] ? general_protection+0x28/0x30
[59826.077733]  [<ffffffff812405d6>] ? kmem_cache_alloc_node+0xa6/0x1c0
[59826.077733]  [<ffffffff812406c5>] ? kmem_cache_alloc_node+0x195/0x1c0
[59826.077733]  [<ffffffff818d5e36>] ? mutex_lock+0x16/0x40
[59826.077733]  [<ffffffff81792687>] ? __alloc_skb+0x57/0x200
[59826.077733]  [<ffffffff81792687>] ? __alloc_skb+0x57/0x200
[59826.077733]  [<ffffffff817e2af0>] ? netlink_attachskb+0x1b0/0x1b0
[59826.077733]  [<ffffffff817df900>] ? netlink_overrun+0x50/0x50
[59826.077733]  [<ffffffff817e319c>] ? netlink_ack+0x6c/0x170
[59826.077733]  [<ffffffff8115526b>] ? audit_receive+0x8b/0xa0
[59826.077733]  [<ffffffff817e30aa>] ? netlink_unicast+0x19a/0x220
[59826.077733]  [<ffffffff817e3ed2>] ? netlink_sendmsg+0x302/0x380
[59826.077733]  [<ffffffff81787ba9>] ? sock_sendmsg+0x49/0x60
[59826.077733]  [<ffffffff8178a006>] ? SYSC_sendto+0x116/0x170
[59826.077733]  [<ffffffff81287efc>] ? __alloc_fd+0x4c/0x1b0
[59826.077733]  [<ffffffff810033c1>] ? syscall_trace_enter+0x201/0x2c0
[59826.077733]  [<ffffffff8115b199>] ? __audit_syscall_exit+0x229/0x2b0
[59826.077733]  [<ffffffff8178a06e>] ? SyS_sendto+0xe/0x10
[59826.077733]  [<ffffffff81003e7a>] ? do_syscall_64+0x7a/0x240
[59826.077733]  [<ffffffff8106f1a7>] ? do_page_fault+0x37/0x90
[59826.077733]  [<ffffffff818d83eb>] ? entry_SYSCALL64_slow_path+0x25/0x25
[59826.077733] Code: 05 91 86 dc 7e 4d 8b 08 49 8b 40 10 4d 85 c9 0f 84 24 01 00
00 48 85 c0 0f 84 1b 01 00 00 49 63 44 24 20 49 8b 3c 24 48 8d 4a 01 <49>
8b 1c 01 4c 89 c8 48 8d 37 e8 ec 9
f 1e 00 3c 01 75 b6 49 63 
[59826.077733] RIP  [<ffffffff81241ad5>] kmem_cache_alloc+0x85/0x1b0
  [59826.077733]  RSP <ffff88001f803ba8>
[59826.100347] ---[ end trace 58d7c3379dbe9d6a ]---
[59826.100533] Kernel panic - not syncing: Fatal exception in interrupt
[59826.100721] Kernel Offset: disabled
(XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds.


Regards,

-- 
Piotr Gackiewicz
Intertele S.A. - operator system?w ITL.PL i DOMENY.ITL.PL
al. T. Rejtana 10, 35-310 Rzesz?w
TEL: +48 17 8507580, FAX: +48 17 8520275

http://www.itl.pl       - niezawodne us?ugi hostingowe
http://domeny.itl.pl    - tanie domeny internetowe
http://www.intertele.pl