Howdy,
I've got a CentOS 7 VM that occasionally becomes unresponsive. There's
a "list_add corruption" entry in /var/log/messages, included below.
This is on VMware. We have lots of other VMs which are running just fine. Only
a few are CentOS 7 VMs, though, so I can't rule out some kind of environment
issue.
It's up-to-date on OS patches, and running kernel
3.10.0-514.26.2.el7.x86_64. When it locks up, there isn't any indication on
its console, other than it looks like it's waiting at a login prompt. I did
some Googling, which suggested similar issues have been around a while, but the
results I found tended to be for CentOS 6 and older kernels.
Can anyone help me understand what this means?
Jul 31 16:58:15 den-nagios kernel: ------------[ cut here ]------------
Jul 31 16:58:15 den-nagios kernel: WARNING: at lib/list_debug.c:33
__list_add+0xac/0xc0()
Jul 31 16:58:15 den-nagios kernel: list_add corruption. prev->next should be
next (ffff880174795358), but was 64756f6c6373646e. (prev=ffff880806a44458).
Jul 31 16:58:15 den-nagios kernel: Modules linked in: dell_rbu dcdbas
ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack
ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat
nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_
mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security
iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter
vmw_vsock_vmci_transport vsock vfat fat intel_power
clamp coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw
gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon sg pcspkr vmw_vmci
shpchp i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom
ata_generic pata_acpi sd_mod crc_t10dif crct10dif_gen
eric
Jul 31 16:58:15 den-nagios kernel: vmwgfx drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops crct10dif_pclmul crct10dif_common ttm crc32c_intel
ata_piix drm serio_raw libata vmxnet3 vmw_pvscsi i2c_core fjes dm_mirror
dm_region_hash dm_log dm_mod
Jul 31 16:58:15 den-nagios kernel: CPU: 6 PID: 127439 Comm: ssh Not tainted
3.10.0-514.26.2.el7.x86_64 #1
Jul 31 16:58:15 den-nagios kernel: Hardware name: VMware, Inc. VMware7,1/440BX
Desktop Reference Platform, BIOS VMW71.00V.0.B64.1506250318 06/25/2015
Jul 31 16:58:15 den-nagios kernel: ffff880634f83c00 00000000ac63a691
ffff880634f83bb8 ffffffff81687133
Jul 31 16:58:15 den-nagios kernel: ffff880634f83bf0 ffffffff81085cb0
ffff880782e74210 ffff880174795358
Jul 31 16:58:15 den-nagios kernel: ffff880806a44458 ffff880174794000
0000000000000000 ffff880634f83c58
Jul 31 16:58:15 den-nagios kernel: Call Trace:
Jul 31 16:58:15 den-nagios kernel: [<ffffffff81687133>]
dump_stack+0x19/0x1b
Jul 31 16:58:15 den-nagios kernel: [<ffffffff81085cb0>]
warn_slowpath_common+0x70/0xb0
Jul 31 16:58:15 den-nagios kernel: [<ffffffff81085d4c>]
warn_slowpath_fmt+0x5c/0x80
Jul 31 16:58:15 den-nagios kernel: [<ffffffff815b6410>] ?
ip_fragment.constprop.54+0x90/0x90
Jul 31 16:58:15 den-nagios kernel: [<ffffffff8133399c>]
__list_add+0xac/0xc0
Jul 31 16:58:15 den-nagios kernel: [<ffffffff81096c6b>]
__internal_add_timer+0xab/0x130
Jul 31 16:58:15 den-nagios kernel: [<ffffffff81096d22>]
internal_add_timer+0x32/0x70
Jul 31 16:58:15 den-nagios kernel: [<ffffffff810992ed>]
mod_timer+0x13d/0x220
Jul 31 16:58:15 den-nagios kernel: [<ffffffff815591a8>]
sk_reset_timer+0x18/0x30
Jul 31 16:58:15 den-nagios kernel: [<ffffffff815d0cae>]
tcp_connect+0x74e/0x9f0
Jul 31 16:58:15 den-nagios kernel: [<ffffffff810eb045>] ?
ktime_get_real+0x25/0x70
Jul 31 16:58:15 den-nagios kernel: [<ffffffff81568f99>] ?
secure_tcp_sequence_number+0x69/0x90
Jul 31 16:58:15 den-nagios kernel: [<ffffffff815d5886>]
tcp_v4_connect+0x376/0x4e0
Jul 31 16:58:15 den-nagios kernel: [<ffffffff815ec685>]
__inet_stream_connect+0xb5/0x330
Jul 31 16:58:15 den-nagios kernel: [<ffffffff811b1461>] ?
handle_mm_fault+0x6b1/0x1000
Jul 31 16:58:15 den-nagios kernel: [<ffffffff815ec938>]
inet_stream_connect+0x38/0x50
Jul 31 16:58:15 den-nagios kernel: [<ffffffff81555f07>]
SYSC_connect+0xe7/0x120
Jul 31 16:58:15 den-nagios kernel: [<ffffffff8121d72d>] ?
set_close_on_exec+0x4d/0x70
Jul 31 16:58:15 den-nagios kernel: [<ffffffff81556d0e>]
SyS_connect+0xe/0x10
Jul 31 16:58:15 den-nagios kernel: [<ffffffff81697809>]
system_call_fastpath+0x16/0x1b
Jul 31 16:58:15 den-nagios kernel: ---[ end trace 64494d91939a3eb0 ]---
Thanks,
-Chris
