Hi Jim,
? ? ? Have you tried to update the kernel as suggested by Changwei??
? ? ? The messages seem to indicate kernel 4.4.0, a google search shows this
ubuntu version should be able to use kernel 4.10.
Best Regards,Luis? ? ??
On Wednesday, January 10, 2018 4:12 PM, Jim Okken <jim at jokken.com>
wrote:
hello again list,
We seem to be having issues on more servers where according to the linux
developers here: "the kernel is stuck in a spin lock during a disk
operation."
The call traces are below, I see a lot of ocfs in the call traces, but I
don't know how to read them, please tell me does the issue come from
ocfs?thanks?--Jim
2018-01-06T17:10:02.194362+00:00 node-115 kernel: [87885.155288] Modules linked
in: vhost_net vhost macvtap macvlan ip6table_raw xt_mac xt_tcpudp xt_physdev
br_netfilter veth ebtable_filter ebtables openvswitch ocfs2 quota_tree
ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue
configfs ip6table_filter ip6_tables xt_multiport xt_conntrack iptable_filter
xt_comment xt_CT iptable_raw ip_tables x_tables xfs bridge 8021q garp mrp stp
llc intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul
ipmi_ssif crc32_pclmul ghash_clmulni_intel kvm_intel aesni_intel aes_x86_64 kvm
lrw gf128mul glue_helper ablk_helper irqbypass cryptd hpilo 8250_fintek
serio_raw ioatdma ipmi_si sb_edac edac_core ipmi_msghandler shpchp dca
acpi_power_meter lpc_ich mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad
ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
nf_conntrack_proto_gre nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4
nf_defrag_ipv4 nf_conntrack autofs4 raid10 raid456 async_raid6_recov
async_memcpy async_pq async_xor async_tx xor dm_round_robin ses enclosure
scsi_transport_sas raid6_pq libcrc32c raid1 raid0 multipath linear uas
usb_storage psmouse lpfc be2net vxlan ip6_udp_tunnel scsi_transport_fc
udp_tunnel wmi fjes scsi_dh_emc scsi_dh_rdac scsi_dh_alua
dm_multipath2018-01-06T17:10:02.194364+00:00 node-115 kernel: [87885.157143]
CPU: 15 PID: 11936 Comm: qemu-system-x86 Not tainted 4.4.0-98-generic
#121-Ubuntu2018-01-06T17:10:02.194366+00:00 node-115 kernel: [87885.157144]
Hardware name: HP ProLiant BL460c Gen9, BIOS I36
02/17/20172018-01-06T17:10:02.194367+00:00 node-115 kernel: [87885.157280] task:
ffff882036ff0000 ti: ffff881f80ca0000 task.ti:
ffff881f80ca00002018-01-06T17:10:02.194400+00:00 node-115 kernel: [87885.157281]
RIP: 0010:[<ffffffff810cb27c>]? [<ffffffff810cb27c>]
native_queued_spin_lock_slowpath+0x15c/0x1702018-01-06T17:10:02.194414+00:00
node-115 kernel: [87885.157566] RSP: 0018:ffff88203f143c30? EFLAGS:
000002022018-01-06T17:10:02.194416+00:00 node-115 kernel: [87885.157567] RAX:
0000000000000101 RBX: ffff8820046c83f0 RCX:
00000000000000012018-01-06T17:10:02.194418+00:00 node-115 kernel: [87885.157705]
RDX: 0000000000000101 RSI: 0000000000000001 RDI:
ffff8820046c83ec2018-01-06T17:10:02.194440+00:00 node-115 kernel: [87885.157705]
RBP: ffff88203f143c30 R08: 0000000000000101 R09:
ffffffff811924a72018-01-06T17:10:02.194442+00:00 node-115 kernel: [87885.157706]
R10: ffffea0040d6d680 R11: 0000000000000800 R12:
ffff8820046c83ec2018-01-06T17:10:02.194443+00:00 node-115 kernel: [87885.157707]
R13: 0000000000000800 R14: 000000004c63ee00 R15:
00000000000008002018-01-06T17:10:02.194444+00:00 node-115 kernel: [87885.157708]
FS:? 00007fbcbb7eec00(0000) GS:ffff88203f140000(0000)
knlGS:00000000000000002018-01-06T17:10:02.194444+00:00 node-115 kernel:
[87885.157709] CS:? 0010 DS: 0000 ES: 0000 CR0:
00000000800500332018-01-06T17:10:02.194445+00:00 node-115 kernel: [87885.157710]
CR2: 00007f54266a8000 CR3: 0000000fcc2f2000 CR4:
00000000001426e02018-01-06T17:10:02.194446+00:00 node-115 kernel: [87885.157711]
Stack:2018-01-06T17:10:02.194448+00:00 node-115 kernel: [87885.157712]?
ffff88203f143c40 ffffffff81844421 ffff88203f143c60
ffffffff818425352018-01-06T17:10:02.194449+00:00 node-115 kernel:
[87885.157714]? ffff881e88a9ca80 ffff8820046c84b0 ffff88203f143c70
ffffffff8184257b2018-01-06T17:10:02.194450+00:00 node-115 kernel:
[87885.157716]? ffff88203f143ca0 ffffffffc074158d ffff881e5d3beb80
00000000000008002018-01-06T17:10:02.194450+00:00 node-115 kernel: [87885.157717]
Call Trace:2018-01-06T17:10:02.194451+00:00 node-115 kernel: [87885.157718]?
<IRQ>2018-01-06T17:10:02.194453+00:00 node-115 kernel: [87885.157725]?
[<ffffffff81844421>]
_raw_spin_lock+0x21/0x302018-01-06T17:10:02.194454+00:00 node-115 kernel:
[87885.157727]? [<ffffffff81842535>]
__mutex_unlock_slowpath+0x25/0x502018-01-06T17:10:02.194456+00:00 node-115
kernel: [87885.157729]? [<ffffffff8184257b>]
mutex_unlock+0x1b/0x202018-01-06T17:10:02.194457+00:00 node-115 kernel:
[87885.157766]? [<ffffffffc074158d>] ocfs2_dio_end_io+0x6d/0x80
[ocfs2]2018-01-06T17:10:02.194458+00:00 node-115 kernel: [87885.157770]?
[<ffffffff8124e57c>]
dio_complete+0x11c/0x1c02018-01-06T17:10:02.194460+00:00 node-115 kernel:
[87885.157771]? [<ffffffff8124e693>]
dio_bio_end_aio+0x73/0x1002018-01-06T17:10:02.194461+00:00 node-115 kernel:
[87885.157774]? [<ffffffff813c3edf>]
bio_endio+0x3f/0x602018-01-06T17:10:02.194463+00:00 node-115 kernel:
[87885.157777]? [<ffffffff813cb897>]
blk_update_request+0x87/0x3102018-01-06T17:10:02.194464+00:00 node-115 kernel:
[87885.157780]? [<ffffffff816bbd66>]
end_clone_bio+0x46/0x702018-01-06T17:10:02.194465+00:00 node-115 kernel:
[87885.157782]? [<ffffffff813c3edf>]
bio_endio+0x3f/0x602018-01-06T17:10:02.194465+00:00 node-115 kernel:
[87885.157783]? [<ffffffff813cb897>]
blk_update_request+0x87/0x3102018-01-06T17:10:02.194467+00:00 node-115 kernel:
[87885.157786]? [<ffffffff815c52f3>]
scsi_end_request+0x33/0x1d02018-01-06T17:10:02.194468+00:00 node-115 kernel:
[87885.157788]? [<ffffffff815c8a26>]
scsi_io_completion+0x1b6/0x6902018-01-06T17:10:02.194469+00:00 node-115 kernel:
[87885.157792]? [<ffffffff810beb46>] ?
rebalance_domains+0x166/0x2d02018-01-06T17:10:02.194470+00:00 node-115 kernel:
[87885.157795]? [<ffffffff815bf64f>]
scsi_finish_command+0xcf/0x1202018-01-06T17:10:02.194471+00:00 node-115 kernel:
[87885.157796]? [<ffffffff815c81b4>]
scsi_softirq_done+0x124/0x1502018-01-06T17:10:02.194471+00:00 node-115 kernel:
[87885.157799]? [<ffffffff813d3787>]
blk_done_softirq+0x87/0xb02018-01-06T17:10:02.194480+00:00 node-115 kernel:
[87885.157803]? [<ffffffff81085dc1>]
__do_softirq+0x101/0x2902018-01-06T17:10:02.194481+00:00 node-115 kernel:
[87885.157805]? [<ffffffff810860c3>]
irq_exit+0xa3/0xb02018-01-06T17:10:02.194483+00:00 node-115 kernel:
[87885.157809]? [<ffffffff81050e93>]
smp_call_function_single_interrupt+0x33/0x402018-01-06T17:10:02.194483+00:00
node-115 kernel: [87885.157811]? [<ffffffff81845ae2>]
call_function_single_interrupt+0x82/0x902018-01-06T17:10:02.194484+00:00
node-115 kernel: [87885.157812]? <EOI>2018-01-06T17:10:02.194484+00:00
node-115 kernel: [87885.157814]? [<ffffffff81844414>] ?
_raw_spin_lock+0x14/0x302018-01-06T17:10:02.194486+00:00 node-115 kernel:
[87885.157815]? [<ffffffff81842422>]
__mutex_lock_slowpath+0x72/0x1302018-01-06T17:10:02.194487+00:00 node-115
kernel: [87885.157829]? [<ffffffffc0758099>] ?
ocfs2_inode_unlock+0x119/0x120 [ocfs2]2018-01-06T17:10:02.194488+00:00 node-115
kernel: [87885.157831]? [<ffffffff818424ff>]
mutex_lock+0x1f/0x302018-01-06T17:10:02.194489+00:00 node-115 kernel:
[87885.157843]? [<ffffffffc076177a>] ocfs2_file_write_iter+0x95a/0xdf0
[ocfs2]2018-01-06T17:10:02.194490+00:00 node-115 kernel: [87885.157847]?
[<ffffffff812252c0>] ?
poll_select_copy_remaining+0x140/0x1402018-01-06T17:10:02.194490+00:00 node-115
kernel: [87885.157858]? [<ffffffffc0760e20>] ?
ocfs2_check_range_for_refcount+0x150/0x150
[ocfs2]2018-01-06T17:10:02.194492+00:00 node-115 kernel: [87885.157862]?
[<ffffffff812613ea>]
aio_run_iocb+0x26a/0x2d02018-01-06T17:10:02.194493+00:00 node-115 kernel:
[87885.157865]? [<ffffffff8122e8e5>] ?
__fget_light+0x25/0x602018-01-06T17:10:02.194495+00:00 node-115 kernel:
[87885.157867]? [<ffffffff8122e933>] ?
__fdget+0x13/0x202018-01-06T17:10:02.194496+00:00 node-115 kernel:
[87885.157868]? [<ffffffff812622cf>]
do_io_submit+0x25f/0x5002018-01-06T17:10:02.194497+00:00 node-115 kernel:
[87885.157871]? [<ffffffff81262580>]
SyS_io_submit+0x10/0x202018-01-06T17:10:02.194499+00:00 node-115 kernel:
[87885.157873]? [<ffffffff818446b2>]
entry_SYSCALL_64_fastpath+0x16/0x712018-01-06T17:10:02.194500+00:00 node-115
kernel: [87885.157874] Code: 01 48 8b 02 48 85 c0 75 0a f3 90 48 8b 02 48 85 c0
74 f6 c7 40 08 01 00 00 00 e9 63 ff ff ff 83 fa 01 75 07 e9 c4 fe ff ff f3 90
<8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 5d c3 0f 1f 40 00
0f2018-01-06T17:10:30.192979+00:00 node-115 kernel: [87913.154413] Modules
linked in: vhost_net vhost macvtap macvlan ip6table_raw xt_mac xt_tcpudp
xt_physdev br_netfilter veth ebtable_filter ebtables openvswitch ocfs2
quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager
ocfs2_stackglue configfs ip6table_filter ip6_tables xt_multiport xt_conntrack
iptable_filter xt_comment xt_CT iptable_raw ip_tables x_tables xfs bridge 8021q
garp mrp stp llc intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp
crct10dif_pclmul ipmi_ssif crc32_pclmul ghash_clmulni_intel kvm_intel
aesni_intel aes_x86_64 kvm lrw gf128mul glue_helper ablk_helper irqbypass cryptd
hpilo 8250_fintek serio_raw ioatdma ipmi_si sb_edac edac_core ipmi_msghandler
shpchp dca acpi_power_meter lpc_ich mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa
ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
nf_conntrack_proto_gre nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4
nf_defrag_ipv4 nf_conntrack autofs4 raid10 raid456 async_raid6_recov
async_memcpy async_pq async_xor async_tx xor dm_round_robin ses enclosure
scsi_transport_sas raid6_pq libcrc32c raid1 raid0 multipath linear uas
usb_storage psmouse lpfc be2net vxlan ip6_udp_tunnel scsi_transport_fc
udp_tunnel wmi fjes scsi_dh_emc scsi_dh_rdac scsi_dh_alua
dm_multipath2018-01-06T17:10:30.192984+00:00 node-115 kernel: [87913.155150]
CPU: 15 PID: 11936 Comm: qemu-system-x86 Tainted: G? ? ? ? ? ? ?L?
4.4.0-98-generic #121-Ubuntu2018-01-06T17:10:30.192987+00:00 node-115 kernel:
[87913.155151] Hardware name: HP ProLiant BL460c Gen9, BIOS I36
02/17/20172018-01-06T17:10:30.192988+00:00 node-115 kernel: [87913.155153] task:
ffff882036ff0000 ti: ffff881f80ca0000 task.ti:
ffff881f80ca00002018-01-06T17:10:30.192990+00:00 node-115 kernel: [87913.155154]
RIP: 0010:[<ffffffff810cb27e>]? [<ffffffff810cb27e>]
native_queued_spin_lock_slowpath+0x15e/0x1702018-01-06T17:10:30.192992+00:00
node-115 kernel: [87913.155160] RSP: 0018:ffff88203f143c30? EFLAGS:
000002022018-01-06T17:10:30.192994+00:00 node-115 kernel: [87913.155161] RAX:
0000000000000101 RBX: ffff8820046c83f0 RCX:
00000000000000012018-01-06T17:10:30.192996+00:00 node-115 kernel: [87913.155162]
RDX: 0000000000000101 RSI: 0000000000000001 RDI:
ffff8820046c83ec2018-01-06T17:10:30.193019+00:00 node-115 kernel: [87913.155163]
RBP: ffff88203f143c30 R08: 0000000000000101 R09:
ffffffff811924a72018-01-06T17:10:30.193023+00:00 node-115 kernel: [87913.155164]
R10: ffffea0040d6d680 R11: 0000000000000800 R12:
ffff8820046c83ec2018-01-06T17:10:30.193024+00:00 node-115 kernel: [87913.155165]
R13: 0000000000000800 R14: 000000004c63ee00 R15:
00000000000008002018-01-06T17:10:30.193026+00:00 node-115 kernel: [87913.155166]
FS:? 00007fbcbb7eec00(0000) GS:ffff88203f140000(0000)
knlGS:00000000000000002018-01-06T17:10:30.193028+00:00 node-115 kernel:
[87913.155167] CS:? 0010 DS: 0000 ES: 0000 CR0:
00000000800500332018-01-06T17:10:30.193030+00:00 node-115 kernel: [87913.155168]
CR2: 00007f54266a8000 CR3: 0000000fcc2f2000 CR4:
00000000001426e02018-01-06T17:10:30.193032+00:00 node-115 kernel: [87913.155169]
Stack:2018-01-06T17:10:30.193034+00:00 node-115 kernel: [87913.155170]?
ffff88203f143c40 ffffffff81844421 ffff88203f143c60
ffffffff818425352018-01-06T17:10:30.193036+00:00 node-115 kernel:
[87913.155172]? ffff881e88a9ca80 ffff8820046c84b0 ffff88203f143c70
ffffffff8184257b2018-01-06T17:10:30.193037+00:00 node-115 kernel:
[87913.155173]? ffff88203f143ca0 ffffffffc074158d ffff881e5d3beb80
00000000000008002018-01-06T17:10:30.193039+00:00 node-115 kernel: [87913.155175]
Call Trace:2018-01-06T17:10:30.193040+00:00 node-115 kernel: [87913.155176]?
<IRQ>2018-01-06T17:10:30.193042+00:00 node-115 kernel: [87913.155183]?
[<ffffffff81844421>]
_raw_spin_lock+0x21/0x302018-01-06T17:10:30.193044+00:00 node-115 kernel:
[87913.155186]? [<ffffffff81842535>]
__mutex_unlock_slowpath+0x25/0x502018-01-06T17:10:30.193046+00:00 node-115
kernel: [87913.155187]? [<ffffffff8184257b>]
mutex_unlock+0x1b/0x202018-01-06T17:10:30.193047+00:00 node-115 kernel:
[87913.155224]? [<ffffffffc074158d>] ocfs2_dio_end_io+0x6d/0x80
[ocfs2]2018-01-06T17:10:30.193049+00:00 node-115 kernel: [87913.155228]?
[<ffffffff8124e57c>]
dio_complete+0x11c/0x1c02018-01-06T17:10:30.193051+00:00 node-115 kernel:
[87913.155230]? [<ffffffff8124e693>]
dio_bio_end_aio+0x73/0x1002018-01-06T17:10:30.193053+00:00 node-115 kernel:
[87913.155233]? [<ffffffff813c3edf>]
bio_endio+0x3f/0x602018-01-06T17:10:30.193055+00:00 node-115 kernel:
[87913.155235]? [<ffffffff813cb897>]
blk_update_request+0x87/0x3102018-01-06T17:10:30.193058+00:00 node-115 kernel:
[87913.155239]? [<ffffffff816bbd66>]
end_clone_bio+0x46/0x702018-01-06T17:10:30.193060+00:00 node-115 kernel:
[87913.155240]? [<ffffffff813c3edf>]
bio_endio+0x3f/0x602018-01-06T17:10:30.193062+00:00 node-115 kernel:
[87913.155242]? [<ffffffff813cb897>]
blk_update_request+0x87/0x3102018-01-06T17:10:30.193064+00:00 node-115 kernel:
[87913.155245]? [<ffffffff815c52f3>]
scsi_end_request+0x33/0x1d02018-01-06T17:10:30.193066+00:00 node-115 kernel:
[87913.155247]? [<ffffffff815c8a26>]
scsi_io_completion+0x1b6/0x6902018-01-06T17:10:30.193068+00:00 node-115 kernel:
[87913.155251]? [<ffffffff810beb46>] ?
rebalance_domains+0x166/0x2d02018-01-06T17:10:30.193069+00:00 node-115 kernel:
[87913.155254]? [<ffffffff815bf64f>]
scsi_finish_command+0xcf/0x1202018-01-06T17:10:30.193070+00:00 node-115 kernel:
[87913.155256]? [<ffffffff815c81b4>]
scsi_softirq_done+0x124/0x1502018-01-06T17:10:30.193071+00:00 node-115 kernel:
[87913.155258]? [<ffffffff813d3787>]
blk_done_softirq+0x87/0xb02018-01-06T17:10:30.193087+00:00 node-115 kernel:
[87913.155263]? [<ffffffff81085dc1>]
__do_softirq+0x101/0x2902018-01-06T17:10:30.193090+00:00 node-115 kernel:
[87913.155265]? [<ffffffff810860c3>]
irq_exit+0xa3/0xb02018-01-06T17:10:30.193092+00:00 node-115 kernel:
[87913.155269]? [<ffffffff81050e93>]
smp_call_function_single_interrupt+0x33/0x402018-01-06T17:10:30.193094+00:00
node-115 kernel: [87913.155270]? [<ffffffff81845ae2>]
call_function_single_interrupt+0x82/0x902018-01-06T17:10:30.193095+00:00
node-115 kernel: [87913.155271]? <EOI>2018-01-06T17:10:30.193096+00:00
node-115 kernel: [87913.155273]? [<ffffffff81844414>] ?
_raw_spin_lock+0x14/0x302018-01-06T17:10:30.193098+00:00 node-115 kernel:
[87913.155275]? [<ffffffff81842422>]
__mutex_lock_slowpath+0x72/0x1302018-01-06T17:10:30.193099+00:00 node-115
kernel: [87913.155289]? [<ffffffffc0758099>] ?
ocfs2_inode_unlock+0x119/0x120 [ocfs2]2018-01-06T17:10:30.193101+00:00 node-115
kernel: [87913.155291]? [<ffffffff818424ff>]
mutex_lock+0x1f/0x302018-01-06T17:10:30.193102+00:00 node-115 kernel:
[87913.155303]? [<ffffffffc076177a>] ocfs2_file_write_iter+0x95a/0xdf0
[ocfs2]2018-01-06T17:10:30.193104+00:00 node-115 kernel: [87913.155306]?
[<ffffffff812252c0>] ?
poll_select_copy_remaining+0x140/0x1402018-01-06T17:10:30.193105+00:00 node-115
kernel: [87913.155317]? [<ffffffffc0760e20>] ?
ocfs2_check_range_for_refcount+0x150/0x150
[ocfs2]2018-01-06T17:10:30.193106+00:00 node-115 kernel: [87913.155321]?
[<ffffffff812613ea>]
aio_run_iocb+0x26a/0x2d02018-01-06T17:10:30.193107+00:00 node-115 kernel:
[87913.155324]? [<ffffffff8122e8e5>] ?
__fget_light+0x25/0x602018-01-06T17:10:30.193108+00:00 node-115 kernel:
[87913.155325]? [<ffffffff8122e933>] ?
__fdget+0x13/0x202018-01-06T17:10:30.193109+00:00 node-115 kernel:
[87913.155327]? [<ffffffff812622cf>]
do_io_submit+0x25f/0x5002018-01-06T17:10:30.193109+00:00 node-115 kernel:
[87913.155329]? [<ffffffff81262580>]
SyS_io_submit+0x10/0x202018-01-06T17:10:30.193110+00:00 node-115 kernel:
[87913.155331]? [<ffffffff818446b2>]
entry_SYSCALL_64_fastpath+0x16/0x712018-01-06T17:10:30.193111+00:00 node-115
kernel: [87913.155332] Code: 8b 02 48 85 c0 75 0a f3 90 48 8b 02 48 85 c0 74 f6
c7 40 08 01 00 00 00 e9 63 ff ff ff 83 fa 01 75 07 e9 c4 fe ff ff f3 90 8b 07
<84> c0 75 f8 b8 01 00 00 00 66 89 07 5d c3 0f 1f 40 00 0f 1f 44
_______________________________________________
Ocfs2-users mailing list
Ocfs2-users at oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://oss.oracle.com/pipermail/ocfs2-users/attachments/20180110/a5a15ab0/attachment-0001.html