Ocfs2 users - Dec 2017 - issues with my ocfs2 cluster

hi list,

I have a ocfs2 filesystem setup as a shared filesystem between 12 openstack
compute nodes which are Ubuntu 16.04.3.
I have a very big concern of stability.
A month ago I lost a good deal of files, I don't know the real reason, but
things seemed to point to the ofcs2 cluster.
Last week I found many of my compute nodes with the nova service down. The
node which went down first has a "stuck" file/directory in the ocfs2
filesystem

root at node-99:/mnt/MSA_FC_Vol1/nodes/cb5c94d0-ed4f-457d-88b0-17d49eb7006a# ls

directory in the command above has vHD files in it. Running a simple
'ls'
command hangs indefinitely (i've left it hung 5 days now, it is stuck,
never completes). at the end of this email I've pasted the end of the dmesg
output.....

I ran fsck.ocfs2 on the filesystem and it did fix somethings. but running
'ls' again in that directory still becomes stuck, and the nova service
still comes down on all nodes.

when I restart nova services on all these node they come down again after
some time. when I stop ocfs2 on all these nodes they no longer come down.

I have other openstack compute nodes that are identical except they use
local storage and do not use ocfs2 and these have always been stable.

maybe ocfs2 just isn't stable on Ubuntu 16.04.3? I am using version
1.6.4-3.1

any advice or comments would be appreciated!!!

[Thu Dec 21 20:22:35 2017] INFO: task ls:11052 blocked for more than 120
seconds.
[Thu Dec 21 20:22:35 2017]       Not tainted 4.4.0-98-generic #121-Ubuntu
[Thu Dec 21 20:22:35 2017] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Dec 21 20:22:35 2017] ls              D ffff880074c7b8d8     0 11052
    1 0x00000004
[Thu Dec 21 20:22:35 2017]  ffff880074c7b8d8 ffff882016f1d400
ffff882038f70e00 ffff880035a8c600
[Thu Dec 21 20:22:35 2017]  ffff880074c7c000 ffff880074c7ba80
ffff880074c7ba78 ffff880035a8c600
[Thu Dec 21 20:22:35 2017]  0000000000000000 ffff880074c7b8f0
ffffffff81840585 7fffffffffffffff
[Thu Dec 21 20:22:35 2017] Call Trace:
[Thu Dec 21 20:22:35 2017]  [<ffffffff81840585>] schedule+0x35/0x80
[Thu Dec 21 20:22:35 2017]  [<ffffffff818436d5>]
schedule_timeout+0x1b5/0x270
[Thu Dec 21 20:22:35 2017]  [<ffffffff81840fe3>]
wait_for_completion+0xb3/0x140
[Thu Dec 21 20:22:35 2017]  [<ffffffff810ac630>] ? wake_up_q+0x70/0x70
[Thu Dec 21 20:22:35 2017]  [<ffffffffc0779145>]
__ocfs2_cluster_lock.isra.34+0x415/0x750 [ocfs2]
[Thu Dec 21 20:22:35 2017]  [<ffffffff810f634b>] ? ktime_get+0x3b/0xb0
[Thu Dec 21 20:22:35 2017]  [<ffffffffc077a20a>]
ocfs2_inode_lock_full_nested+0x16a/0x920 [ocfs2]
[Thu Dec 21 20:22:35 2017]  [<ffffffffc0786ee9>] ocfs2_iget+0x499/0x6c0
[ocfs2]
[Thu Dec 21 20:22:35 2017]  [<ffffffffc0770ab8>] ?
ocfs2_free_dir_lookup_result+0x28/0x50 [ocfs2]
[Thu Dec 21 20:22:35 2017]  [<ffffffffc077264e>] ?
ocfs2_lookup_ino_from_name+0x4e/0x70 [ocfs2]
[Thu Dec 21 20:22:35 2017]  [<ffffffffc0796ff5>] ocfs2_lookup+0x145/0x2f0
[ocfs2]
[Thu Dec 21 20:22:35 2017]  [<ffffffff8121a54d>] lookup_real+0x1d/0x60
[Thu Dec 21 20:22:35 2017]  [<ffffffff8121be42>] __lookup_hash+0x42/0x60
[Thu Dec 21 20:22:35 2017]  [<ffffffff8121d1d6>]
walk_component+0x226/0x300
[Thu Dec 21 20:22:35 2017]  [<ffffffffc0774e33>] ?
ocfs2_should_refresh_lock_res+0x113/0x160 [ocfs2]
[Thu Dec 21 20:22:35 2017]  [<ffffffff8121eb1d>] path_lookupat+0x5d/0x110
[Thu Dec 21 20:22:35 2017]  [<ffffffff81220761>]
filename_lookup+0xb1/0x180
[Thu Dec 21 20:22:35 2017]  [<ffffffffc07855d3>] ?
ocfs2_inode_revalidate+0x93/0x180 [ocfs2]
[Thu Dec 21 20:22:35 2017]  [<ffffffff811eebb7>] ?
kmem_cache_alloc+0x187/0x1f0
[Thu Dec 21 20:22:35 2017]  [<ffffffff81220366>] ?
getname_flags+0x56/0x1f0
[Thu Dec 21 20:22:35 2017]  [<ffffffff81220906>]
user_path_at_empty+0x36/0x40
[Thu Dec 21 20:22:35 2017]  [<ffffffff81215616>] vfs_fstatat+0x66/0xc0
[Thu Dec 21 20:22:35 2017]  [<ffffffff81215bd1>] SYSC_newlstat+0x31/0x60
[Thu Dec 21 20:22:35 2017]  [<ffffffff81215d0e>] SyS_newlstat+0xe/0x10
[Thu Dec 21 20:22:35 2017]  [<ffffffff818446b2>]
entry_SYSCALL_64_fastpath+0x16/0x71
[Thu Dec 21 20:33:10 2017] perf interrupt took too long (7066 > 5000),
lowering kernel.perf_event_max_sample_rate to 25000
[Thu Dec 21 22:05:02 2017] perf interrupt took too long (10271 > 10000),
lowering kernel.perf_event_max_sample_rate to 12500
[Fri Dec 22 00:00:01 2017] Process accounting resumed
[Fri Dec 22 00:10:25 2017] perf interrupt took too long (20273 > 20000),
lowering kernel.perf_event_max_sample_rate to 6250
[Fri Dec 22 07:19:10 2017] perf interrupt took too long (41761 > 38461),
lowering kernel.perf_event_max_sample_rate to 3250
[Fri Dec 22 23:59:58 2017] Process accounting resumed
[Sat Dec 23 07:19:11 2017] perf interrupt took too long (76936 > 71428),
lowering kernel.perf_event_max_sample_rate to 1750
[Sat Dec 23 23:59:56 2017] Process accounting resumed
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://oss.oracle.com/pipermail/ocfs2-users/attachments/20171227/a538a7f8/attachment.html

> I have a ocfs2 filesystem setup as a shared filesystem between
> 12 openstack compute nodes which are Ubuntu 16.04.3.
I am amazed by how unconstrained are the imaginations of some
other people. That is a truly astonishing setup.
> I have a very big concern of stability.  A month ago I lost a
> good deal of files, I don't know the real reason, but things
> seemed to point to the ofcs2 cluster.
That also seems to me unconstrained by concern about mere
details.
> Last week I found many of my compute nodes with the nova
> service down. The node which went down first has a "stuck"
> file/directory in the ocfs2 filesystem [ ... ]
The stack trace seems to point at a "lost" wakeup from the OCFS2
lock manager.
> I have other openstack compute nodes that are identical except
> they use local storage and do not use ocfs2 and these have
> always been stable.
But OCFS2 is meant to work with local physical storage on a
local phyical machine. What's your current setup?
> maybe ocfs2 just isn't stable on Ubuntu 16.04.3? I am using
> version 1.6.4-3.1
OCFS2 has been extremely stable for many years on very high load
share-disk clusters for many users. OpenStack and perhaps newer
kernels not necessarily so.

Also OCSF2 requires a storage subsystem with specific features
and a high degree of reliable operation. It is astonishing but
fairly typical that this reports contains no mention of the
setup or of the state of the storage subsystem.