Gluster users - Nov 2018 - rdma.management: could not create QP [Permission denied]

Hello All,

I'm having trouble mounting gluster volumes over RDMA. Everything seems to
work fine until I try to mount the gluster volume. At this point it seems
to crash glusterd and give me an error "could not create QP [Permission
Denied]. If I change change the transport to tcp, everything works as
expected. I am pretty new to Infiniband and RDMA, so this may very well be
something I'm doing wrong. That said, I was able to get NFSoRDMA (and
qperf) working on these same hosts, so I'm confident my hardware is
working. This is on CentOS 7.5 using gluster 3.12.15 from the CentOS
repository. I can't find anything regarding this specific error and any
help is much appreciated.

# rdma works...
[root at gluster1 ~]# qperf --use_bits_per_sec gluster2-ib rc_bi_bw
rc_bi_bw:
    bw  =  35.3 Gb/sec

# NFSoRDMA works....
[root at gluster1 data]# mount -t nfs gluster2-ib:/test /mnt/test -o
rdma,port=20049
[root at gluster1 ~]#
[root at gluster1 data]# df -h|grep root
/dev/mapper/centos_gluster1-root   33G  7.8G   26G  24% /
gluster2-ib:/test            33G   29G  4.1G  88% /mnt/test

[root at gluster1 ~]# gluster volume create data replica 3 transport rdma
gluster1-ib:/zp1/data vmh2-ib:/zp1/data vmh3-ib:/zp1/data force
volume create: data: success: please start the volume to access data
[root at gluster1 ~]#
[root at gluster1 ~]# gluster volume start data
volume start: data: success
[root at gluster1 ~]#
[root at gluster1 ~]# gluster volume info data

Volume Name: data
Type: Replicate
Volume ID: 6f32999c-2d20-4fc2-838b-82972171df7e
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: rdma
Bricks:
Brick1: gluster1-ib:/zp1/data
Brick2: gluster2-ib:/zp1/data
Brick3: gluster3-ib:/zp1/data
Options Reconfigured:
nfs.disable: on
performance.client-io-threads: off
[root at gluster1 ~]#
[root at gluster1 ~]# mount -t glusterfs -o transport=rdma gluster1:/data
/mnt/data
Mount failed. Please check the log file for more details.
[root at gluster1 ~]#
[root at gluster1 ~]# gluster volume info data
Connection failed. Please check if gluster daemon is operational.
[root at gluster1 ~]#
[root at gluster1 ~]# tail -n 30 /var/log/messages|grep glusterd
Nov 10 11:18:07 gluster1 glusterd[1881]: [2018-11-10 16:18:07.457818] C
[MSGID: 103021] [rdma.c:3263:gf_rdma_create_qp] 0-rdma.management:
rdma.management: could not create QP [Permission denied]
Nov 10 11:18:07 gluster1 glusterd[1881]: pending frames:
Nov 10 11:18:07 gluster1 glusterd[1881]: patchset: git://
git.gluster.org/glusterfs.git
Nov 10 11:18:07 gluster1 glusterd[1881]: signal received: 11
Nov 10 11:18:07 gluster1 glusterd[1881]: time of crash:
Nov 10 11:18:07 gluster1 glusterd[1881]: 2018-11-10 16:18:07
Nov 10 11:18:07 gluster1 glusterd[1881]: configuration details:
Nov 10 11:18:07 gluster1 glusterd[1881]: argp 1
Nov 10 11:18:07 gluster1 glusterd[1881]: backtrace 1
Nov 10 11:18:07 gluster1 glusterd[1881]: dlfcn 1
Nov 10 11:18:07 gluster1 glusterd[1881]: libpthread 1
Nov 10 11:18:07 gluster1 glusterd[1881]: llistxattr 1
Nov 10 11:18:07 gluster1 glusterd[1881]: setfsid 1
Nov 10 11:18:07 gluster1 glusterd[1881]: spinlock 1
Nov 10 11:18:07 gluster1 glusterd[1881]: epoll.h 1
Nov 10 11:18:07 gluster1 glusterd[1881]: xattr.h 1
Nov 10 11:18:07 gluster1 glusterd[1881]: st_atim.tv_nsec 1
Nov 10 11:18:07 gluster1 glusterd[1881]: package-string: glusterfs 3.12.15
Nov 10 11:18:07 gluster1 glusterd[1881]: ---------
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20181110/745e58e2/attachment.html>

10.11.2018 20:53, Thomas Simmons ?????:
> Hello All,
> 
> I'm having trouble mounting gluster volumes over RDMA. Everything seems
> to work fine until I try to mount the gluster volume. At this point it 
> seems to crash glusterd and give me an error "could not create QP 
> [Permission Denied].
Please see my post with subject "glusterd SIGSEGV crash when create 
volume with transport=rdma" at 07.11.2018

I'm getting this errors&crashes too.

But I'm resolve it via creating SELinux module with audit2allow... and I 
get worse situation %)

Now, when SELinux allow rule "Infiniband pkeys" and I start a RDMA 
volume, server (peer) silently REBOOT with no traces..

I enable persistent journal storage, but after reboot I get NO records 
about reboots : journalctl --last-boots show EMPTY list, no boots at all.
Usually it show 1 last boot... I think XFS lose some files when accident 
reboot happens.

And more, XFS lose vol files (made it zero-size) and one of my peers 
cannot start ALL bricks (from all volumes, not only RDMA, but also tcp).

-- 
Mike

10.11.2018 20:53, Thomas Simmons ?????:
> Hello All,
> 
> I'm having trouble mounting gluster volumes over RDMA. Everything seems
> to work fine until I try to mount the gluster volume. At this point it 
> seems to crash glusterd and give me an error "could not create QP 
Are you tried to permit it in SELinux? after permitting are glusterfs 
works as expected ?

-- 
Mike