Hi,
seems like RHEL/CentOS package glusterfs-rdma (from the gluster repo) is
not working together with the Mellanox ofed.
Guess the problem is in the dependency of libibverbs which depends on
the rdma package of RHEL/CentOS.
Installation of glusterfs-rdma might break your Mellanox ofed
installation. At least on the four systems (with three different
kernels) I tried, the openibd.service was unusable because of the
rdma_cm module installed by the rdma package.
Is there something I can do in case I would like to use glusterfs but
need the Mellanox ofed or do I have to discard the idea of using
glusterfs for my setup?
Best regards,
Jochen
Am 09.11.2015 um 23:31 schrieb Jochen Becker:> Hi folks,
>
> I have some problems starting a replica volume on a two node
> infiniband setup.
> Both systems are running the same hardware and Infiniband (ipoib,
> ibverbs) seems to work well.
>
> OS is Centos 7.1 fresh install and updated, Mellanox Ofed is in use,
> openibd is running both gluster peers are in connected state with each
> other. Creating the volume was no problem at all, but starting always
> fails. Using the force option the volume seems to be started but
> cannot be mounted.
>
> Here are the entries of mnt-bricks-instances.log that happen during
> the command gluster volume start instances:
>
> [2015-11-09 22:01:00.153360] I [MSGID: 100030]
> [glusterfsd.c:2318:main] 0-/usr/sbin/glusterfsd: Started running
> /usr/sbin/glusterfsd version 3.7.5 (args: /usr/sbin/glusterfsd -s
> compute02 --volfile-id instances.compute02.mnt-bricks-instances -p
> /var/lib/glusterd/vols/instances/run/compute02-mnt-bricks-instances.pid -S
> /var/run/gluster/8f5e59a0b8d5949b51b4c276192b0725.socket --brick-name
> /mnt/bricks/instances -l
> /var/log/glusterfs/bricks/mnt-bricks-instances.log --xlator-option
> *-posix.glusterd-uuid=52109ce5-6173-4d22-bffc-a03c71d24791
> --brick-port 49152 --xlator-option instances-server.listen-port=49152
> --volfile-server-transport=rdma)
> [2015-11-09 22:01:00.169326] I [MSGID: 101190]
> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started
> thread with index 1
> [2015-11-09 22:01:00.177472] I [graph.c:269:gf_add_cmdline_options]
> 0-instances-server: adding option 'listen-port' for volume
> 'instances-server' with value '49152'
> [2015-11-09 22:01:00.177519] I [graph.c:269:gf_add_cmdline_options]
> 0-instances-posix: adding option 'glusterd-uuid' for volume
> 'instances-posix' with value
'52109ce5-6173-4d22-bffc-a03c71d24791'
> [2015-11-09 22:01:00.177826] I [MSGID: 115034]
> [server.c:403:_check_for_auth_option] 0-/mnt/bricks/instances: skip
> format check for non-addr auth option
> auth.login./mnt/bricks/instances.allow
> [2015-11-09 22:01:00.177913] I [MSGID: 115034]
> [server.c:403:_check_for_auth_option] 0-/mnt/bricks/instances: skip
> format check for non-addr auth option
> auth.login.9d64b3ec-9d24-41ac-ba84-cf58c67c9b21.password
> [2015-11-09 22:01:00.177916] I [MSGID: 101190]
> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started
> thread with index 2
> [2015-11-09 22:01:00.179299] I
> [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service:
> Configured rpc.outstanding-rpc-limit with value 64
> [2015-11-09 22:01:00.181636] W [MSGID: 101002]
> [options.c:957:xl_opt_validate] 0-instances-server: option
> 'listen-port' is deprecated, preferred is
> 'transport.rdma.listen-port', continuing with correction
> [2015-11-09 22:01:00.183742] W [MSGID: 103071]
> [rdma.c:4592:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event
> channel creation failed [Keine Berechtigung]
> [2015-11-09 22:01:00.183782] W [MSGID: 103055] [rdma.c:4899:init]
> 0-rdma.instances-server: Failed to initialize IB Device
> [2015-11-09 22:01:00.183796] W
> [rpc-transport.c:359:rpc_transport_load] 0-rpc-transport: 'rdma'
> initialization failed
> [2015-11-09 22:01:00.183866] W [rpcsvc.c:1597:rpcsvc_transport_create]
> 0-rpc-service: cannot create listener, initing the transport failed
> [2015-11-09 22:01:00.183884] W [MSGID: 115045] [server.c:1019:init]
> 0-instances-server: creation of listener failed
> [2015-11-09 22:01:00.183898] E [MSGID: 101019]
> [xlator.c:428:xlator_init] 0-instances-server: Initialization of
> volume 'instances-server' failed, review your volfile again
> [2015-11-09 22:01:00.183912] E [graph.c:322:glusterfs_graph_init]
> 0-instances-server: initializing translator failed
> [2015-11-09 22:01:00.183921] E [graph.c:661:glusterfs_graph_activate]
> 0-graph: init failed
> [2015-11-09 22:01:00.184429] W [glusterfsd.c:1236:cleanup_and_exit]
> (-->/usr/sbin/glusterfsd(mgmt_getspec_cbk+0x331) [0x7f7b6aee02f1]
> -->/usr/sbin/glusterfsd(glusterfs_process_volfp+0x126)
> [0x7f7b6aedb0f6] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x69)
> [0x7f7b6aeda6d9] ) 0-: received signum (0), shutting down
>
> I think the problem starts at 2015-11-09 22:01:00.183742 where the
> channel creation with rdma_cm fails. "Keine Berechtigung" means
> something like missing permissions/rights. Module rdma_cm is loaded
> and I can't find any other problem with the Infiniband or rdma. I have
> no clue what is going wrong here, so any hints on how to proceed are
> appreciated.
>
> Cheers,
>
> Jochen
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
--
Linux System Engineer
mail j.becker at uvensys.de
uvensys GmbH
Firmensitz und Sitz der Gesellschaft:
uvensys GmbH
Elsa-Brandstr?m-Stra?e 3
35510 Butzbach
HRB: AG Friedberg, 7780
USt-Id: DE282879294
Gesch?ftsf?hrer:
Dr. Thomas Licht, t.licht at uvensys.de
Volker Lieder, v.lieder at uvensys.de
E-Mail: info at uvensys.de
Internet: http://www.uvensys.de
Durchwahl: 06033 - 9756552
Zentrale: 06033 - 9756940
Fax: 06033 - 9756554
=========================================================
Jegliche Stellungnahmen und Meinungen dieser E-Mail sind
alleine die des Autors und nicht notwendigerweise die der
Firma. Falls erforderlich, k?nnen Sie eine gesonderte
schriftliche Best?tigung anfordern.
Any views or opinions presented in this email are solely
those of the author and do not necessarily represent those
of the company. If verification is required please request
a hard-copy version.