Deepak Naidu
2017-Mar-03 08:08 UTC
[Gluster-users] [ovirt-users] Hot to force glusterfs to use RDMA?
>> As you can see from my previous email that the RDMA connection tested with qperf.I think you have wrong command. Your testing TCP & not RDMA. Also check if you have RDMA & IB modules loaded on your hosts. root at clei26 ~]# qperf clei22.vib tcp_bw tcp_lat tcp_bw: bw = 475 MB/sec tcp_lat: latency = 52.8 us [root at clei26 ~]# Please run below command to test RDMA [root at storageN2 ~]# qperf storageN1 ud_lat ud_bw ud_lat: latency = 7.51 us ud_bw: send_bw = 9.21 GB/sec recv_bw = 9.21 GB/sec [root at sc-sdgx-202 ~]# Read qperf man pages for more info. * To run a TCP bandwidth and latency test: qperf myserver tcp_bw tcp_lat * To run a UDP latency test and then cause the server to terminate: qperf myserver udp_lat quit * To measure the RDMA UD latency and bandwidth: qperf myserver ud_lat ud_bw * To measure RDMA UC bi-directional bandwidth: qperf myserver rc_bi_bw * To get a range of TCP latencies with a message size from 1 to 64K qperf myserver -oo msg_size:1:64K:*2 -vu tcp_lat Check if you have RDMA & IB modules loaded lsmod | grep -i ib lsmod | grep -i rdma -- Deepak From: Arman Khalatyan [mailto:arm2arm at gmail.com] Sent: Thursday, March 02, 2017 10:57 PM To: Deepak Naidu Cc: Rafi Kavungal Chundattu Parambil; gluster-users at gluster.org; users; Sahina Bose Subject: RE: [Gluster-users] [ovirt-users] Hot to force glusterfs to use RDMA? Dear Deepak, thank you for the hints, which gluster are you using? As you can see from my previous email that the RDMA connection tested with qperf. It is working as expected. In my case the clients are servers as well, they are hosts for the ovirt. Disabling selinux is nor recommended by ovirt, but i will give a try. Am 03.03.2017 7:50 vorm. schrieb "Deepak Naidu" <dnaidu at nvidia.com<mailto:dnaidu at nvidia.com>>: I have been testing glusterfs over RDMA & below is the command I use. Reading up the logs, it looks like your IB(InfiniBand) device is not being initialized. I am not sure if u have an issue on the client IB or the storage server IB. Also have you configured ur IB devices correctly. I am using IPoIB. Can you check your firewall, disable selinux, I think, you might have checked it already ? mount -t glusterfs -o transport=rdma storageN1:/vol0 /mnt/vol0 ? The below error seems if you have issue starting your volume. I had issue, when my transport was set to tcp,rdma. I had to force start my volume. If I had set it only to tcp on the volume, the volume would start easily. [2017-03-02 11:49:47.829391] E [MSGID: 114022] [client.c:2530:client_init_rpc] 0-GluReplica-client-2: failed to initialize RPC [2017-03-02 11:49:47.829413] E [MSGID: 101019] [xlator.c:433:xlator_init] 0-GluReplica-client-2: Initialization of volume 'GluReplica-client-2' failed, review your volfile again [2017-03-02 11:49:47.829425] E [MSGID: 101066] [graph.c:324:glusterfs_graph_init] 0-GluReplica-client-2: initializing translator failed [2017-03-02 11:49:47.829436] E [MSGID: 101176] [graph.c:673:glusterfs_graph_activate] 0-graph: init failed ? The below error seems if you have issue with IB device. If not configured properly. [2017-03-02 11:49:47.828996] W [MSGID: 103071] [rdma.c:4589:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-03-02 11:49:47.829067] W [MSGID: 103055] [rdma.c:4896:init] 0-GluReplica-client-2: Failed to initialize IB Device [2017-03-02 11:49:47.829080] W [rpc-transport.c:354:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed -- Deepak From: gluster-users-bounces at gluster.org<mailto:gluster-users-bounces at gluster.org> [mailto:gluster-users-bounces at gluster.org<mailto:gluster-users-bounces at gluster.org>] On Behalf Of Sahina Bose Sent: Thursday, March 02, 2017 10:26 PM To: Arman Khalatyan; gluster-users at gluster.org<mailto:gluster-users at gluster.org>; Rafi Kavungal Chundattu Parambil Cc: users Subject: Re: [Gluster-users] [ovirt-users] Hot to force glusterfs to use RDMA? [Adding gluster users to help with error] [2017-03-02 11:49:47.828996] W [MSGID: 103071] [rdma.c:4589:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] On Thu, Mar 2, 2017 at 5:36 PM, Arman Khalatyan <arm2arm at gmail.com<mailto:arm2arm at gmail.com>> wrote: BTW RDMA is working as expected: root at clei26 ~]# qperf clei22.vib tcp_bw tcp_lat tcp_bw: bw = 475 MB/sec tcp_lat: latency = 52.8 us [root at clei26 ~]# thank you beforehand. Arman. On Thu, Mar 2, 2017 at 12:54 PM, Arman Khalatyan <arm2arm at gmail.com<mailto:arm2arm at gmail.com>> wrote: just for reference: gluster volume info Volume Name: GluReplica Type: Replicate Volume ID: ee686dfe-203a-4caa-a691-26353460cc48 Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp,rdma Bricks: Brick1: 10.10.10.44:/zclei22/01/glu Brick2: 10.10.10.42:/zclei21/01/glu Brick3: 10.10.10.41:/zclei26/01/glu (arbiter) Options Reconfigured: network.ping-timeout: 30 server.allow-insecure: on storage.owner-gid: 36 storage.owner-uid: 36 cluster.data-self-heal-algorithm: full features.shard: on cluster.server-quorum-type: server cluster.quorum-type: auto network.remote-dio: enable cluster.eager-lock: enable performance.stat-prefetch: off performance.io-cache: off performance.read-ahead: off performance.quick-read: off performance.readdir-ahead: on nfs.disable: on [root at clei21 ~]# gluster volume status Status of volume: GluReplica Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.10.10.44:/zclei22/01/glu 49158 49159 Y 15870 Brick 10.10.10.42:/zclei21/01/glu 49156 49157 Y 17473 Brick 10.10.10.41:/zclei26/01/glu 49153 49154 Y 18897 Self-heal Daemon on localhost N/A N/A Y 17502 Self-heal Daemon on 10.10.10.41 N/A N/A Y 13353 Self-heal Daemon on 10.10.10.44 N/A N/A Y 32745 Task Status of Volume GluReplica ------------------------------------------------------------------------------ There are no active volume tasks On Thu, Mar 2, 2017 at 12:52 PM, Arman Khalatyan <arm2arm at gmail.com<mailto:arm2arm at gmail.com>> wrote: I am not able to mount with RDMA over cli.... Are there some volfile parameters needs to be tuned? /usr/bin/mount -t glusterfs -o backup-volfile-servers=10.10.10.44:10.10.10.42:10.10.10.41,transport=rdma 10.10.10.44:/GluReplica /mnt [2017-03-02 11:49:47.795511] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.9 (args: /usr/sbin/glusterfs --volfile-server=10.10.10.44 --volfile-server=10.10.10.44 --volfile-server=10.10.10.42 --volfile-server=10.10.10.41 --volfile-server-transport=rdma --volfile-id=/GluReplica.rdma /mnt) [2017-03-02 11:49:47.812699] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-03-02 11:49:47.825210] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2017-03-02 11:49:47.828996] W [MSGID: 103071] [rdma.c:4589:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-03-02 11:49:47.829067] W [MSGID: 103055] [rdma.c:4896:init] 0-GluReplica-client-2: Failed to initialize IB Device [2017-03-02 11:49:47.829080] W [rpc-transport.c:354:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-03-02 11:49:47.829272] W [rpc-clnt.c:1070:rpc_clnt_connection_init] 0-GluReplica-client-2: loading of new rpc-transport failed [2017-03-02 11:49:47.829325] I [MSGID: 101053] [mem-pool.c:641:mem_pool_destroy] 0-GluReplica-client-2: size=588 max=0 total=0 [2017-03-02 11:49:47.829371] I [MSGID: 101053] [mem-pool.c:641:mem_pool_destroy] 0-GluReplica-client-2: size=124 max=0 total=0 [2017-03-02 11:49:47.829391] E [MSGID: 114022] [client.c:2530:client_init_rpc] 0-GluReplica-client-2: failed to initialize RPC [2017-03-02 11:49:47.829413] E [MSGID: 101019] [xlator.c:433:xlator_init] 0-GluReplica-client-2: Initialization of volume 'GluReplica-client-2' failed, review your volfile again [2017-03-02 11:49:47.829425] E [MSGID: 101066] [graph.c:324:glusterfs_graph_init] 0-GluReplica-client-2: initializing translator failed [2017-03-02 11:49:47.829436] E [MSGID: 101176] [graph.c:673:glusterfs_graph_activate] 0-graph: init failed [2017-03-02 11:49:47.830003] W [glusterfsd.c:1327:cleanup_and_exit] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x3c1) [0x7f524c9dbeb1] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x172) [0x7f524c9d65d2] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f524c9d5b4b] ) 0-: received signum (1), shutting down [2017-03-02 11:49:47.830053] I [fuse-bridge.c:5794:fini] 0-fuse: Unmounting '/mnt'. [2017-03-02 11:49:47.831014] W [glusterfsd.c:1327:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f524b343dc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f524c9d5cd5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f524c9d5b4b] ) 0-: received signum (15), shutting down [2017-03-02 11:49:47.831014] W [glusterfsd.c:1327:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f524b343dc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f524c9d5cd5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f524c9d5b4b] ) 0-: received signum (15), shutting down On Thu, Mar 2, 2017 at 12:11 PM, Sahina Bose <sabose at redhat.com<mailto:sabose at redhat.com>> wrote: You will need to pass additional mount options while creating the storage domain (transport=rdma) Please let us know if this works. On Thu, Mar 2, 2017 at 2:42 PM, Arman Khalatyan <arm2arm at gmail.com<mailto:arm2arm at gmail.com>> wrote: Hi, Are there way to force the connections over RDMA only? If I check host mounts I cannot see rdma mount option: mount -l| grep gluster 10.10.10.44:/GluReplica on /rhev/data-center/mnt/glusterSD/10.10.10.44:_GluReplica type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) I have glusterized 3 nodes: GluReplica Volume ID: ee686dfe-203a-4caa-a691-26353460cc48 Volume Type: Replicate (Arbiter) Replica Count: 2 + 1 Number of Bricks: 3 Transport Types: TCP, RDMA Maximum no of snapshots: 256 Capacity: 3.51 TiB total, 190.56 GiB used, 3.33 TiB free _______________________________________________ Users mailing list Users at ovirt.org<mailto:Users at ovirt.org> http://lists.ovirt.org/mailman/listinfo/users ________________________________ This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. ________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170303/f88cc3ca/attachment.html>
Arman Khalatyan
2017-Mar-03 10:51 UTC
[Gluster-users] [ovirt-users] Hot to force glusterfs to use RDMA?
Thank you all for the nice hints.
Somehow my host was not able to access the userspace RDMA, after
installing:
yum install -y libmlx4.x86_64
I can mount:
/usr/bin/mount -t glusterfs -o
backup-volfile-servers=10.10.10.44:10.10.10.42:10.10.10.41,transport=rdma
10.10.10.44:/GluReplica /mnt
10.10.10.44:/GluReplica.rdma 3770662912 407817216 3362845696 11% /mnt
Looks the rdma and gluster are working except ovirt GUI:(
With MountOptions:
backup-volfile-servers=10.10.10.44:10.10.10.42:10.10.10.41,transport=rdma
I am not able to activate storage.
---Gluster Status ----
gluster volume status
Status of volume: GluReplica
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 10.10.10.44:/zclei22/01/glu 49162 49163 Y
17173
Brick 10.10.10.42:/zclei21/01/glu 49156 49157 Y
17113
Brick 10.10.10.41:/zclei26/01/glu 49157 49158 Y
16404
Self-heal Daemon on localhost N/A N/A Y
16536
Self-heal Daemon on clei21.vib N/A N/A Y
17134
Self-heal Daemon on 10.10.10.44 N/A N/A Y
17329
Task Status of Volume GluReplica
------------------------------------------------------------------------------
There are no active volume tasks
-----IB status -----
ibstat
CA 'mlx4_0'
CA type: MT26428
Number of ports: 1
Firmware version: 2.7.700
Hardware version: b0
Node GUID: 0x002590ffff163758
System image GUID: 0x002590ffff16375b
Port 1:
State: Active
Physical state: LinkUp
Rate: 10
Base lid: 273
LMC: 0
SM lid: 3
Capability mask: 0x02590868
Port GUID: 0x002590ffff163759
Link layer: InfiniBand
Not bad for SDR switch ! :-P
qperf clei22.vib ud_lat ud_bw
ud_lat:
latency = 23.6 us
ud_bw:
send_bw = 981 MB/sec
recv_bw = 980 MB/sec
On Fri, Mar 3, 2017 at 9:08 AM, Deepak Naidu <dnaidu at nvidia.com> wrote:
> >> As you can see from my previous email that the RDMA connection
tested
> with qperf.
>
> I think you have wrong command. Your testing *TCP & not RDMA. *Also
check
> if you have RDMA & IB modules loaded on your hosts.
>
> root at clei26 ~]# qperf clei22.vib tcp_bw tcp_lat
> tcp_bw:
> bw = 475 MB/sec
> tcp_lat:
> latency = 52.8 us
> [root at clei26 ~]#
>
>
>
> *Please run below command to test RDMA*
>
>
>
> *[root at storageN2 ~]# qperf storageN1 ud_lat ud_bw*
>
> *ud_lat**:*
>
> * latency = 7.51 us*
>
> *ud_bw**:*
>
> * send_bw = 9.21 GB/sec*
>
> * recv_bw = 9.21 GB/sec*
>
> *[root at sc-sdgx-202 ~]#*
>
>
>
> Read qperf man pages for more info.
>
>
>
> * To run a TCP bandwidth and latency test:
>
> qperf myserver tcp_bw tcp_lat
>
> * To run a UDP latency test and then cause the server to terminate:
>
> qperf myserver udp_lat quit
>
> * To measure the RDMA UD latency and bandwidth:
>
> qperf myserver ud_lat ud_bw
>
> * To measure RDMA UC bi-directional bandwidth:
>
> qperf myserver rc_bi_bw
>
> * To get a range of TCP latencies with a message size from 1 to 64K
>
> qperf myserver -oo msg_size:1:64K:*2 -vu tcp_lat
>
>
>
>
>
> *Check if you have RDMA & IB modules loaded*
>
>
>
> lsmod | grep -i ib
>
>
>
> lsmod | grep -i rdma
>
>
>
>
>
>
>
> --
>
> Deepak
>
>
>
>
>
>
>
> *From:* Arman Khalatyan [mailto:arm2arm at gmail.com]
> *Sent:* Thursday, March 02, 2017 10:57 PM
> *To:* Deepak Naidu
> *Cc:* Rafi Kavungal Chundattu Parambil; gluster-users at gluster.org;
users;
> Sahina Bose
> *Subject:* RE: [Gluster-users] [ovirt-users] Hot to force glusterfs to
> use RDMA?
>
>
>
> Dear Deepak, thank you for the hints, which gluster are you using?
>
> As you can see from my previous email that the RDMA connection tested with
> qperf. It is working as expected. In my case the clients are servers as
> well, they are hosts for the ovirt. Disabling selinux is nor recommended by
> ovirt, but i will give a try.
>
>
>
> Am 03.03.2017 7:50 vorm. schrieb "Deepak Naidu" <dnaidu at
nvidia.com>:
>
> I have been testing glusterfs over RDMA & below is the command I use.
> Reading up the logs, it looks like your IB(InfiniBand) device is not being
> initialized. I am not sure if u have an issue on the client IB or the
> storage server IB. Also have you configured ur IB devices correctly. I am
> using IPoIB.
>
> Can you check your firewall, disable selinux, I think, you might have
> checked it already ?
>
>
>
> *mount -t glusterfs -o transport=rdma storageN1:/vol0 /mnt/vol0*
>
>
>
>
>
> ? *The below error seems if you have issue starting your volume.
> I had issue, when my transport was set to tcp,rdma. I had to force start my
> volume. If I had set it only to tcp on the volume, the volume would start
> easily.*
>
>
>
> [2017-03-02 11:49:47.829391] E [MSGID: 114022]
[client.c:2530:client_init_rpc]
> 0-GluReplica-client-2: failed to initialize RPC
> [2017-03-02 11:49:47.829413] E [MSGID: 101019] [xlator.c:433:xlator_init]
> 0-GluReplica-client-2: Initialization of volume
'GluReplica-client-2'
> failed, review your volfile again
> [2017-03-02 11:49:47.829425] E [MSGID: 101066]
> [graph.c:324:glusterfs_graph_init] 0-GluReplica-client-2: initializing
> translator failed
> [2017-03-02 11:49:47.829436] E [MSGID: 101176]
> [graph.c:673:glusterfs_graph_activate] 0-graph: init failed
>
>
>
> ? *The below error seems if you have issue with IB device. If not
> configured properly.*
>
>
>
> [2017-03-02 11:49:47.828996] W [MSGID: 103071]
[rdma.c:4589:__gf_rdma_ctx_create]
> 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such
device]
> [2017-03-02 11:49:47.829067] W [MSGID: 103055] [rdma.c:4896:init]
> 0-GluReplica-client-2: Failed to initialize IB Device
> [2017-03-02 11:49:47.829080] W [rpc-transport.c:354:rpc_transport_load]
> 0-rpc-transport: 'rdma' initialization failed
>
>
>
>
>
> --
>
> Deepak
>
>
>
>
>
> *From:* gluster-users-bounces at gluster.org [mailto:gluster-users-bounces@
> gluster.org] *On Behalf Of *Sahina Bose
> *Sent:* Thursday, March 02, 2017 10:26 PM
> *To:* Arman Khalatyan; gluster-users at gluster.org; Rafi Kavungal
Chundattu
> Parambil
> *Cc:* users
> *Subject:* Re: [Gluster-users] [ovirt-users] Hot to force glusterfs to
> use RDMA?
>
>
>
> [Adding gluster users to help with error]
>
> [2017-03-02 11:49:47.828996] W [MSGID: 103071]
[rdma.c:4589:__gf_rdma_ctx_create]
> 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such
device]
>
>
>
> On Thu, Mar 2, 2017 at 5:36 PM, Arman Khalatyan <arm2arm at
gmail.com> wrote:
>
> BTW RDMA is working as expected:
> root at clei26 ~]# qperf clei22.vib tcp_bw tcp_lat
> tcp_bw:
> bw = 475 MB/sec
> tcp_lat:
> latency = 52.8 us
> [root at clei26 ~]#
>
> thank you beforehand.
>
> Arman.
>
>
>
> On Thu, Mar 2, 2017 at 12:54 PM, Arman Khalatyan <arm2arm at
gmail.com>
> wrote:
>
> just for reference:
> gluster volume info
>
> Volume Name: GluReplica
> Type: Replicate
> Volume ID: ee686dfe-203a-4caa-a691-26353460cc48
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x (2 + 1) = 3
> Transport-type: tcp,rdma
> Bricks:
> Brick1: 10.10.10.44:/zclei22/01/glu
> Brick2: 10.10.10.42:/zclei21/01/glu
> Brick3: 10.10.10.41:/zclei26/01/glu (arbiter)
> Options Reconfigured:
> network.ping-timeout: 30
> server.allow-insecure: on
> storage.owner-gid: 36
> storage.owner-uid: 36
> cluster.data-self-heal-algorithm: full
> features.shard: on
> cluster.server-quorum-type: server
> cluster.quorum-type: auto
> network.remote-dio: enable
> cluster.eager-lock: enable
> performance.stat-prefetch: off
> performance.io-cache: off
> performance.read-ahead: off
> performance.quick-read: off
> performance.readdir-ahead: on
> nfs.disable: on
>
>
>
> [root at clei21 ~]# gluster volume status
> Status of volume: GluReplica
> Gluster process TCP Port RDMA Port Online
> Pid
> ------------------------------------------------------------
> ------------------
> Brick 10.10.10.44:/zclei22/01/glu 49158 49159 Y
> 15870
> Brick 10.10.10.42:/zclei21/01/glu 49156 49157 Y
> 17473
> Brick 10.10.10.41:/zclei26/01/glu 49153 49154 Y
> 18897
> Self-heal Daemon on localhost N/A N/A Y
> 17502
> Self-heal Daemon on 10.10.10.41 N/A N/A Y
> 13353
> Self-heal Daemon on 10.10.10.44 N/A N/A Y
> 32745
>
> Task Status of Volume GluReplica
> ------------------------------------------------------------
> ------------------
> There are no active volume tasks
>
>
>
> On Thu, Mar 2, 2017 at 12:52 PM, Arman Khalatyan <arm2arm at
gmail.com>
> wrote:
>
> I am not able to mount with RDMA over cli....
>
> Are there some volfile parameters needs to be tuned?
> /usr/bin/mount -t glusterfs -o backup-volfile-servers=10.10.
> 10.44:10.10.10.42:10.10.10.41,transport=rdma 10.10.10.44:/GluReplica /mnt
>
> [2017-03-02 11:49:47.795511] I [MSGID: 100030] [glusterfsd.c:2454:main]
> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.9
> (args: /usr/sbin/glusterfs --volfile-server=10.10.10.44
> --volfile-server=10.10.10.44 --volfile-server=10.10.10.42
> --volfile-server=10.10.10.41 --volfile-server-transport=rdma
> --volfile-id=/GluReplica.rdma /mnt)
> [2017-03-02 11:49:47.812699] I [MSGID: 101190]
[event-epoll.c:628:event_dispatch_epoll_worker]
> 0-epoll: Started thread with index 1
> [2017-03-02 11:49:47.825210] I [MSGID: 101190]
[event-epoll.c:628:event_dispatch_epoll_worker]
> 0-epoll: Started thread with index 2
> [2017-03-02 11:49:47.828996] W [MSGID: 103071]
[rdma.c:4589:__gf_rdma_ctx_create]
> 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such
device]
> [2017-03-02 11:49:47.829067] W [MSGID: 103055] [rdma.c:4896:init]
> 0-GluReplica-client-2: Failed to initialize IB Device
> [2017-03-02 11:49:47.829080] W [rpc-transport.c:354:rpc_transport_load]
> 0-rpc-transport: 'rdma' initialization failed
> [2017-03-02 11:49:47.829272] W [rpc-clnt.c:1070:rpc_clnt_connection_init]
> 0-GluReplica-client-2: loading of new rpc-transport failed
> [2017-03-02 11:49:47.829325] I [MSGID: 101053]
[mem-pool.c:641:mem_pool_destroy]
> 0-GluReplica-client-2: size=588 max=0 total=0
> [2017-03-02 11:49:47.829371] I [MSGID: 101053]
[mem-pool.c:641:mem_pool_destroy]
> 0-GluReplica-client-2: size=124 max=0 total=0
> [2017-03-02 11:49:47.829391] E [MSGID: 114022]
[client.c:2530:client_init_rpc]
> 0-GluReplica-client-2: failed to initialize RPC
> [2017-03-02 11:49:47.829413] E [MSGID: 101019] [xlator.c:433:xlator_init]
> 0-GluReplica-client-2: Initialization of volume
'GluReplica-client-2'
> failed, review your volfile again
> [2017-03-02 11:49:47.829425] E [MSGID: 101066]
> [graph.c:324:glusterfs_graph_init] 0-GluReplica-client-2: initializing
> translator failed
> [2017-03-02 11:49:47.829436] E [MSGID: 101176]
> [graph.c:673:glusterfs_graph_activate] 0-graph: init failed
> [2017-03-02 11:49:47.830003] W [glusterfsd.c:1327:cleanup_and_exit]
> (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x3c1) [0x7f524c9dbeb1]
> -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x172) [0x7f524c9d65d2]
> -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f524c9d5b4b] ) 0-:
> received signum (1), shutting down
> [2017-03-02 11:49:47.830053] I [fuse-bridge.c:5794:fini] 0-fuse:
> Unmounting '/mnt'.
> [2017-03-02 11:49:47.831014] W [glusterfsd.c:1327:cleanup_and_exit]
> (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f524b343dc5]
> -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f524c9d5cd5]
> -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f524c9d5b4b] ) 0-:
> received signum (15), shutting down
> [2017-03-02 11:49:47.831014] W [glusterfsd.c:1327:cleanup_and_exit]
> (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f524b343dc5]
> -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f524c9d5cd5]
> -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f524c9d5b4b] ) 0-:
> received signum (15), shutting down
>
>
>
> On Thu, Mar 2, 2017 at 12:11 PM, Sahina Bose <sabose at redhat.com>
wrote:
>
> You will need to pass additional mount options while creating the storage
> domain (transport=rdma)
>
> Please let us know if this works.
>
>
>
> On Thu, Mar 2, 2017 at 2:42 PM, Arman Khalatyan <arm2arm at
gmail.com> wrote:
>
> Hi,
>
> Are there way to force the connections over RDMA only?
>
> If I check host mounts I cannot see rdma mount option:
> mount -l| grep gluster
> 10.10.10.44:/GluReplica on
/rhev/data-center/mnt/glusterSD/10.10.10.44:_GluReplica
> type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,
> allow_other,max_read=131072)
>
> I have glusterized 3 nodes:
>
> GluReplica
> Volume ID:
> ee686dfe-203a-4caa-a691-26353460cc48
> Volume Type:
> Replicate (Arbiter)
> Replica Count:
> 2 + 1
> Number of Bricks:
> 3
> Transport Types:
> TCP, RDMA
> Maximum no of snapshots:
> 256
> Capacity:
> 3.51 TiB total, 190.56 GiB used, 3.33 TiB free
>
>
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
>
>
>
>
>
>
>
> ------------------------------
>
> This email message is for the sole use of the intended recipient(s) and
> may contain confidential information. Any unauthorized review, use,
> disclosure or distribution is prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
> ------------------------------
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170303/d9f786ba/attachment.html>