Fedele Stabile
2017-Jan-18 11:56 UTC
[Gluster-users] RDMA transport problems in GLUSTER on host with MIC
Hi, it happens that RDMA gluster transport does not works anymore after I have configured ibscif virtual connector for Infiniband?in a server with a XeonPHI coprocessor. I have CentOS 6.6 and GLUSTER 3.8.5, OFED 3.12-1 MPSS 3.5.2 and I have followed the installation instructions of MPSS_Users_Guide (Revision 3.5) that suggested to remove compat-rdma-devel and compat-rdma packages. I have noticed that running the command: ib_send_bw gives the following error: # ib_send_bw ************************************ * Waiting for client to connect... * ************************************ --------------------------------------------------------------------- ------------------ ????????????????????Send BW Test ?Dual-port???????: OFF????????Device?????????: scif0 ?Number of qps???: 1????????Transport type : IW ?Connection type : RC????????Using SRQ??????: OFF ?RX depth????????: 512 ?CQ Moderation???: 100 ?Mtu?????????????: 2048[B] ?Link type???????: Ethernet ?Gid index???????: 0 ?Max inline data : 0[B] ?rdma_cm QPs?????: OFF ?Data ex. method : Ethernet --------------------------------------------------------------------- ------------------ ?local address: LID 0x3e8 QPN 0x0003 PSN 0x123123 ?GID: 76:121:186:102:03:119:00:00:00:00:00:00:00:00:00:00 ethernet_read_keys: Couldn't read remote address ?Unable to read to socket/rdam_cm Failed to exchange data between server and clients Instead using the output of the command ib_send_bw -d qib0 gives correct results: # ib_send_bw -d qib0 ************************************ * Waiting for client to connect... * ************************************ --------------------------------------------------------------------- ------------------ ????????????????????Send BW Test ?Dual-port???????: OFF????????Device?????????: qib0 ?Number of qps???: 1????????Transport type : IB ?Connection type : RC????????Using SRQ??????: OFF ?RX depth????????: 512 ?CQ Moderation???: 100 ?Mtu?????????????: 2048[B] ?Link type???????: IB ?Max inline data : 0[B] ?rdma_cm QPs?????: OFF ?Data ex. method : Ethernet --------------------------------------------------------------------- ------------------ ?local address: LID 0x0a QPN 0x0169 PSN 0xe0b768 ?remote address: LID 0x20 QPN 0x28b280 PSN 0xc3008c --------------------------------------------------------------------- ------------------ ?#bytes?????#iterations????BW peak[MB/sec]????BW average[MB/sec]???MsgRate[Mpps] ?65536??????1000?????????????0.00???????????????2160.87???????????0.034 574 --------------------------------------------------------------------- ------------------
Anoop C S
2017-Jan-20 07:28 UTC
[Gluster-users] RDMA transport problems in GLUSTER on host with MIC
On Wed, 2017-01-18 at 12:56 +0100, Fedele Stabile wrote:> Hi, > it happens that RDMA gluster transport does not works anymore > after I have configured ibscif virtual connector for Infiniband?in a > server with a XeonPHI coprocessor. > > I have CentOS 6.6 and GLUSTER 3.8.5, OFED 3.12-1 MPSS 3.5.2 and I have > followed the installation instructions of MPSS_Users_Guide (Revision > 3.5) that suggested to remove > compat-rdma-devel and compat-rdma packages. >It would help if you could somehow clearly understand the reason for removing those packages. May be they are critical and not intended to be removed. Please ask for help from OFED.> I have noticed that running the command: > ib_send_bw > gives the following error: > > # ib_send_bw > > ************************************ > * Waiting for client to connect... * > ************************************ > --------------------------------------------------------------------- > ------------------ > ????????????????????Send BW Test > ?Dual-port???????: OFF????????Device?????????: scif0 > ?Number of qps???: 1????????Transport type : IW > ?Connection type : RC????????Using SRQ??????: OFF > ?RX depth????????: 512 > ?CQ Moderation???: 100 > ?Mtu?????????????: 2048[B] > ?Link type???????: Ethernet > ?Gid index???????: 0 > ?Max inline data : 0[B] > ?rdma_cm QPs?????: OFF > ?Data ex. method : Ethernet > --------------------------------------------------------------------- > ------------------ > ?local address: LID 0x3e8 QPN 0x0003 PSN 0x123123 > ?GID: 76:121:186:102:03:119:00:00:00:00:00:00:00:00:00:00 > ethernet_read_keys: Couldn't read remote address > ?Unable to read to socket/rdam_cm > Failed to exchange data between server and clients >The above error have nothing to do with GlusterFS. Can you please give more context on what failed for you while trying out GlusterFS with RDMA transport?> Instead using the output of the command > > ib_send_bw -d qib0 > > gives correct results: > > # ib_send_bw -d qib0 > > ************************************ > * Waiting for client to connect... * > ************************************ > --------------------------------------------------------------------- > ------------------ > ????????????????????Send BW Test > ?Dual-port???????: OFF????????Device?????????: qib0 > ?Number of qps???: 1????????Transport type : IB > ?Connection type : RC????????Using SRQ??????: OFF > ?RX depth????????: 512 > ?CQ Moderation???: 100 > ?Mtu?????????????: 2048[B] > ?Link type???????: IB > ?Max inline data : 0[B] > ?rdma_cm QPs?????: OFF > ?Data ex. method : Ethernet > --------------------------------------------------------------------- > ------------------ > ?local address: LID 0x0a QPN 0x0169 PSN 0xe0b768 > ?remote address: LID 0x20 QPN 0x28b280 PSN 0xc3008c > --------------------------------------------------------------------- > ------------------ > ?#bytes?????#iterations????BW peak[MB/sec]????BW > average[MB/sec]???MsgRate[Mpps] > ?65536??????1000?????????????0.00???????????????2160.87???????????0.034 > 574 > --------------------------------------------------------------------- > ------------------ > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users