Gluster users - Nov 2011 - RDMA/Ethernet wi ROCEE - failed to modify QP to RTR

Did any RDMA/Ethernet users see this Gluster error?  If so do you know what
caused it and how to fix?  If you haven't seen it, what RPMs and
configuration do you use specific to RDMA/Ethernet?

[2011-11-10 10:30:20.595801] C
[rdma.c:2417:rdma_connect_qp]0-rpc-transport/rdma: Failed to modify QP to RTR
[2011-11-10 10:30:20.595930] E [rdma.c:4159:rdma_handshake_pollin]
0-rpc-transport/rdma: rdma.management: failed to connect with remote QP

I see this when I run RDMA over Ethernet using ROCEE RPMs, but when I run over
Infiniband using RHEL 6.2-, it runs fine.  On the same Ethernet configuration,
Gluster/TCP runs fine, NFS/RDMA runs fine as does AMQP app.  But qperf and rping
utilities fail in the same way.  Firmware on the HCAs is not the latest, is it
worth risk to upgrade?

I went into debugger and found line where qperf fails, it's near line 2056
in rdma.c in qperf sources (qperf-debuginfo,  I did Makefile)

(gdb)
2088        } else if (dev->trans == IBV_QPT_RC) {
(gdb)
2090            flags = IBV_QP_STATE              |
(gdb)
2097            if (ibv_modify_qp(dev->qp, &rtr_attr, flags) != 0)
(gdb)
2098                error(SYS, "failed to modify QP to RTR");
(gdb)

Gluster fails in rdma_connect_qp() calling the same routine, but perhaps with
different parameters.

Hi Ben,

Did any RDMA/Ethernet users see this Gluster error?  If so do you know
what
> caused it and how to fix?  If you haven't seen it, what RPMs and
> configuration do you use specific to RDMA/Ethernet?
>
>
We haven't tested Gluster with RDMA/Ethernet till date. If you have the
setup, can you try testing with master branch code with below patches
applied. (apply the patches in order).

http://review.gluster.com/148
http://review.gluster.com/149
http://review.gluster.com/201

Let us know if it works for you with these patches.

Regards,
Amar


> [2011-11-10 10:30:20.595801] C
> [rdma.c:2417:rdma_connect_qp]0-rpc-transport/rdma: Failed to modify QP to
> RTR
> [2011-11-10 10:30:20.595930] E [rdma.c:4159:rdma_handshake_pollin]
> 0-rpc-transport/rdma: rdma.management: failed to connect with remote QP
>
> I see this when I run RDMA over Ethernet using ROCEE RPMs, but when I run
> over Infiniband using RHEL 6.2-, it runs fine.  On the same Ethernet
> configuration, Gluster/TCP runs fine, NFS/RDMA runs fine as does AMQP app.
>  But qperf and rping utilities fail in the same way.  Firmware on the HCAs
> is not the latest, is it worth risk to upgrade?
>
> I went into debugger and found line where qperf fails, it's near line
2056
> in rdma.c in qperf sources (qperf-debuginfo,  I did Makefile)
>
> (gdb)
> 2088        } else if (dev->trans == IBV_QPT_RC) {
> (gdb)
> 2090            flags = IBV_QP_STATE              |
> (gdb)
> 2097            if (ibv_modify_qp(dev->qp, &rtr_attr, flags) != 0)
> (gdb)
> 2098                error(SYS, "failed to modify QP to RTR");
> (gdb)
>
> Gluster fails in rdma_connect_qp() calling the same routine, but perhaps
> with different parameters.
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20111114/845ac44d/attachment.html>