Gluster users - Dec 2012 - infiniband replicated distributed setup.--- network setup question...

We have two networks and I have 4 servers I'm trying to turn into a
distributed/replicated Gluster storage cluster.

Each of the servers has two addresses, one ethernet, one IB.
The externet addresses are 155.42.48.1-4 and
the Infiniband addresses are 172.24.215.1-4

I login to the first server ("1") which is 155.52.48.1.

Let's say the servers are named 1,2,1R and 2R (on the ethernet side), but
 1-IB, 2-IB, 1-IB-R and 2-IB-R on he Infiniband side.
The Volume is to be mounted by other servers in the IB network by native
Gluster clients.

so, I login to "1" over the ethernet network and do a peer probe.
All looks good from "1" with peer probe and peer status.

I probe the peers like this, successfully

gluster peer probe 2-ib
gluster peer probe 1-ib-r
gluster peer prove 2-ib-r

Peer status looks right.
I issue the following command which fails, of course:

gluster volume create gf2 replica 2 transport tcp,rdma \
1-ib:/mnt/d0-0 1-ib-r:/mnt/d0-0 2-ib:/mnt/d0-0 2-ib-r:/mnt/d0-0
>From any of those nodes, peer status is wrong:
For instance, if I ssh over to 2-ib,
and ask for a peer status, it shows the peers to to be

1-ib-r
2-ib-r
and *155.52.48.1* (the last is the ethernet side of, not the IB side)

I guess my question is this:   How do I get the first node to use the ib
side when
connecting to the other nodes. (both of these networks are public.)   Do I
have to
do something interesting with routes?

I need to fix this.   Don't most people with IB also have an internet side
to things?
How do their networks and any local routing info get set up?
Am I ,missing something really obvious?

Matt


------
Matt Temple
Director, Research Computing
Dana-Farber Cancer Institute.

We have two
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121218/ccb3f734/attachment.html>

On 12/18/2012 7:33 PM, Matthew Temple wrote:
> From any of those nodes, peer status is wrong:
> For instance, if I ssh over to 2-ib,
> and ask for a peer status, it shows the peers to to be
>
> 1-ib-r
> 2-ib-r
> and *155.52.48.1* (the last is the ethernet side of, not the IB side)
>
> I guess my question is this:   How do I get the first node to use the 
> ib side when
> connecting to the other nodes. (both of these networks are public.)   
> Do I have to
> do something interesting with routes?
Just probe the first server by name from any of the other three.  That 
will fix the peers.

Thanks,
Shawn