For instance, if I ssh over to 2-ib, and ask for a peer status, it shows the peers to to be 1-ib-r 2-ib-r and *155.52.48.1* (the last is the ethernet side of, not the IB side) I guess my question is this: How do I get the first node to use the ib side when connecting to the other nodes. (both of these networks are public.) Do I have to do something interesting with routes? I need to fix this. Don''t most people with IB also have an internet side to things? How do their networks and any local routing info get set up? Am I ,missing something really obvious? Matt ------ Matt Temple Director, Research Computing Dana-Farber Cancer Institute. We have two --f46d042f9326318ea804d12b703c Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>We have two networks and I have 4 servers I'm trying to turn into a distributed/replicated Gluster storage cluster.<br><br></div><div>Each of the servers has two addresses, one ethernet, one IB.<br> </div><div>The externet addresses are 155.42.48.1-4 and <br></div><div>the Infiniband addresses are 172.24.215.1-4<br></div><div><br>I login to the first server ("1") which is 155.52.48.1.<br><br></div><div>Let's say the servers are named 1,2,1R and 2R (on the ethernet side), but <br> =A01-IB, 2-IB, 1-IB-R and 2-IB-R on he Infiniband side.<br></div><div>The Volume is to be mounted by other servers in the IB network by native Gluster clients.<br><br></div><div>so, I login to "1" over the ethernet network and do a peer probe.<br> </div><div>All looks good from "1" with peer probe and peer status.<br><br></div><div>I probe the peers like this, successfully<br><br></div><div><font face=3D"courier new,monospace">gluster peer probe 2-ib<br></font></div> <div><font face=3D"courier new,monospace">gluster peer probe 1-ib-r<br></font></div><div><font face=3D"courier new,monospace">gluster peer prove 2-ib-r<br><br></font></div><div><font face=3D"courier new,monospace"><font face=3D"arial,helvetica,sans-serif">Peer status looks right.<br> </font></font>I issue the following command which fails, of course:<br><br><span style=3D"font-size:15px;font-family:'Courier New';color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">gluster volume create gf2 replica 2 transport tcp,rdma \</span><br> <span style=3D"font-size:15px;font-family:'Courier New';color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">1-ib:/mnt/d0-0 1-ib-r:/mnt/d0-0 </span><span style=3D"font-size:15px;font-family:'Courier New';color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">2-ib:/mnt/d0-0 2-ib-r:/mnt/d0-0</span></div> <div><font face=3D"courier new,monospace"><font face=3D"arial,helvetica,sans-serif"><br>From any of those nodes, peer status is wrong:<br>For instance, if I ssh over to 2-ib, <br>and ask for a peer status, it shows the peers to to be <br> <br></font></font></div><div><font face=3D"courier new,monospace"><font face=3D"arial,helvetica,sans-serif"><font face=3D"courier new,monospace">1-ib-r<br>2-ib-r <br>and <b>155.52.48.1</b></font> (the last is the ethernet side of, not the IB side)<br> <br></font></font></div><div><font face=3D"courier new,monospace"><font face=3D"arial,helvetica,sans-serif">I guess my question is this:=A0=A0 How do I get the first node to use the ib side when<br></font></font></div><div><font face=3D"courier new,monospace"><font face=3D"arial,helvetica,sans-serif">connecting to the other nodes. (both of these networks are public.)=A0=A0 Do I have to<br> </font></font></div><div><font face=3D"courier new,monospace"><font face=3D"arial,helvetica,sans-serif">do something interesting with routes?<br></font></font></div><div><font face=3D"courier new,monospace"><font face=3D"arial,helvetica,sans-serif"><br> </font></font></div><div>I need to fix this.=A0=A0 Don't most people with IB also have an internet side to things?<br></div><div>How do their networks and any local routing info get set up?<br></div><div>Am I ,missing something really obvious?<br> <br></div><div>Matt<br></div><div><br></div><br><div><div>------<br>Matt Temple<br>Director, Research Computing<br>Dana-Farber Cancer Institute.<br></div><br></div>We have two=A0 </div> --f46d042f9326318ea804d12b703c--