I would like to use a client as a router between two tcp networks (eth0 and eth1) but it is unclear to me how to configure this in modprobe.conf. Any suggestions would be greatly appreciated. _________________________________________ Ron Jerome Programmer/Analyst National Research Council Canada M-2, 1200 Montreal Road, Ottawa, Ontario K1A 0R6 Government of Canada _________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080115/062f44cf/attachment-0002.html
On Tue, Jan 15, 2008 at 02:27:39PM -0500, Jerome, Ron wrote:> I would like to use a client as a router between two tcp networks (eth0 > and eth1) but it is unclear to me how to configure this in > modprobe.conf. > > > Any suggestions would be greatly appreciated.http://manual.lustre.org/manual/LustreManual16_HTML/DynamicHTML-06-1.html Please read section 3.2.2. Isaac
Hi Isaac, Yes I had read the documentation prior to posting my question but a couple of things were still unclear to me, but first a bit more background... My lustre filesystem & my cluster reside on a private (192.169.xxx.xxx) network I have a couple of machines which are connected to both the private network and our public network and I would like a machine which only has access to the public network to mount the lustre filesystem on the private network via one of the nodes which spans both. So my questions are... a) is it possible to route to and from a private (192.168.xxx.xxx) Ethernet network to a public one (132.246.xxx.xxx)? The documents only show routing between Ethernet and other networks such as elan, Myrinet etc. b) if this is possible, what is the correct syntax for the modprobe.conf file? I currently have... options lnet networks="tcp0(eth0),tcp1(eth1)" routes="tcp1 192.168.0.2 at tcp" c) from a client on the public network what would the syntax be for accessing (mounting) the mgs through the routing node? Thanks very much, Ron.> -----Original Message----- > From: He.Huang at Sun.COM [mailto:He.Huang at Sun.COM] > Sent: January 16, 2008 8:25 AM > To: Jerome, Ron > Cc: Lustre-discuss at clusterfs.com > Subject: Re: [Lustre-discuss] How to configure routing > > On Tue, Jan 15, 2008 at 02:27:39PM -0500, Jerome, Ron wrote: > > I would like to use a client as a router between two tcp networks > (eth0 > > and eth1) but it is unclear to me how to configure this in > > modprobe.conf. > > > > > > Any suggestions would be greatly appreciated. > > http://manual.lustre.org/manual/LustreManual16_HTML/DynamicHTML-06- > 1.html > > Please read section 3.2.2. > > Isaac
Hi,
Interesting scenario if routing between public and private
subnets for mounting lustre filesystem is possible
without disturbing the network admins for public subnet :-) One way
around would be
to build tunnels like ipip for example between the gateway machine on
the cluster
that has access to both nets and with other hosts on the public net.
Then Lustre mount
could work over the tunnel with static routes to access the MGS/MDS/OST
over the tunnel.
Regards
Balagopal
Jerome, Ron wrote:> Hi Isaac,
>
> Yes I had read the documentation prior to posting my question but a
> couple of things were still unclear to me, but first a bit more
> background... My lustre filesystem & my cluster reside on a private
> (192.169.xxx.xxx) network I have a couple of machines which are
> connected to both the private network and our public network and I would
> like a machine which only has access to the public network to mount the
> lustre filesystem on the private network via one of the nodes which
> spans both.
>
> So my questions are...
>
> a) is it possible to route to and from a private (192.168.xxx.xxx)
> Ethernet network to a public one (132.246.xxx.xxx)? The documents only
> show routing between Ethernet and other networks such as elan, Myrinet
> etc.
>
> b) if this is possible, what is the correct syntax for the modprobe.conf
> file? I currently have...
>
> options lnet networks="tcp0(eth0),tcp1(eth1)" routes="tcp1
> 192.168.0.2 at tcp"
>
> c) from a client on the public network what would the syntax be for
> accessing (mounting) the mgs through the routing node?
>
>
> Thanks very much,
>
> Ron.
>
>
>> -----Original Message-----
>> From: He.Huang at Sun.COM [mailto:He.Huang at Sun.COM]
>> Sent: January 16, 2008 8:25 AM
>> To: Jerome, Ron
>> Cc: Lustre-discuss at clusterfs.com
>> Subject: Re: [Lustre-discuss] How to configure routing
>>
>> On Tue, Jan 15, 2008 at 02:27:39PM -0500, Jerome, Ron wrote:
>>
>>> I would like to use a client as a router between two tcp
networks
>>>
>> (eth0
>>
>>> and eth1) but it is unclear to me how to configure this in
>>> modprobe.conf.
>>>
>>>
>>> Any suggestions would be greatly appreciated.
>>>
>> http://manual.lustre.org/manual/LustreManual16_HTML/DynamicHTML-06-
>> 1.html
>>
>> Please read section 3.2.2.
>>
>> Isaac
>>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at clusterfs.com
> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
>
Ron,
You don''t have to rely on IP routing. You can use Lustre routing
instead. You have the basic config line below. The external clients
need to know about both networks and the routers. You should be able to
use the same LNET module options everywhere. You would do a mount
like...
mount -t lustre <mds ip>@tcp1:/lustre/client /mnt
Assuming the private network is tcp1 and tcp0 is the public network.
--Shane
-----Original Message-----
From: lustre-discuss-bounces at clusterfs.com
[mailto:lustre-discuss-bounces at clusterfs.com] On Behalf Of Balagopal
Pillai
Sent: Wednesday, January 16, 2008 9:27 AM
To: Jerome, Ron
Cc: He.Huang at Sun.COM; Lustre-discuss at clusterfs.com
Subject: Re: [Lustre-discuss] How to configure routing
Hi,
Interesting scenario if routing between public and private
subnets for mounting lustre filesystem is possible
without disturbing the network admins for public subnet :-) One way
around would be
to build tunnels like ipip for example between the gateway machine on
the cluster
that has access to both nets and with other hosts on the public net.
Then Lustre mount
could work over the tunnel with static routes to access the MGS/MDS/OST
over the tunnel.
Regards
Balagopal
Jerome, Ron wrote:> Hi Isaac,
>
> Yes I had read the documentation prior to posting my question but a
> couple of things were still unclear to me, but first a bit more
> background... My lustre filesystem & my cluster reside on a private
> (192.169.xxx.xxx) network I have a couple of machines which are
> connected to both the private network and our public network and I
would> like a machine which only has access to the public network to mount
the> lustre filesystem on the private network via one of the nodes which
> spans both.
>
> So my questions are...
>
> a) is it possible to route to and from a private (192.168.xxx.xxx)
> Ethernet network to a public one (132.246.xxx.xxx)? The documents only
> show routing between Ethernet and other networks such as elan, Myrinet
> etc.
>
> b) if this is possible, what is the correct syntax for the
modprobe.conf> file? I currently have...
>
> options lnet networks="tcp0(eth0),tcp1(eth1)" routes="tcp1
> 192.168.0.2 at tcp"
>
> c) from a client on the public network what would the syntax be for
> accessing (mounting) the mgs through the routing node?
>
>
> Thanks very much,
>
> Ron.
>
>
>> -----Original Message-----
>> From: He.Huang at Sun.COM [mailto:He.Huang at Sun.COM]
>> Sent: January 16, 2008 8:25 AM
>> To: Jerome, Ron
>> Cc: Lustre-discuss at clusterfs.com
>> Subject: Re: [Lustre-discuss] How to configure routing
>>
>> On Tue, Jan 15, 2008 at 02:27:39PM -0500, Jerome, Ron wrote:
>>
>>> I would like to use a client as a router between two tcp
networks
>>>
>> (eth0
>>
>>> and eth1) but it is unclear to me how to configure this in
>>> modprobe.conf.
>>>
>>>
>>> Any suggestions would be greatly appreciated.
>>>
>> http://manual.lustre.org/manual/LustreManual16_HTML/DynamicHTML-06-
>> 1.html
>>
>> Please read section 3.2.2.
>>
>> Isaac
>>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at clusterfs.com
> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
>
_______________________________________________
Lustre-discuss mailing list
Lustre-discuss at clusterfs.com
https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
That''s what I was hoping, but I''m still having trouble getting
the
routing to work.
My MGS is at 192.168.0.200 at tcp
On my gateway node, I have...
options lnet networks="tcp0(eth0),tcp1(eth1)" routes="tcp1
192.168.0.2 at tcp ;\
tcp0
132.246.xxx.xxx at tcp1"
Then from the node solely connected to the public (132.246) network, I
try to mount
mount -t lustre 192.168.0.200 at tcp:/data /mnt/data
but regrettably this does not work :-(
Ron.
> -----Original Message-----
> From: Canon, Richard Shane [mailto:canonrs at ornl.gov]
> Sent: January 16, 2008 10:03 AM
> To: Jerome, Ron
> Cc: He.Huang at Sun.COM; Lustre-discuss at clusterfs.com
> Subject: RE: [Lustre-discuss] How to configure routing
>
>
> Ron,
>
> You don''t have to rely on IP routing. You can use Lustre routing
> instead. You have the basic config line below. The external clients
> need to know about both networks and the routers. You should be able
> to
> use the same LNET module options everywhere. You would do a mount
> like...
>
> mount -t lustre <mds ip>@tcp1:/lustre/client /mnt
>
> Assuming the private network is tcp1 and tcp0 is the public network.
>
>
> --Shane
>
> -----Original Message-----
> From: lustre-discuss-bounces at clusterfs.com
> [mailto:lustre-discuss-bounces at clusterfs.com] On Behalf Of Balagopal
> Pillai
> Sent: Wednesday, January 16, 2008 9:27 AM
> To: Jerome, Ron
> Cc: He.Huang at Sun.COM; Lustre-discuss at clusterfs.com
> Subject: Re: [Lustre-discuss] How to configure routing
>
> Hi,
>
> Interesting scenario if routing between public and private
> subnets for mounting lustre filesystem is possible
> without disturbing the network admins for public subnet :-) One way
> around would be
> to build tunnels like ipip for example between the gateway machine on
> the cluster
> that has access to both nets and with other hosts on the public net.
> Then Lustre mount
> could work over the tunnel with static routes to access the
MGS/MDS/OST> over the tunnel.
>
> Regards
> Balagopal
>
> Jerome, Ron wrote:
> > Hi Isaac,
> >
> > Yes I had read the documentation prior to posting my question but a
> > couple of things were still unclear to me, but first a bit more
> > background... My lustre filesystem & my cluster reside on a
private
> > (192.169.xxx.xxx) network I have a couple of machines which are
> > connected to both the private network and our public network and I
> would
> > like a machine which only has access to the public network to mount
> the
> > lustre filesystem on the private network via one of the nodes which
> > spans both.
> >
> > So my questions are...
> >
> > a) is it possible to route to and from a private (192.168.xxx.xxx)
> > Ethernet network to a public one (132.246.xxx.xxx)? The documents
> only
> > show routing between Ethernet and other networks such as elan,
> Myrinet
> > etc.
> >
> > b) if this is possible, what is the correct syntax for the
> modprobe.conf
> > file? I currently have...
> >
> > options lnet networks="tcp0(eth0),tcp1(eth1)"
routes="tcp1
> > 192.168.0.2 at tcp"
> >
> > c) from a client on the public network what would the syntax be for
> > accessing (mounting) the mgs through the routing node?
> >
> >
> > Thanks very much,
> >
> > Ron.
> >
> >
> >> -----Original Message-----
> >> From: He.Huang at Sun.COM [mailto:He.Huang at Sun.COM]
> >> Sent: January 16, 2008 8:25 AM
> >> To: Jerome, Ron
> >> Cc: Lustre-discuss at clusterfs.com
> >> Subject: Re: [Lustre-discuss] How to configure routing
> >>
> >> On Tue, Jan 15, 2008 at 02:27:39PM -0500, Jerome, Ron wrote:
> >>
> >>> I would like to use a client as a router between two tcp
> networks
> >>>
> >> (eth0
> >>
> >>> and eth1) but it is unclear to me how to configure this in
> >>> modprobe.conf.
> >>>
> >>>
> >>> Any suggestions would be greatly appreciated.
> >>>
> >>
http://manual.lustre.org/manual/LustreManual16_HTML/DynamicHTML-06-
> >> 1.html
> >>
> >> Please read section 3.2.2.
> >>
> >> Isaac
> >>
> >
> > _______________________________________________
> > Lustre-discuss mailing list
> > Lustre-discuss at clusterfs.com
> > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
> >
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at clusterfs.com
> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
Lnet routing is generic - it''s not only able to route between different network types (e.g. tcp0 and o2ib0) but also capable of routing between networks of a same type (e.g. tcp0 and tcp1). Before trying to mount lustre, I''d suggest you ensure lnet is working. A simple way is: lctl ping 192.168.0.200 at tcp. If it works, then go on and try to mount lustre. Otherwise, ''dmesg'' shall shed some light on what went wrong. Isaac On Wed, Jan 16, 2008 at 10:54:31AM -0500, Jerome, Ron wrote:> That''s what I was hoping, but I''m still having trouble getting the > routing to work. > > My MGS is at 192.168.0.200 at tcp > > On my gateway node, I have... > > options lnet networks="tcp0(eth0),tcp1(eth1)" routes="tcp1 > 192.168.0.2 at tcp ;\ > tcp0 > 132.246.xxx.xxx at tcp1" > > Then from the node solely connected to the public (132.246) network, I > try to mount > > mount -t lustre 192.168.0.200 at tcp:/data /mnt/data > > but regrettably this does not work :-( > > > Ron. > > > -----Original Message----- > > From: Canon, Richard Shane [mailto:canonrs at ornl.gov] > > Sent: January 16, 2008 10:03 AM > > To: Jerome, Ron > > Cc: He.Huang at Sun.COM; Lustre-discuss at clusterfs.com > > Subject: RE: [Lustre-discuss] How to configure routing > > > > > > Ron, > > > > You don''t have to rely on IP routing. You can use Lustre routing > > instead. You have the basic config line below. The external clients > > need to know about both networks and the routers. You should be able > > to > > use the same LNET module options everywhere. You would do a mount > > like... > > > > mount -t lustre <mds ip>@tcp1:/lustre/client /mnt > > > > Assuming the private network is tcp1 and tcp0 is the public network. > > > > > > --Shane > > > > -----Original Message----- > > From: lustre-discuss-bounces at clusterfs.com > > [mailto:lustre-discuss-bounces at clusterfs.com] On Behalf Of Balagopal > > Pillai > > Sent: Wednesday, January 16, 2008 9:27 AM > > To: Jerome, Ron > > Cc: He.Huang at Sun.COM; Lustre-discuss at clusterfs.com > > Subject: Re: [Lustre-discuss] How to configure routing > > > > Hi, > > > > Interesting scenario if routing between public and private > > subnets for mounting lustre filesystem is possible > > without disturbing the network admins for public subnet :-) One way > > around would be > > to build tunnels like ipip for example between the gateway machine on > > the cluster > > that has access to both nets and with other hosts on the public net. > > Then Lustre mount > > could work over the tunnel with static routes to access the > MGS/MDS/OST > > over the tunnel. > > > > Regards > > Balagopal > > > > Jerome, Ron wrote: > > > Hi Isaac, > > > > > > Yes I had read the documentation prior to posting my question but a > > > couple of things were still unclear to me, but first a bit more > > > background... My lustre filesystem & my cluster reside on a private > > > (192.169.xxx.xxx) network I have a couple of machines which are > > > connected to both the private network and our public network and I > > would > > > like a machine which only has access to the public network to mount > > the > > > lustre filesystem on the private network via one of the nodes which > > > spans both. > > > > > > So my questions are... > > > > > > a) is it possible to route to and from a private (192.168.xxx.xxx) > > > Ethernet network to a public one (132.246.xxx.xxx)? The documents > > only > > > show routing between Ethernet and other networks such as elan, > > Myrinet > > > etc. > > > > > > b) if this is possible, what is the correct syntax for the > > modprobe.conf > > > file? I currently have... > > > > > > options lnet networks="tcp0(eth0),tcp1(eth1)" routes="tcp1 > > > 192.168.0.2 at tcp" > > > > > > c) from a client on the public network what would the syntax be for > > > accessing (mounting) the mgs through the routing node? > > > > > > > > > Thanks very much, > > > > > > Ron. > > > > > > > > >> -----Original Message----- > > >> From: He.Huang at Sun.COM [mailto:He.Huang at Sun.COM] > > >> Sent: January 16, 2008 8:25 AM > > >> To: Jerome, Ron > > >> Cc: Lustre-discuss at clusterfs.com > > >> Subject: Re: [Lustre-discuss] How to configure routing > > >> > > >> On Tue, Jan 15, 2008 at 02:27:39PM -0500, Jerome, Ron wrote: > > >> > > >>> I would like to use a client as a router between two tcp > > networks > > >>> > > >> (eth0 > > >> > > >>> and eth1) but it is unclear to me how to configure this in > > >>> modprobe.conf. > > >>> > > >>> > > >>> Any suggestions would be greatly appreciated. > > >>> > > >> http://manual.lustre.org/manual/LustreManual16_HTML/DynamicHTML-06- > > >> 1.html > > >> > > >> Please read section 3.2.2. > > >> > > >> Isaac > > >> > > > > > > _______________________________________________ > > > Lustre-discuss mailing list > > > Lustre-discuss at clusterfs.com > > > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > > > > > > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-discuss at clusterfs.com > > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Ron, I am not exactly familiar with your particular setup, but when using tcp is there any reason why you can''t use a single lnet network? I believe that it would be easier for you to set up one socklnd network named tcp0 that contains both interfaces. Then all you have to do is make sure that it is possible through host routes (or other means) for the two networks to route to each other. - -Herb Jerome, Ron wrote:> I would like to use a client as a router between two tcp networks (eth0 > and eth1) but it is unclear to me how to configure this in modprobe.conf. > > > > Any suggestions would be greatly appreciated. > > > > _________________________________________ > Ron Jerome > Programmer/Analyst > National Research Council Canada > M-2, 1200 Montreal Road, Ottawa, Ontario K1A 0R6 > Government of Canada > _________________________________________ > > > ------------------------------------------------------------------------ > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFHjlugP/62XqEEbMYRCuYuAKCPFQiNSJLWcE614FNAni/Q17fzNgCfY7jh SfvQb0mV8uVkvqbRjbbci8c=hZmw -----END PGP SIGNATURE-----
The issue is the fact that the luster filesystem is on a private (192.168.xxx.xxx) network and the client is on a public network, so bidirectional ip traffic routing gets messy (NATing, port forwarding etc). I was hoping to avoid this by using a gateway lustre node that is connected to both networks. Ron.> -----Original Message----- > From: lustre-discuss-bounces at clusterfs.com [mailto:lustre-discuss- > bounces at clusterfs.com] On Behalf Of Herb Wartens > Sent: January 16, 2008 2:32 PM > To: Lustre Discuss > Subject: Re: [Lustre-discuss] How to configure routing > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > > Ron, > I am not exactly familiar with your particular setup, but when using > tcp > is there any reason why you can''t use a single lnet network? Ibelieve> that > it would be easier for you to set up one socklnd network named tcp0 > that > contains both interfaces. Then all you have to do is make sure thatit> is possible > through host routes (or other means) for the two networks to route to > each other. > > - -Herb > > Jerome, Ron wrote: > > I would like to use a client as a router between two tcp networks > (eth0 > > and eth1) but it is unclear to me how to configure this in > modprobe.conf. > > > > > > > > Any suggestions would be greatly appreciated. > > > > > > > > _________________________________________ > > Ron Jerome > > Programmer/Analyst > > National Research Council Canada > > M-2, 1200 Montreal Road, Ottawa, Ontario K1A 0R6 > > Government of Canada > > _________________________________________ > > > > > >---------------------------------------------------------------------> --- > > > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-discuss at clusterfs.com > > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.7 (GNU/Linux) > Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org > > iD8DBQFHjlugP/62XqEEbMYRCuYuAKCPFQiNSJLWcE614FNAni/Q17fzNgCfY7jh > SfvQb0mV8uVkvqbRjbbci8c> =hZmw > -----END PGP SIGNATURE----- > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 IMHO it would be better for you to set up host routing (or whatever it is that you wanted) to be ablr to route between these networks. It would be best to avoid the routing code if it is possible since you would be introducing some new problems by having the routing turned on. With routing enabled you would not have immediate notification that a server went down for example. This causes longer delays on the client when doing things like failover since you would have to wait for a timeout to occur before the client would actually try the failover server. I think with the adaptive timeouts this could be resolved, but I think that things like this would be better to avoid if you don''t especially need to use it IMHO (we mainly use it here to route between different transports ethernet to infiniband for example). - -Herb Jerome, Ron wrote:> The issue is the fact that the luster filesystem is on a private > (192.168.xxx.xxx) network and the client is on a public network, so > bidirectional ip traffic routing gets messy (NATing, port forwarding > etc). I was hoping to avoid this by using a gateway lustre node that is > connected to both networks. > > Ron. > >> -----Original Message----- >> From: lustre-discuss-bounces at clusterfs.com [mailto:lustre-discuss- >> bounces at clusterfs.com] On Behalf Of Herb Wartens >> Sent: January 16, 2008 2:32 PM >> To: Lustre Discuss >> Subject: Re: [Lustre-discuss] How to configure routing >> > > Ron, > I am not exactly familiar with your particular setup, but when using > tcp > is there any reason why you can''t use a single lnet network? I >> believe > that > it would be easier for you to set up one socklnd network named tcp0 > that > contains both interfaces. Then all you have to do is make sure that >> it > is possible > through host routes (or other means) for the two networks to route to > each other. > > -Herb > > Jerome, Ron wrote: >>>> I would like to use a client as a router between two tcp networks > (eth0 >>>> and eth1) but it is unclear to me how to configure this in > modprobe.conf. >>>> >>>> >>>> Any suggestions would be greatly appreciated. >>>> >>>> >>>> >>>> _________________________________________ >>>> Ron Jerome >>>> Programmer/Analyst >>>> National Research Council Canada >>>> M-2, 1200 Montreal Road, Ottawa, Ontario K1A 0R6 >>>> Government of Canada >>>> _________________________________________ >>>> >>>> >>>> >> --------------------------------------------------------------------- > --- >>>> _______________________________________________ >>>> Lustre-discuss mailing list >>>> Lustre-discuss at clusterfs.com >>>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >>_______________________________________________ Lustre-discuss mailing list Lustre-discuss at clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre-discuss -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFHjoApP/62XqEEbMYRCjNaAJ9f17fLOlcJzMb62LjBD/rt4u4MFQCgkWRD H69WcXuJaVymWWJEEc2ufv4=IUFY -----END PGP SIGNATURE-----
Jerome, Ron wrote:> That''s what I was hoping, but I''m still having trouble getting the > routing to work. > > My MGS is at 192.168.0.200 at tcp > > On my gateway node, I have... > > options lnet networks="tcp0(eth0),tcp1(eth1)" routes="tcp1 > 192.168.0.2 at tcp ;\ > tcp0 > 132.246.xxx.xxx at tcp1" > > Then from the node solely connected to the public (132.246) network, I > try to mount > > mount -t lustre 192.168.0.200 at tcp:/data /mnt/data > > but regrettably this does not work :-(All nodes involved need to have routes defined, are the routes defined on the clients and servers also? cliffw> > > Ron. > >> -----Original Message----- >> From: Canon, Richard Shane [mailto:canonrs at ornl.gov] >> Sent: January 16, 2008 10:03 AM >> To: Jerome, Ron >> Cc: He.Huang at Sun.COM; Lustre-discuss at clusterfs.com >> Subject: RE: [Lustre-discuss] How to configure routing >> >> >> Ron, >> >> You don''t have to rely on IP routing. You can use Lustre routing >> instead. You have the basic config line below. The external clients >> need to know about both networks and the routers. You should be able >> to >> use the same LNET module options everywhere. You would do a mount >> like... >> >> mount -t lustre <mds ip>@tcp1:/lustre/client /mnt >> >> Assuming the private network is tcp1 and tcp0 is the public network. >> >> >> --Shane >> >> -----Original Message----- >> From: lustre-discuss-bounces at clusterfs.com >> [mailto:lustre-discuss-bounces at clusterfs.com] On Behalf Of Balagopal >> Pillai >> Sent: Wednesday, January 16, 2008 9:27 AM >> To: Jerome, Ron >> Cc: He.Huang at Sun.COM; Lustre-discuss at clusterfs.com >> Subject: Re: [Lustre-discuss] How to configure routing >> >> Hi, >> >> Interesting scenario if routing between public and private >> subnets for mounting lustre filesystem is possible >> without disturbing the network admins for public subnet :-) One way >> around would be >> to build tunnels like ipip for example between the gateway machine on >> the cluster >> that has access to both nets and with other hosts on the public net. >> Then Lustre mount >> could work over the tunnel with static routes to access the > MGS/MDS/OST >> over the tunnel. >> >> Regards >> Balagopal >> >> Jerome, Ron wrote: >>> Hi Isaac, >>> >>> Yes I had read the documentation prior to posting my question but a >>> couple of things were still unclear to me, but first a bit more >>> background... My lustre filesystem & my cluster reside on a private >>> (192.169.xxx.xxx) network I have a couple of machines which are >>> connected to both the private network and our public network and I >> would >>> like a machine which only has access to the public network to mount >> the >>> lustre filesystem on the private network via one of the nodes which >>> spans both. >>> >>> So my questions are... >>> >>> a) is it possible to route to and from a private (192.168.xxx.xxx) >>> Ethernet network to a public one (132.246.xxx.xxx)? The documents >> only >>> show routing between Ethernet and other networks such as elan, >> Myrinet >>> etc. >>> >>> b) if this is possible, what is the correct syntax for the >> modprobe.conf >>> file? I currently have... >>> >>> options lnet networks="tcp0(eth0),tcp1(eth1)" routes="tcp1 >>> 192.168.0.2 at tcp" >>> >>> c) from a client on the public network what would the syntax be for >>> accessing (mounting) the mgs through the routing node? >>> >>> >>> Thanks very much, >>> >>> Ron. >>> >>> >>>> -----Original Message----- >>>> From: He.Huang at Sun.COM [mailto:He.Huang at Sun.COM] >>>> Sent: January 16, 2008 8:25 AM >>>> To: Jerome, Ron >>>> Cc: Lustre-discuss at clusterfs.com >>>> Subject: Re: [Lustre-discuss] How to configure routing >>>> >>>> On Tue, Jan 15, 2008 at 02:27:39PM -0500, Jerome, Ron wrote: >>>> >>>>> I would like to use a client as a router between two tcp >> networks >>>> (eth0 >>>> >>>>> and eth1) but it is unclear to me how to configure this in >>>>> modprobe.conf. >>>>> >>>>> >>>>> Any suggestions would be greatly appreciated. >>>>> >>>> http://manual.lustre.org/manual/LustreManual16_HTML/DynamicHTML-06- >>>> 1.html >>>> >>>> Please read section 3.2.2. >>>> >>>> Isaac >>>> >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss at clusterfs.com >>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >>> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at clusterfs.com >> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
Ahhh, that might be the problem. I only defined a route on the gateway node. Is it possible to dynamically add a route? Since I''m dealing with a production file system, I can''t really shut it down to play with the routing. Ron. ________________________________ From: Cliff White [mailto:Cliff.White at Sun.COM] Sent: Wed 1/16/2008 6:07 PM To: Jerome, Ron Cc: Lustre-discuss at clusterfs.com Subject: Re: [Lustre-discuss] How to configure routing Jerome, Ron wrote:> That''s what I was hoping, but I''m still having trouble getting the > routing to work. > > My MGS is at 192.168.0.200 at tcp > > On my gateway node, I have... > > options lnet networks="tcp0(eth0),tcp1(eth1)" routes="tcp1 > 192.168.0.2 at tcp ;\ > tcp0 > 132.246.xxx.xxx at tcp1" > > Then from the node solely connected to the public (132.246) network, I > try to mount > > mount -t lustre 192.168.0.200 at tcp:/data /mnt/data > > but regrettably this does not work :-(All nodes involved need to have routes defined, are the routes defined on the clients and servers also? cliffw> > > Ron. > >> -----Original Message----- >> From: Canon, Richard Shane [mailto:canonrs at ornl.gov] >> Sent: January 16, 2008 10:03 AM >> To: Jerome, Ron >> Cc: He.Huang at Sun.COM; Lustre-discuss at clusterfs.com >> Subject: RE: [Lustre-discuss] How to configure routing >> >> >> Ron, >> >> You don''t have to rely on IP routing. You can use Lustre routing >> instead. You have the basic config line below. The external clients >> need to know about both networks and the routers. You should be able >> to >> use the same LNET module options everywhere. You would do a mount >> like... >> >> mount -t lustre <mds ip>@tcp1:/lustre/client /mnt >> >> Assuming the private network is tcp1 and tcp0 is the public network. >> >> >> --Shane >> >> -----Original Message----- >> From: lustre-discuss-bounces at clusterfs.com >> [mailto:lustre-discuss-bounces at clusterfs.com] On Behalf Of Balagopal >> Pillai >> Sent: Wednesday, January 16, 2008 9:27 AM >> To: Jerome, Ron >> Cc: He.Huang at Sun.COM; Lustre-discuss at clusterfs.com >> Subject: Re: [Lustre-discuss] How to configure routing >> >> Hi, >> >> Interesting scenario if routing between public and private >> subnets for mounting lustre filesystem is possible >> without disturbing the network admins for public subnet :-) One way >> around would be >> to build tunnels like ipip for example between the gateway machine on >> the cluster >> that has access to both nets and with other hosts on the public net. >> Then Lustre mount >> could work over the tunnel with static routes to access the > MGS/MDS/OST >> over the tunnel. >> >> Regards >> Balagopal >> >> Jerome, Ron wrote: >>> Hi Isaac, >>> >>> Yes I had read the documentation prior to posting my question but a >>> couple of things were still unclear to me, but first a bit more >>> background... My lustre filesystem & my cluster reside on a private >>> (192.169.xxx.xxx) network I have a couple of machines which are >>> connected to both the private network and our public network and I >> would >>> like a machine which only has access to the public network to mount >> the >>> lustre filesystem on the private network via one of the nodes which >>> spans both. >>> >>> So my questions are... >>> >>> a) is it possible to route to and from a private (192.168.xxx.xxx) >>> Ethernet network to a public one (132.246.xxx.xxx)? The documents >> only >>> show routing between Ethernet and other networks such as elan, >> Myrinet >>> etc. >>> >>> b) if this is possible, what is the correct syntax for the >> modprobe.conf >>> file? I currently have... >>> >>> options lnet networks="tcp0(eth0),tcp1(eth1)" routes="tcp1 >>> 192.168.0.2 at tcp" >>> >>> c) from a client on the public network what would the syntax be for >>> accessing (mounting) the mgs through the routing node? >>> >>> >>> Thanks very much, >>> >>> Ron. >>> >>> >>>> -----Original Message----- >>>> From: He.Huang at Sun.COM [mailto:He.Huang at Sun.COM] >>>> Sent: January 16, 2008 8:25 AM >>>> To: Jerome, Ron >>>> Cc: Lustre-discuss at clusterfs.com >>>> Subject: Re: [Lustre-discuss] How to configure routing >>>> >>>> On Tue, Jan 15, 2008 at 02:27:39PM -0500, Jerome, Ron wrote: >>>> >>>>> I would like to use a client as a router between two tcp >> networks >>>> (eth0 >>>> >>>>> and eth1) but it is unclear to me how to configure this in >>>>> modprobe.conf. >>>>> >>>>> >>>>> Any suggestions would be greatly appreciated. >>>>> >>>> http://manual.lustre.org/manual/LustreManual16_HTML/DynamicHTML-06- >>>> 1.html >>>> >>>> Please read section 3.2.2. >>>> >>>> Isaac >>>> >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss at clusterfs.com >>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >>> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at clusterfs.com >> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080116/fa89c1bf/attachment-0002.html
I hate to disagree with you Herb, but there are a great number of reasons you would not want to set up routing between those networks. First of all, making ''private'' subnets visible to the public network is a no-no. Second, private networks are often secluded for security reasons. Bridging the gap could have many unintended consequences. Lustre should be able to do all the routing with no problem. Servers: LNET tcp0 network Routers: LNET tcp0,tcp1 networks Clients: LNET tcp1 network server (192.168.x) <----> (192.168.x) router (132.246.x) <----> (132.246.x) client Make sure you have ip_forwarding enabled on the router node. Make sure ''options lnet forwarding="enabled"'' is set on the router node. The router will find routes automatically. You just need to specify both networks: options lnet networks=tcp0(eth0),tcp1(eth1) On the client you need: options lnet networks=tcp0(eth0) routes "tcp1 132.246.x at tcp0" (routers 132.246.x IP addr) On the server you need: options lnet networks=tcp1(eth0) routes "tcp0 192.168.x at tcp1" (routers 192.168.x IP addr) Does that make sense? -Marc ---- D. Marc Stearman LC Lustre Systems Administrator marc at llnl.gov 925.423.9670 Pager: 1.888.203.0641 On Jan 16, 2008, at 2:07 PM, Herb Wartens wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > IMHO it would be better for you to set up host routing (or whatever it > is that you wanted) to be ablr to route between these networks. It > would > be best to avoid the routing code if it is possible since you would > be introducing > some new problems by having the routing turned on. > With routing enabled you would not have immediate notification that > a server went down > for example. This causes longer delays on the client when doing > things like failover > since you would have to wait for a timeout to occur before the > client would actually > try the failover server. I think with the adaptive timeouts this > could be resolved, > but I think that things like this would be better to avoid if you > don''t especially > need to use it IMHO (we mainly use it here to route between > different transports > ethernet to infiniband for example). > > - -Herb > > Jerome, Ron wrote: >> The issue is the fact that the luster filesystem is on a private >> (192.168.xxx.xxx) network and the client is on a public network, so >> bidirectional ip traffic routing gets messy (NATing, port forwarding >> etc). I was hoping to avoid this by using a gateway lustre node >> that is >> connected to both networks. >> >> Ron. >> >>> -----Original Message----- >>> From: lustre-discuss-bounces at clusterfs.com [mailto:lustre-discuss- >>> bounces at clusterfs.com] On Behalf Of Herb Wartens >>> Sent: January 16, 2008 2:32 PM >>> To: Lustre Discuss >>> Subject: Re: [Lustre-discuss] How to configure routing >>> >> >> Ron, >> I am not exactly familiar with your particular setup, but when using >> tcp >> is there any reason why you can''t use a single lnet network? I >>> believe >> that >> it would be easier for you to set up one socklnd network named tcp0 >> that >> contains both interfaces. Then all you have to do is make sure that >>> it >> is possible >> through host routes (or other means) for the two networks to route to >> each other. >> >> -Herb >> >> Jerome, Ron wrote: >>>>> I would like to use a client as a router between two tcp networks >> (eth0 >>>>> and eth1) but it is unclear to me how to configure this in >> modprobe.conf. >>>>> >>>>> >>>>> Any suggestions would be greatly appreciated. >>>>> >>>>> >>>>> >>>>> _________________________________________ >>>>> Ron Jerome >>>>> Programmer/Analyst >>>>> National Research Council Canada >>>>> M-2, 1200 Montreal Road, Ottawa, Ontario K1A 0R6 >>>>> Government of Canada >>>>> _________________________________________ >>>>> >>>>> >>>>> >>> -------------------------------------------------------------------- >>> - >> --- >>>>> _______________________________________________ >>>>> Lustre-discuss mailing list >>>>> Lustre-discuss at clusterfs.com >>>>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >>> > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.7 (GNU/Linux) > Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org > > iD8DBQFHjoApP/62XqEEbMYRCjNaAJ9f17fLOlcJzMb62LjBD/rt4u4MFQCgkWRD > H69WcXuJaVymWWJEEc2ufv4> =IUFY > -----END PGP SIGNATURE----- > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
That''s fantastic Marc, Just the information I was looking for. I''ll give it a try. Ron.> -----Original Message----- > From: lustre-discuss-bounces at clusterfs.com [mailto:lustre-discuss- > bounces at clusterfs.com] On Behalf Of D. Marc Stearman > Sent: January 17, 2008 11:31 AM > To: Lustre Discuss > Subject: Re: [Lustre-discuss] How to configure routing > > I hate to disagree with you Herb, but there are a great number of > reasons you would not want to set up routing between those networks. > First of all, making ''private'' subnets visible to the public network > is a no-no. Second, private networks are often secluded for security > reasons. Bridging the gap could have many unintended consequences. > > Lustre should be able to do all the routing with no problem. > > Servers: LNET tcp0 network > Routers: LNET tcp0,tcp1 networks > Clients: LNET tcp1 network > > server (192.168.x) <----> (192.168.x) router (132.246.x) <----> > (132.246.x) client > > Make sure you have ip_forwarding enabled on the router node. > Make sure ''options lnet forwarding="enabled"'' is set on the router > node. > > The router will find routes automatically. You just need to specify > both networks: > options lnet networks=tcp0(eth0),tcp1(eth1) > > On the client you need: > options lnet networks=tcp0(eth0) routes "tcp1 > 132.246.x at tcp0" (routers 132.246.x IP addr) > > On the server you need: > options lnet networks=tcp1(eth0) routes "tcp0 > 192.168.x at tcp1" (routers 192.168.x IP addr) > > Does that make sense? > > -Marc > > ---- > D. Marc Stearman > LC Lustre Systems Administrator > marc at llnl.gov > 925.423.9670 > Pager: 1.888.203.0641 > > > On Jan 16, 2008, at 2:07 PM, Herb Wartens wrote: > > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA512 > > > > IMHO it would be better for you to set up host routing (or whatever > it > > is that you wanted) to be ablr to route between these networks. It > > would > > be best to avoid the routing code if it is possible since you would > > be introducing > > some new problems by having the routing turned on. > > With routing enabled you would not have immediate notification that > > a server went down > > for example. This causes longer delays on the client when doing > > things like failover > > since you would have to wait for a timeout to occur before the > > client would actually > > try the failover server. I think with the adaptive timeouts this > > could be resolved, > > but I think that things like this would be better to avoid if you > > don''t especially > > need to use it IMHO (we mainly use it here to route between > > different transports > > ethernet to infiniband for example). > > > > - -Herb > > > > Jerome, Ron wrote: > >> The issue is the fact that the luster filesystem is on a private > >> (192.168.xxx.xxx) network and the client is on a public network, so > >> bidirectional ip traffic routing gets messy (NATing, portforwarding> >> etc). I was hoping to avoid this by using a gateway lustre node > >> that is > >> connected to both networks. > >> > >> Ron. > >> > >>> -----Original Message----- > >>> From: lustre-discuss-bounces at clusterfs.com [mailto:lustre-discuss- > >>> bounces at clusterfs.com] On Behalf Of Herb Wartens > >>> Sent: January 16, 2008 2:32 PM > >>> To: Lustre Discuss > >>> Subject: Re: [Lustre-discuss] How to configure routing > >>> > >> > >> Ron, > >> I am not exactly familiar with your particular setup, but whenusing> >> tcp > >> is there any reason why you can''t use a single lnet network? I > >>> believe > >> that > >> it would be easier for you to set up one socklnd network named tcp0 > >> that > >> contains both interfaces. Then all you have to do is make surethat> >>> it > >> is possible > >> through host routes (or other means) for the two networks to route > to > >> each other. > >> > >> -Herb > >> > >> Jerome, Ron wrote: > >>>>> I would like to use a client as a router between two tcpnetworks> >> (eth0 > >>>>> and eth1) but it is unclear to me how to configure this in > >> modprobe.conf. > >>>>> > >>>>> > >>>>> Any suggestions would be greatly appreciated. > >>>>> > >>>>> > >>>>> > >>>>> _________________________________________ > >>>>> Ron Jerome > >>>>> Programmer/Analyst > >>>>> National Research Council Canada > >>>>> M-2, 1200 Montreal Road, Ottawa, Ontario K1A 0R6 > >>>>> Government of Canada > >>>>> _________________________________________ > >>>>> > >>>>> > >>>>> > >>>-------------------------------------------------------------------> - > >>> - > >> --- > >>>>> _______________________________________________ > >>>>> Lustre-discuss mailing list > >>>>> Lustre-discuss at clusterfs.com > >>>>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > >>> > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-discuss at clusterfs.com > > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > > > > -----BEGIN PGP SIGNATURE----- > > Version: GnuPG v1.4.7 (GNU/Linux) > > Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org > > > > iD8DBQFHjoApP/62XqEEbMYRCjNaAJ9f17fLOlcJzMb62LjBD/rt4u4MFQCgkWRD > > H69WcXuJaVymWWJEEc2ufv4> > =IUFY > > -----END PGP SIGNATURE----- > > > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-discuss at clusterfs.com > > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 No problem with disagreeing...=) I guess I was just trying to impress upon the group that before enabling routing it would be bet to make sure that is what you really want to do since it can cause its own set of problems (like the one I described with failover)...=) - -Herb D. Marc Stearman wrote:> I hate to disagree with you Herb, but there are a great number of > reasons you would not want to set up routing between those networks. > First of all, making ''private'' subnets visible to the public network > is a no-no. Second, private networks are often secluded for security > reasons. Bridging the gap could have many unintended consequences. > > Lustre should be able to do all the routing with no problem. > > Servers: LNET tcp0 network > Routers: LNET tcp0,tcp1 networks > Clients: LNET tcp1 network > > server (192.168.x) <----> (192.168.x) router (132.246.x) <----> > (132.246.x) client > > Make sure you have ip_forwarding enabled on the router node. > Make sure ''options lnet forwarding="enabled"'' is set on the router node. > > The router will find routes automatically. You just need to specify > both networks: > options lnet networks=tcp0(eth0),tcp1(eth1) > > On the client you need: > options lnet networks=tcp0(eth0) routes "tcp1 > 132.246.x at tcp0" (routers 132.246.x IP addr) > > On the server you need: > options lnet networks=tcp1(eth0) routes "tcp0 > 192.168.x at tcp1" (routers 192.168.x IP addr) > > Does that make sense? > > -Marc > > ---- > D. Marc Stearman > LC Lustre Systems Administrator > marc at llnl.gov > 925.423.9670 > Pager: 1.888.203.0641 > > > On Jan 16, 2008, at 2:07 PM, Herb Wartens wrote: > > IMHO it would be better for you to set up host routing (or whatever it > is that you wanted) to be ablr to route between these networks. It > would > be best to avoid the routing code if it is possible since you would > be introducing > some new problems by having the routing turned on. > With routing enabled you would not have immediate notification that > a server went down > for example. This causes longer delays on the client when doing > things like failover > since you would have to wait for a timeout to occur before the > client would actually > try the failover server. I think with the adaptive timeouts this > could be resolved, > but I think that things like this would be better to avoid if you > don''t especially > need to use it IMHO (we mainly use it here to route between > different transports > ethernet to infiniband for example). > > -Herb > > Jerome, Ron wrote: >>>> The issue is the fact that the luster filesystem is on a private >>>> (192.168.xxx.xxx) network and the client is on a public network, so >>>> bidirectional ip traffic routing gets messy (NATing, port forwarding >>>> etc). I was hoping to avoid this by using a gateway lustre node >>>> that is >>>> connected to both networks. >>>> >>>> Ron. >>>> >>>>> -----Original Message----- >>>>> From: lustre-discuss-bounces at clusterfs.com [mailto:lustre-discuss- >>>>> bounces at clusterfs.com] On Behalf Of Herb Wartens >>>>> Sent: January 16, 2008 2:32 PM >>>>> To: Lustre Discuss >>>>> Subject: Re: [Lustre-discuss] How to configure routing >>>>> >>>> Ron, >>>> I am not exactly familiar with your particular setup, but when using >>>> tcp >>>> is there any reason why you can''t use a single lnet network? I >>>>> believe >>>> that >>>> it would be easier for you to set up one socklnd network named tcp0 >>>> that >>>> contains both interfaces. Then all you have to do is make sure that >>>>> it >>>> is possible >>>> through host routes (or other means) for the two networks to route to >>>> each other. >>>> >>>> -Herb >>>> >>>> Jerome, Ron wrote: >>>>>>> I would like to use a client as a router between two tcp networks >>>> (eth0 >>>>>>> and eth1) but it is unclear to me how to configure this in >>>> modprobe.conf. >>>>>>> >>>>>>> Any suggestions would be greatly appreciated. >>>>>>> >>>>>>> >>>>>>> >>>>>>> _________________________________________ >>>>>>> Ron Jerome >>>>>>> Programmer/Analyst >>>>>>> National Research Council Canada >>>>>>> M-2, 1200 Montreal Road, Ottawa, Ontario K1A 0R6 >>>>>>> Government of Canada >>>>>>> _________________________________________ >>>>>>> >>>>>>> >>>>>>> >>>>> -------------------------------------------------------------------- >>>>> - >>>> --- >>>>>>> _______________________________________________ >>>>>>> Lustre-discuss mailing list >>>>>>> Lustre-discuss at clusterfs.com >>>>>>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > >>_______________________________________________ Lustre-discuss mailing list Lustre-discuss at clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre-discuss> _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFHj57UP/62XqEEbMYRClQAAKCRkRZRU+wEGsLpU51I4mAZDCGEuACg2ZAy gFcrhk+SiXob4xsWMdlBeQs=yLUj -----END PGP SIGNATURE-----
On Thu, Jan 17, 2008 at 08:30:47AM -0800, D. Marc Stearman wrote:> ...... > Lustre should be able to do all the routing with no problem. > > Servers: LNET tcp0 network > Routers: LNET tcp0,tcp1 networks > Clients: LNET tcp1 network > > server (192.168.x) <----> (192.168.x) router (132.246.x) <----> > (132.246.x) client > > Make sure you have ip_forwarding enabled on the router node. > Make sure ''options lnet forwarding="enabled"'' is set on the router node. >AFAIK, lnet routing doesn''t depend on ip_forwarding to work over two tcp networks. BTW, you don''t have to explicitly enable lnet forwarding on router nodes. When a node notices that it''s used as a router in one of the routes, it will enable forwarding by itself. This gives us a minor administration convenience because the same routing parameters can be used over a whole cluster. In this case, it shall suffice to just specify routes on all nodes as: routes="tcp1 132.246.x at tcp0"; tcp0 192.168.x at tcp1" Hope this helps, Isaac