Lundgren, Andrew
2008-Jan-15 17:28 UTC
[Lustre-discuss] How do you make an MGS/OSS listen on 2 NICs?
I am running on CentOS 5 distribution without adding any updates from CentOS. I am using the lustre 1.6.4.1 kernel and software. I have two NICs that run though different switches. I have the lustre options in my modprobe.conf to look like this: options lnet networks=tcp0(eth1,eth0) My MGS seems to be only listening on the first interface however. When I try and ping the 1st interface (eth1) , it works when I go for the second (eth0) it does not. # lctl ping 192.168.135.81 at tcp<mailto:192.168.135.81 at tcp> 12345-0 at lo<mailto:12345-0 at lo> 12345-192.168.135.81 at tcp<mailto:12345-192.168.135.81 at tcp> # lctl ping 192.168.135.80 at tcp<mailto:192.168.135.80 at tcp> failed to ping 192.168.135.80 at tcp<mailto:192.168.135.80 at tcp>: Input/output error The following is in /var/log/messages Jan 15 17:18:15 dint0001 kernel: LustreError: 120-3: Refusing connection from 192.168.135.80 for 192.168.135.80 at tcp<mailto:192.168.135.80 at tcp>: No matching NI Jan 15 17:18:15 dint0001 kernel: LustreError: 3251:0:(socklnd_cb.c:2167:ksocknal_recv_hello()) Error -104 reading HELLO from 192.168.135.80 Jan 15 17:18:15 dint0001 kernel: LustreError: 11b-b: Connection to 192.168.135.80 at tcp<mailto:192.168.135.80 at tcp> at host 192.168.135.80 on port 988 was reset: is it running a compatible version of Lustre and is 192.168.135.80 at tcp<mailto:192.168.135.80 at tcp> one of its NIDs? Where/How do I configure it to listen on both devices? Thank you! -- Andrew Lundgren -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080115/fbbc6ef6/attachment-0002.html
Klaus Steden
2008-Jan-15 18:49 UTC
[Lustre-discuss] How do you make an MGS/OSS listen on 2 NICs?
Try putting quotes around the argument to the ?networks=? statement. If you?ve got only eth0 and eth1 in your system, you don?t need to specify them explicitly, either, Lustre will automatically bind all available interfaces. Here?s what my config looks like: -- cut -- [root at mds-0-0 ~]# lctl list_nids 172.16.129.252 at tcp 172.16.128.252 at tcp1 root at mds-0-0 ~]# lctl ping 172.16.128.250 at tcp1 12345-0 at lo 12345-172.16.129.250 at tcp 12345-172.16.128.250 at tcp1 [root at mds-0-0 ~]# lctl ping 172.16.128.250 at tcp0 ^C [root at mds-0-0 ~]# lctl ping 172.16.129.250 at tcp0 12345-0 at lo 12345-172.16.129.250 at tcp 12345-172.16.128.250 at tcp1 [root at mds-0-0 ~]# lctl ping 172.16.129.250 at tcp1 ^C [root at mds-0-0 ~]# grep lnet /etc/modprobe.conf options lnet networks="tcp0(eth0),tcp1(bond0)" -- cut -- hth, Klaus On 1/15/08 9:28 AM, "Lundgren, Andrew" <Andrew.Lundgren at Level3.com>did etch on stone tablets:> I am running on CentOS 5 distribution without adding any updates from CentOS. > I am using the lustre 1.6.4.1 kernel and software. > > I have two NICs that run though different switches. > > I have the lustre options in my modprobe.conf to look like this: > > options lnet networks=tcp0(eth1,eth0) > > My MGS seems to be only listening on the first interface however. > > When I try and ping the 1st interface (eth1) , it works when I go for the > second (eth0) it does not. > > # lctl ping 192.168.135.81 at tcp > 12345-0 at lo > 12345-192.168.135.81 at tcp > # lctl ping 192.168.135.80 at tcp > failed to ping 192.168.135.80 at tcp: Input/output error > > The following is in /var/log/messages > > Jan 15 17:18:15 dint0001 kernel: LustreError: 120-3: Refusing connection from > 192.168.135.80 for 192.168.135.80 at tcp <mailto:192.168.135.80 at tcp> : No > matching NI > Jan 15 17:18:15 dint0001 kernel: LustreError: > 3251:0:(socklnd_cb.c:2167:ksocknal_recv_hello()) Error -104 reading HELLO from > 192.168.135.80 > Jan 15 17:18:15 dint0001 kernel: LustreError: 11b-b: Connection to > 192.168.135.80 at tcp <mailto:192.168.135.80 at tcp> at host 192.168.135.80 on port > 988 was reset: is it running a compatible version of Lustre and is > 192.168.135.80 at tcp <mailto:192.168.135.80 at tcp> one of its NIDs? > > Where/How do I configure it to listen on both devices? > > Thank you! > > -- > Andrew Lundgren > > > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080115/fe5f97ea/attachment-0002.html
Lundgren, Andrew
2008-Jan-15 20:35 UTC
[Lustre-discuss] How do you make an MGS/OSS listen on 2 NICs?
We actually set lustre up to run on some non-routable aliased IP addresses, so we wanted to limit it two two specific NIC cards. I modified my line to look like yours, where I specify tcp1 for the second interface and now it is pingable there, but I am not positive it is correct. When I didn''t have any lnet options, it just grabbed the 1st NIC it saw and ignored the rest. -- Andrew ________________________________ From: Klaus Steden [mailto:klaus.steden at thomson.net] Sent: Tuesday, January 15, 2008 11:50 AM To: Lundgren, Andrew; ''Lustre-discuss at clusterfs.com'' Subject: Re: [Lustre-discuss] How do you make an MGS/OSS listen on 2 NICs? Try putting quotes around the argument to the ''networks='' statement. If you''ve got only eth0 and eth1 in your system, you don''t need to specify them explicitly, either, Lustre will automatically bind all available interfaces. Here''s what my config looks like: -- cut -- [root at mds-0-0 ~]# lctl list_nids 172.16.129.252 at tcp 172.16.128.252 at tcp1 root at mds-0-0 ~]# lctl ping 172.16.128.250 at tcp1 12345-0 at lo 12345-172.16.129.250 at tcp 12345-172.16.128.250 at tcp1 [root at mds-0-0 ~]# lctl ping 172.16.128.250 at tcp0 ^C [root at mds-0-0 ~]# lctl ping 172.16.129.250 at tcp0 12345-0 at lo 12345-172.16.129.250 at tcp 12345-172.16.128.250 at tcp1 [root at mds-0-0 ~]# lctl ping 172.16.129.250 at tcp1 ^C [root at mds-0-0 ~]# grep lnet /etc/modprobe.conf options lnet networks="tcp0(eth0),tcp1(bond0)" -- cut -- hth, Klaus On 1/15/08 9:28 AM, "Lundgren, Andrew" <Andrew.Lundgren at Level3.com>did etch on stone tablets: I am running on CentOS 5 distribution without adding any updates from CentOS. I am using the lustre 1.6.4.1 kernel and software. I have two NICs that run though different switches. I have the lustre options in my modprobe.conf to look like this: options lnet networks=tcp0(eth1,eth0) My MGS seems to be only listening on the first interface however. When I try and ping the 1st interface (eth1) , it works when I go for the second (eth0) it does not. # lctl ping 192.168.135.81 at tcp 12345-0 at lo 12345-192.168.135.81 at tcp # lctl ping 192.168.135.80 at tcp failed to ping 192.168.135.80 at tcp: Input/output error The following is in /var/log/messages Jan 15 17:18:15 dint0001 kernel: LustreError: 120-3: Refusing connection from 192.168.135.80 for 192.168.135.80 at tcp <mailto:192.168.135.80 at tcp><mailto:192.168.135.80 at tcp> : No matching NI Jan 15 17:18:15 dint0001 kernel: LustreError: 3251:0:(socklnd_cb.c:2167:ksocknal_recv_hello()) Error -104 reading HELLO from 192.168.135.80 Jan 15 17:18:15 dint0001 kernel: LustreError: 11b-b: Connection to 192.168.135.80 at tcp <mailto:192.168.135.80 at tcp><mailto:192.168.135.80 at tcp> at host 192.168.135.80 on port 988 was reset: is it running a compatible version of Lustre and is 192.168.135.80 at tcp <mailto:192.168.135.80 at tcp><mailto:192.168.135.80 at tcp> one of its NIDs? Where/How do I configure it to listen on both devices? Thank you! -- Andrew Lundgren ________________________________ _______________________________________________ Lustre-discuss mailing list Lustre-discuss at clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080115/02a7ce57/attachment-0002.html
Isaac Huang
2008-Jan-16 13:34 UTC
[Lustre-discuss] How do you make an MGS/OSS listen on 2 NICs?
On Tue, Jan 15, 2008 at 10:28:33AM -0700, Lundgren, Andrew wrote:> I am running on CentOS 5 distribution without adding any updates from > CentOS. I am using the lustre 1.6.4.1 kernel and software. > > > > I have two NICs that run though different switches. > > > > I have the lustre options in my modprobe.conf to look like this: > > > > options lnet networks=tcp0(eth1,eth0) >This way of interface bonding is now a deprecated lnet feature. Please refer to: http://manual.lustre.org/manual/LustreManual16_HTML/DynamicHTML-13-1.html Isaac
Lundgren, Andrew
2008-Jan-16 16:03 UTC
[Lustre-discuss] How do you make an MGS/OSS listen on 2 NICs?
So the only way to use two nics at once is to bond? I am more for redundancy rather than increased throughput.> -----Original Message----- > From: He.Huang at Sun.COM [mailto:He.Huang at Sun.COM] > Sent: Wednesday, January 16, 2008 6:34 AM > To: Lundgren, Andrew > Cc: ''Lustre-discuss at clusterfs.com'' > Subject: Re: [Lustre-discuss] How do you make an MGS/OSS > listen on 2 NICs? > > On Tue, Jan 15, 2008 at 10:28:33AM -0700, Lundgren, Andrew wrote: > > I am running on CentOS 5 distribution without adding any > updates from > > CentOS. I am using the lustre 1.6.4.1 kernel and software. > > > > > > > > I have two NICs that run though different switches. > > > > > > > > I have the lustre options in my modprobe.conf to look like this: > > > > > > > > options lnet networks=tcp0(eth1,eth0) > > > > This way of interface bonding is now a deprecated lnet > feature. Please refer to: > http://manual.lustre.org/manual/LustreManual16_HTML/DynamicHTM > L-13-1.html > > Isaac >
Herb Wartens
2008-Jan-16 19:23 UTC
[Lustre-discuss] How do you make an MGS/OSS listen on 2 NICs?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Andrew, I have not used lustre-1.6.4.X yet, but in previous versions (and most likely the version you are using) Lustre actually listens on all interfaces no matter what you specify in the modprobe.conf. You can verify this by looking at the netstat output for port 988 and look for what ports you are listening on. We here at LLNL regularly use multiple interfaces. I believe that the issue you are referring to is a bug in the lctl ping code where the ping only responds over the first network device specified for a particular lnd. As long as you have properly configured your host routes so that you can ping both interfaces from the other node you should be fine. IMHO this should just be fixed in lnet so you can do an lctl ping from any endpoint to any other endpoint. # ilc6 /root > cat /etc/modprobe.conf options lnet networks="tcp0(eth2,eth3)" # ilc6 /root > netstat -a -t -n | grep 988 | grep LIST tcp 0 0 0.0.0.0:988 0.0.0.0:* LISTEN # ilc6 /root > cat /etc/hosts | grep ilc7 172.16.101.7 ilc7-lnet0 ilc7-eth2 172.16.102.7 ilc7-lnet1 ilc7-eth3 # ilc6 /root > lctl ping 172.16.101.7 at tcp0 12345-0 at lo 12345-172.16.101.7 at tcp # ilc6 /root > lctl ping 172.16.102.7 at tcp0 failed to ping 172.16.102.7 at tcp: Input/output error # ilc6 /root > ping -c 1 172.16.101.7 PING 172.16.101.7 (172.16.101.7) 56(84) bytes of data. 64 bytes from 172.16.101.7: icmp_seq=1 ttl=64 time=0.143 ms - --- 172.16.101.7 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.143/0.143/0.143/0.000 ms # ilc6 /root > ping -c 1 172.16.102.7 PING 172.16.102.7 (172.16.102.7) 56(84) bytes of data. 64 bytes from 172.16.102.7: icmp_seq=1 ttl=64 time=0.094 ms - --- 172.16.102.7 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.094/0.094/0.094/0.000 ms Lundgren, Andrew wrote:> So the only way to use two nics at once is to bond? I am more for redundancy rather than increased throughput. > >> -----Original Message----- >> From: He.Huang at Sun.COM [mailto:He.Huang at Sun.COM] >> Sent: Wednesday, January 16, 2008 6:34 AM >> To: Lundgren, Andrew >> Cc: ''Lustre-discuss at clusterfs.com'' >> Subject: Re: [Lustre-discuss] How do you make an MGS/OSS >> listen on 2 NICs? >> >> On Tue, Jan 15, 2008 at 10:28:33AM -0700, Lundgren, Andrew wrote: >>> I am running on CentOS 5 distribution without adding any >> updates from >>> CentOS. I am using the lustre 1.6.4.1 kernel and software. >>> >>> >>> >>> I have two NICs that run though different switches. >>> >>> >>> >>> I have the lustre options in my modprobe.conf to look like this: >>> >>> >>> >>> options lnet networks=tcp0(eth1,eth0) >>> >> This way of interface bonding is now a deprecated lnet >> feature. Please refer to: >> http://manual.lustre.org/manual/LustreManual16_HTML/DynamicHTM >> L-13-1.html >> >> Isaac >> > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFHjlmyP/62XqEEbMYRCvegAKCK3z6aFIVtGe/O5ruqStf6/tZoLQCcD1L8 lfEg/WwNivOlMxHDdnWpgcA=0xbK -----END PGP SIGNATURE-----
Cliff White
2008-Jan-16 23:11 UTC
[Lustre-discuss] How do you make an MGS/OSS listen on 2 NICs?
Lundgren, Andrew wrote:> So the only way to use two nics at once is to bond? I am more for redundancy rather than increased throughput.Yes. When we first did this with Lustre bonding was not as good, bonding works fine now so we did not see the need to continue to duplicate function. cliffw> >> -----Original Message----- >> From: He.Huang at Sun.COM [mailto:He.Huang at Sun.COM] >> Sent: Wednesday, January 16, 2008 6:34 AM >> To: Lundgren, Andrew >> Cc: ''Lustre-discuss at clusterfs.com'' >> Subject: Re: [Lustre-discuss] How do you make an MGS/OSS >> listen on 2 NICs? >> >> On Tue, Jan 15, 2008 at 10:28:33AM -0700, Lundgren, Andrew wrote: >>> I am running on CentOS 5 distribution without adding any >> updates from >>> CentOS. I am using the lustre 1.6.4.1 kernel and software. >>> >>> >>> >>> I have two NICs that run though different switches. >>> >>> >>> >>> I have the lustre options in my modprobe.conf to look like this: >>> >>> >>> >>> options lnet networks=tcp0(eth1,eth0) >>> >> This way of interface bonding is now a deprecated lnet >> feature. Please refer to: >> http://manual.lustre.org/manual/LustreManual16_HTML/DynamicHTM >> L-13-1.html >> >> Isaac >> > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
Isaac Huang
2008-Jan-17 04:01 UTC
[Lustre-discuss] How do you make an MGS/OSS listen on 2 NICs?
On Wed, Jan 16, 2008 at 11:23:30AM -0800, Herb Wartens wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > Andrew, > I have not used lustre-1.6.4.X yet, but in previous versions (and most likely the version you are using) > Lustre actually listens on all interfaces no matter what you specify in the modprobe.conf. You can verify this > by looking at the netstat output for port 988 and look for what ports you are listening on. We here at LLNL > regularly use multiple interfaces. > I believe that the issue you are referring to is a bug in the lctl ping code where the ping only responds > over the first network device specified for a particular lnd. As long as you have properly configured your > host routes so that you can ping both interfaces from the other node you should be fine. IMHO this should > just be fixed in lnet so you can do an lctl ping from any endpoint to any other endpoint.I don''t think it''s a lctl ping bug.> > # ilc6 /root > cat /etc/modprobe.conf > options lnet networks="tcp0(eth2,eth3)"This config gives the node only one NID: ip_of_eth2 at tcp0. You can verify it by ''lctl list_nids'' on the node.> > # ilc6 /root > netstat -a -t -n | grep 988 | grep LIST > tcp 0 0 0.0.0.0:988 0.0.0.0:* LISTEN > > # ilc6 /root > cat /etc/hosts | grep ilc7 > 172.16.101.7 ilc7-lnet0 ilc7-eth2 > 172.16.102.7 ilc7-lnet1 ilc7-eth3 > > # ilc6 /root > lctl ping 172.16.101.7 at tcp0 > 12345-0 at lo > 12345-172.16.101.7 at tcpWhen you lctl ping a node at any one of its NIDs, the ping reply contains a list of all NIDs of the node. As can be seen from the reply above, 172.16.101.7 at tcp0 has two NIDs: 0 at lo and 172.16.101.7 at tcp. So when you tried ''lctl ping 172.16.102.7 at tcp0'', the ping request could reach 172.16.102.7, but it was rejected since 172.16.102.7 at tcp0 was not one of the node''s NIDs. The socklnd does interface bonding transparently from lnet''s perspective. It exchanges a list of IPs of all NICs under a lnet NID with peers, and creates connections to all IPs of a peer and thus aggregates bandwidth. Lnet has no knowledge of this - all it sees is just one NID, i.e. ip_of_1st_nic at tcp. Isaac> # ilc6 /root > lctl ping 172.16.102.7 at tcp0 > failed to ping 172.16.102.7 at tcp: Input/output error > > # ilc6 /root > ping -c 1 172.16.101.7 > PING 172.16.101.7 (172.16.101.7) 56(84) bytes of data. > 64 bytes from 172.16.101.7: icmp_seq=1 ttl=64 time=0.143 ms > > - --- 172.16.101.7 ping statistics --- > 1 packets transmitted, 1 received, 0% packet loss, time 0ms > rtt min/avg/max/mdev = 0.143/0.143/0.143/0.000 ms > # ilc6 /root > ping -c 1 172.16.102.7 > PING 172.16.102.7 (172.16.102.7) 56(84) bytes of data. > 64 bytes from 172.16.102.7: icmp_seq=1 ttl=64 time=0.094 ms > > - --- 172.16.102.7 ping statistics --- > 1 packets transmitted, 1 received, 0% packet loss, time 0ms > rtt min/avg/max/mdev = 0.094/0.094/0.094/0.000 ms > > > Lundgren, Andrew wrote: > > So the only way to use two nics at once is to bond? I am more for redundancy rather than increased throughput. > > > >> -----Original Message----- > >> From: He.Huang at Sun.COM [mailto:He.Huang at Sun.COM] > >> Sent: Wednesday, January 16, 2008 6:34 AM > >> To: Lundgren, Andrew > >> Cc: ''Lustre-discuss at clusterfs.com'' > >> Subject: Re: [Lustre-discuss] How do you make an MGS/OSS > >> listen on 2 NICs? > >> > >> On Tue, Jan 15, 2008 at 10:28:33AM -0700, Lundgren, Andrew wrote: > >>> I am running on CentOS 5 distribution without adding any > >> updates from > >>> CentOS. I am using the lustre 1.6.4.1 kernel and software. > >>> > >>> > >>> > >>> I have two NICs that run though different switches. > >>> > >>> > >>> > >>> I have the lustre options in my modprobe.conf to look like this: > >>> > >>> > >>> > >>> options lnet networks=tcp0(eth1,eth0) > >>> > >> This way of interface bonding is now a deprecated lnet > >> feature. Please refer to: > >> http://manual.lustre.org/manual/LustreManual16_HTML/DynamicHTM > >> L-13-1.html > >> > >> Isaac > >> > > > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-discuss at clusterfs.com > > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.7 (GNU/Linux) > Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org > > iD8DBQFHjlmyP/62XqEEbMYRCvegAKCK3z6aFIVtGe/O5ruqStf6/tZoLQCcD1L8 > lfEg/WwNivOlMxHDdnWpgcA> =0xbK > -----END PGP SIGNATURE----- > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
D. Marc Stearman
2008-Jan-17 16:19 UTC
[Lustre-discuss] How do you make an MGS/OSS listen on 2 NICs?
If his NICs are connected to two different switches, bonding will not work. If that is incorrect, please enlighten me. -Marc ---- D. Marc Stearman LC Lustre Systems Administrator marc at llnl.gov 925.423.9670 Pager: 1.888.203.0641 On Jan 16, 2008, at 5:34 AM, Isaac Huang wrote:> On Tue, Jan 15, 2008 at 10:28:33AM -0700, Lundgren, Andrew wrote: >> I am running on CentOS 5 distribution without adding any >> updates from >> CentOS. I am using the lustre 1.6.4.1 kernel and software. >> >> >> >> I have two NICs that run though different switches. >> >> >> >> I have the lustre options in my modprobe.conf to look like this: >> >> >> >> options lnet networks=tcp0(eth1,eth0) >> > > This way of interface bonding is now a deprecated lnet feature. Please > refer to: > http://manual.lustre.org/manual/LustreManual16_HTML/ > DynamicHTML-13-1.html > > Isaac > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
Lundgren, Andrew
2008-Jan-17 17:06 UTC
[Lustre-discuss] How do you make an MGS/OSS listen on 2 NICs?
My NICs are connected to different switches... I am still pondering this one.> -----Original Message----- > From: lustre-discuss-bounces at clusterfs.com > [mailto:lustre-discuss-bounces at clusterfs.com] On Behalf Of D. > Marc Stearman > Sent: Thursday, January 17, 2008 9:19 AM > To: Lustre-discuss at clusterfs.com > Subject: Re: [Lustre-discuss] How do you make an MGS/OSS > listen on 2 NICs? > > If his NICs are connected to two different switches, bonding > will not work. If that is incorrect, please enlighten me. > > -Marc > > ---- > D. Marc Stearman > LC Lustre Systems Administrator > marc at llnl.gov > 925.423.9670 > Pager: 1.888.203.0641 > > > On Jan 16, 2008, at 5:34 AM, Isaac Huang wrote: > > > On Tue, Jan 15, 2008 at 10:28:33AM -0700, Lundgren, Andrew wrote: > >> I am running on CentOS 5 distribution without adding > any updates > >> from > >> CentOS. I am using the lustre 1.6.4.1 kernel and software. > >> > >> > >> > >> I have two NICs that run though different switches. > >> > >> > >> > >> I have the lustre options in my modprobe.conf to look like this: > >> > >> > >> > >> options lnet networks=tcp0(eth1,eth0) > >> > > > > This way of interface bonding is now a deprecated lnet > feature. Please > > refer to: > > http://manual.lustre.org/manual/LustreManual16_HTML/ > > DynamicHTML-13-1.html > > > > Isaac > > > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-discuss at clusterfs.com > > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >
D. Marc Stearman
2008-Jan-17 17:21 UTC
[Lustre-discuss] How do you make an MGS/OSS listen on 2 NICs?
Isaac is correct, that you will only have one NID, and it will be the only one pingable, however you should be able to use both interfaces with your configuration. First of all, can you ping the IP addr (not using LNET pings, just ordinary tcp/ip ping)? As you state, your server has two NICS on two different switches. Where is the client connected? Does it have two NICS, and are they on the same switches? Same subnets? A bit more information about how your network is laid out would be useful. -Marc ---- D. Marc Stearman LC Lustre Systems Administrator marc at llnl.gov 925.423.9670 Pager: 1.888.203.0641 On Jan 17, 2008, at 9:06 AM, Lundgren, Andrew wrote:> My NICs are connected to different switches... I am still > pondering this one. > >> -----Original Message----- >> From: lustre-discuss-bounces at clusterfs.com >> [mailto:lustre-discuss-bounces at clusterfs.com] On Behalf Of D. >> Marc Stearman >> Sent: Thursday, January 17, 2008 9:19 AM >> To: Lustre-discuss at clusterfs.com >> Subject: Re: [Lustre-discuss] How do you make an MGS/OSS >> listen on 2 NICs? >> >> If his NICs are connected to two different switches, bonding >> will not work. If that is incorrect, please enlighten me. >> >> -Marc >> >> ---- >> D. Marc Stearman >> LC Lustre Systems Administrator >> marc at llnl.gov >> 925.423.9670 >> Pager: 1.888.203.0641 >> >> >> On Jan 16, 2008, at 5:34 AM, Isaac Huang wrote: >> >>> On Tue, Jan 15, 2008 at 10:28:33AM -0700, Lundgren, Andrew wrote: >>>> I am running on CentOS 5 distribution without adding >> any updates >>>> from >>>> CentOS. I am using the lustre 1.6.4.1 kernel and software. >>>> >>>> >>>> >>>> I have two NICs that run though different switches. >>>> >>>> >>>> >>>> I have the lustre options in my modprobe.conf to look like this: >>>> >>>> >>>> >>>> options lnet networks=tcp0(eth1,eth0) >>>> >>> >>> This way of interface bonding is now a deprecated lnet >> feature. Please >>> refer to: >>> http://manual.lustre.org/manual/LustreManual16_HTML/ >>> DynamicHTML-13-1.html >>> >>> Isaac >>> >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss at clusterfs.com >>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at clusterfs.com >> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >> >
Lundgren, Andrew
2008-Jan-17 18:32 UTC
[Lustre-discuss] How do you make an MGS/OSS listen on 2 NICs?
The clients and servers both have two nics connected to different switches using different subnets. The clients and servers both have their eth0 devices connected to the same switch on one subnet and their eth1 on a second switch using a second subnet. They can talk to each other over the same switch or via routing between the switches. client eth0 <-same switch-> server eth0 client eth1 <-same switch-> server eth1 client eth0 <-via routing between switches-> server eth1 client eth1 <-via routing between switches-> server eth0 -- Andrew> -----Original Message----- > From: lustre-discuss-bounces at clusterfs.com > [mailto:lustre-discuss-bounces at clusterfs.com] On Behalf Of D. > Marc Stearman > Sent: Thursday, January 17, 2008 10:21 AM > To: Lustre Discuss > Subject: Re: [Lustre-discuss] How do you make an MGS/OSS > listen on 2 NICs? > > Isaac is correct, that you will only have one NID, and it > will be the only one pingable, however you should be able to > use both interfaces with your configuration. First of all, > can you ping the IP addr (not using LNET pings, just ordinary > tcp/ip ping)? > > As you state, your server has two NICS on two different switches. > Where is the client connected? Does it have two NICS, and > are they on the same switches? Same subnets? A bit more > information about how your network is laid out would be useful. > > -Marc > > ---- > D. Marc Stearman > LC Lustre Systems Administrator > marc at llnl.gov > 925.423.9670 > Pager: 1.888.203.0641 > > > On Jan 17, 2008, at 9:06 AM, Lundgren, Andrew wrote: > > > My NICs are connected to different switches... I am still > pondering > > this one. > > > >> -----Original Message----- > >> From: lustre-discuss-bounces at clusterfs.com > >> [mailto:lustre-discuss-bounces at clusterfs.com] On Behalf Of D. > >> Marc Stearman > >> Sent: Thursday, January 17, 2008 9:19 AM > >> To: Lustre-discuss at clusterfs.com > >> Subject: Re: [Lustre-discuss] How do you make an MGS/OSS > listen on 2 > >> NICs? > >> > >> If his NICs are connected to two different switches, > bonding will not > >> work. If that is incorrect, please enlighten me. > >> > >> -Marc > >> > >> ---- > >> D. Marc Stearman > >> LC Lustre Systems Administrator > >> marc at llnl.gov > >> 925.423.9670 > >> Pager: 1.888.203.0641 > >> > >> > >> On Jan 16, 2008, at 5:34 AM, Isaac Huang wrote: > >> > >>> On Tue, Jan 15, 2008 at 10:28:33AM -0700, Lundgren, Andrew wrote: > >>>> I am running on CentOS 5 distribution without adding > >> any updates > >>>> from > >>>> CentOS. I am using the lustre 1.6.4.1 kernel and software. > >>>> > >>>> > >>>> > >>>> I have two NICs that run though different switches. > >>>> > >>>> > >>>> > >>>> I have the lustre options in my modprobe.conf to look > like this: > >>>> > >>>> > >>>> > >>>> options lnet networks=tcp0(eth1,eth0) > >>>> > >>> > >>> This way of interface bonding is now a deprecated lnet > >> feature. Please > >>> refer to: > >>> http://manual.lustre.org/manual/LustreManual16_HTML/ > >>> DynamicHTML-13-1.html > >>> > >>> Isaac > >>> > >>> _______________________________________________ > >>> Lustre-discuss mailing list > >>> Lustre-discuss at clusterfs.com > >>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > >> > >> _______________________________________________ > >> Lustre-discuss mailing list > >> Lustre-discuss at clusterfs.com > >> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > >> > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >
D. Marc Stearman
2008-Jan-17 21:40 UTC
[Lustre-discuss] How do you make an MGS/OSS listen on 2 NICs?
That sounds like a proper setup. It''s very similar to what we have here. I believe the socklnd will make connections using both NICs on both client and server. I saw in another message that you had iptables on. Turning that off or allowing the port Andreas suggested should fix your issue. Are you still seeing connection problems with the second NIC? -Marc ---- D. Marc Stearman LC Lustre Systems Administrator marc at llnl.gov 925.423.9670 Pager: 1.888.203.0641 On Jan 17, 2008, at 10:32 AM, Lundgren, Andrew wrote:> The clients and servers both have two nics connected to different > switches using different subnets. The clients and servers both > have their eth0 devices connected to the same switch on one subnet > and their eth1 on a second switch using a second subnet. > > They can talk to each other over the same switch or via routing > between the switches. > > client eth0 <-same switch-> server eth0 > client eth1 <-same switch-> server eth1 > client eth0 <-via routing between switches-> server eth1 > client eth1 <-via routing between switches-> server eth0 > > -- > Andrew > >> -----Original Message----- >> From: lustre-discuss-bounces at clusterfs.com >> [mailto:lustre-discuss-bounces at clusterfs.com] On Behalf Of D. >> Marc Stearman >> Sent: Thursday, January 17, 2008 10:21 AM >> To: Lustre Discuss >> Subject: Re: [Lustre-discuss] How do you make an MGS/OSS >> listen on 2 NICs? >> >> Isaac is correct, that you will only have one NID, and it >> will be the only one pingable, however you should be able to >> use both interfaces with your configuration. First of all, >> can you ping the IP addr (not using LNET pings, just ordinary >> tcp/ip ping)? >> >> As you state, your server has two NICS on two different switches. >> Where is the client connected? Does it have two NICS, and >> are they on the same switches? Same subnets? A bit more >> information about how your network is laid out would be useful. >> >> -Marc >> >> ---- >> D. Marc Stearman >> LC Lustre Systems Administrator >> marc at llnl.gov >> 925.423.9670 >> Pager: 1.888.203.0641 >> >> >> On Jan 17, 2008, at 9:06 AM, Lundgren, Andrew wrote: >> >>> My NICs are connected to different switches... I am still >> pondering >>> this one. >>> >>>> -----Original Message----- >>>> From: lustre-discuss-bounces at clusterfs.com >>>> [mailto:lustre-discuss-bounces at clusterfs.com] On Behalf Of D. >>>> Marc Stearman >>>> Sent: Thursday, January 17, 2008 9:19 AM >>>> To: Lustre-discuss at clusterfs.com >>>> Subject: Re: [Lustre-discuss] How do you make an MGS/OSS >> listen on 2 >>>> NICs? >>>> >>>> If his NICs are connected to two different switches, >> bonding will not >>>> work. If that is incorrect, please enlighten me. >>>> >>>> -Marc >>>> >>>> ---- >>>> D. Marc Stearman >>>> LC Lustre Systems Administrator >>>> marc at llnl.gov >>>> 925.423.9670 >>>> Pager: 1.888.203.0641 >>>> >>>> >>>> On Jan 16, 2008, at 5:34 AM, Isaac Huang wrote: >>>> >>>>> On Tue, Jan 15, 2008 at 10:28:33AM -0700, Lundgren, Andrew wrote: >>>>>> I am running on CentOS 5 distribution without adding >>>> any updates >>>>>> from >>>>>> CentOS. I am using the lustre 1.6.4.1 kernel and software. >>>>>> >>>>>> >>>>>> >>>>>> I have two NICs that run though different switches. >>>>>> >>>>>> >>>>>> >>>>>> I have the lustre options in my modprobe.conf to look >> like this: >>>>>> >>>>>> >>>>>> >>>>>> options lnet networks=tcp0(eth1,eth0) >>>>>> >>>>> >>>>> This way of interface bonding is now a deprecated lnet >>>> feature. Please >>>>> refer to: >>>>> http://manual.lustre.org/manual/LustreManual16_HTML/ >>>>> DynamicHTML-13-1.html >>>>> >>>>> Isaac >>>>> >>>>> _______________________________________________ >>>>> Lustre-discuss mailing list >>>>> Lustre-discuss at clusterfs.com >>>>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >>>> >>>> _______________________________________________ >>>> Lustre-discuss mailing list >>>> Lustre-discuss at clusterfs.com >>>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >>>> >>> >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at clusterfs.com >> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >> >
Herb Wartens
2008-Jan-17 21:59 UTC
[Lustre-discuss] How do you make an MGS/OSS listen on 2 NICs?
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Isaac,
My mistake. I was thinking this issue was similar to an lctl issue that I have
been seeing for
quite a while now, but as you say this is not the case since the node only has a
single NID. I
just oversimplified the problem. The case that I am referring to is what I think
is a bug in the
lctl ping code.
Here is an example below of what I was referring to:
Node1:
ilc6 a lustre server that has two separate ethernet devices eth2 and eth3
# ilc6 /root > cat /etc/modprobe.conf
options lnet networks="tcp0(eth2,eth3)" \
routes="elan0 172.16.3.[4-6]@tcp0"
# ilc6 /root > lctl list_nids
172.16.101.6 at tcp
Node2:
adev4 is a lustre router that has two separate ethernet devices and and elan
device
# adev4 /root > cat /etc/modprobe.conf
options lnet networks="tcp0(eth0,eth1),elan0" \
forwarding="enabled"
# adev4 /root > lctl list_nids
172.16.3.4 at tcp
4 at elan
Node3:
adev3 is a lustre client with only an elan device
# adev3 /root > lctl list_nids
3 at elan
Now the actual problem here is that
1) ilc6 can only successfully issue an lctl ping to the tcp nid even though it
knows
how to get to the elan0 network.
2) adev3 can only successfully issue an lctl ping to the elan nid even though it
knows
how to get to the tcp0 network.
FROM Node1:
# ilc6 /root > lctl ping 172.16.3.4 at tcp0
12345-0 at lo
12345-172.16.3.4 at tcp
12345-4 at elan
# ilc6 /root > lctl ping 3 at elan
12345-0 at lo
12345-3 at elan
ERROR:
# ilc6 /root > lctl ping 4 at elan
failed to ping 4 at elan: Input/output error
FROM Node3:
# adev3 /root > lctl ping 4 at elan
12345-0 at lo
12345-172.16.3.4 at tcp
12345-4 at elan
# adev3 /root > lctl ping 172.16.101.6 at tcp
12345-0 at lo
12345-172.16.101.6 at tcp
ERROR:
# adev3 /root > lctl ping 172.16.3.4 at tcp
failed to ping 172.16.3.4 at tcp: Input/output error
This is the error I was mistakenly trying to describe yesterday.
Isaac Huang wrote:> On Wed, Jan 16, 2008 at 11:23:30AM -0800, Herb Wartens wrote:
> Andrew,
> I have not used lustre-1.6.4.X yet, but in previous versions (and most
likely the version you are using)
> Lustre actually listens on all interfaces no matter what you specify in the
modprobe.conf. You can verify this
> by looking at the netstat output for port 988 and look for what ports you
are listening on. We here at LLNL
> regularly use multiple interfaces.
> I believe that the issue you are referring to is a bug in the lctl ping
code where the ping only responds
> over the first network device specified for a particular lnd. As long as
you have properly configured your
> host routes so that you can ping both interfaces from the other node you
should be fine. IMHO this should
> just be fixed in lnet so you can do an lctl ping from any endpoint to any
other endpoint.
>
>> I don''t think it''s a lctl ping bug.
>
> # ilc6 /root > cat /etc/modprobe.conf
> options lnet networks="tcp0(eth2,eth3)"
>
>> This config gives the node only one NID: ip_of_eth2 at tcp0. You can
>> verify it by ''lctl list_nids'' on the node.
>
> # ilc6 /root > netstat -a -t -n | grep 988 | grep LIST
> tcp 0 0 0.0.0.0:988 0.0.0.0:*
LISTEN
>
> # ilc6 /root > cat /etc/hosts | grep ilc7
> 172.16.101.7 ilc7-lnet0 ilc7-eth2
> 172.16.102.7 ilc7-lnet1 ilc7-eth3
>
> # ilc6 /root > lctl ping 172.16.101.7 at tcp0
> 12345-0 at lo
> 12345-172.16.101.7 at tcp
>
>> When you lctl ping a node at any one of its NIDs, the ping reply
>> contains a list of all NIDs of the node. As can be seen from the reply
>> above, 172.16.101.7 at tcp0 has two NIDs: 0 at lo and 172.16.101.7 at
tcp.
>
>> So when you tried ''lctl ping 172.16.102.7 at tcp0'',
the ping request
>> could reach 172.16.102.7, but it was rejected since 172.16.102.7 at
tcp0
>> was not one of the node''s NIDs.
>
>> The socklnd does interface bonding transparently from lnet''s
>> perspective. It exchanges a list of IPs of all NICs under a lnet NID
>> with peers, and creates connections to all IPs of a peer and thus
>> aggregates bandwidth. Lnet has no knowledge of this - all it sees is
>> just one NID, i.e. ip_of_1st_nic at tcp.
>
>> Isaac
>
> # ilc6 /root > lctl ping 172.16.102.7 at tcp0
> failed to ping 172.16.102.7 at tcp: Input/output error
>
> # ilc6 /root > ping -c 1 172.16.101.7
> PING 172.16.101.7 (172.16.101.7) 56(84) bytes of data.
> 64 bytes from 172.16.101.7: icmp_seq=1 ttl=64 time=0.143 ms
>
> --- 172.16.101.7 ping statistics ---
> 1 packets transmitted, 1 received, 0% packet loss, time 0ms
> rtt min/avg/max/mdev = 0.143/0.143/0.143/0.000 ms
> # ilc6 /root > ping -c 1 172.16.102.7
> PING 172.16.102.7 (172.16.102.7) 56(84) bytes of data.
> 64 bytes from 172.16.102.7: icmp_seq=1 ttl=64 time=0.094 ms
>
> --- 172.16.102.7 ping statistics ---
> 1 packets transmitted, 1 received, 0% packet loss, time 0ms
> rtt min/avg/max/mdev = 0.094/0.094/0.094/0.000 ms
>
>
> Lundgren, Andrew wrote:
>>>> So the only way to use two nics at once is to bond? I am more
for redundancy rather than increased throughput.
>>>>
>>>>> -----Original Message-----
>>>>> From: He.Huang at Sun.COM [mailto:He.Huang at Sun.COM]
>>>>> Sent: Wednesday, January 16, 2008 6:34 AM
>>>>> To: Lundgren, Andrew
>>>>> Cc: ''Lustre-discuss at clusterfs.com''
>>>>> Subject: Re: [Lustre-discuss] How do you make an MGS/OSS
>>>>> listen on 2 NICs?
>>>>>
>>>>> On Tue, Jan 15, 2008 at 10:28:33AM -0700, Lundgren, Andrew
wrote:
>>>>>> I am running on CentOS 5 distribution without adding
any
>>>>> updates from
>>>>>> CentOS. I am using the lustre 1.6.4.1 kernel and
software.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have two NICs that run though different switches.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have the lustre options in my modprobe.conf to
look like this:
>>>>>>
>>>>>>
>>>>>>
>>>>>> options lnet networks=tcp0(eth1,eth0)
>>>>>>
>>>>> This way of interface bonding is now a deprecated lnet
>>>>> feature. Please refer to:
>>>>>
http://manual.lustre.org/manual/LustreManual16_HTML/DynamicHTM
>>>>> L-13-1.html
>>>>>
>>>>> Isaac
>>>>>
>>>> _______________________________________________
>>>> Lustre-discuss mailing list
>>>> Lustre-discuss at clusterfs.com
>>>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
>>
_______________________________________________
Lustre-discuss mailing list
Lustre-discuss at clusterfs.com
https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFHj8/SP/62XqEEbMYRCtrrAKC4q4EWSdmjKmLaR9itrEoa4gdd0gCgn32S
OI4G8yg8Czvy1lsLNYHqBcY=R7ZB
-----END PGP SIGNATURE-----
Isaac Huang
2008-Jan-18 15:23 UTC
[Lustre-discuss] How do you make an MGS/OSS listen on 2 NICs?
On Thu, Jan 17, 2008 at 01:59:46PM -0800, Herb Wartens wrote:> ...... > Here is an example below of what I was referring to: > > Node1: > ilc6 a lustre server that has two separate ethernet devices eth2 and eth3 > > # ilc6 /root > cat /etc/modprobe.conf > options lnet networks="tcp0(eth2,eth3)" \ > routes="elan0 172.16.3.[4-6]@tcp0" > > # ilc6 /root > lctl list_nids > 172.16.101.6 at tcp > > Node2: > adev4 is a lustre router that has two separate ethernet devices and and elan device > > # adev4 /root > cat /etc/modprobe.conf > options lnet networks="tcp0(eth0,eth1),elan0" \ > forwarding="enabled" > > # adev4 /root > lctl list_nids > 172.16.3.4 at tcp > 4 at elan > > Node3: > adev3 is a lustre client with only an elan device > > # adev3 /root > lctl list_nids > 3 at elan > > > Now the actual problem here is that > 1) ilc6 can only successfully issue an lctl ping to the tcp nid even though it knows > how to get to the elan0 network. > 2) adev3 can only successfully issue an lctl ping to the elan nid even though it knows > how to get to the tcp0 network. > > FROM Node1: > # ilc6 /root > lctl ping 172.16.3.4 at tcp0 > 12345-0 at lo > 12345-172.16.3.4 at tcp > 12345-4 at elan > > # ilc6 /root > lctl ping 3 at elan > 12345-0 at lo > 12345-3 at elan > > ERROR: > # ilc6 /root > lctl ping 4 at elan > failed to ping 4 at elan: Input/output error >The router rejected the ping request message because it believed that ilc6 could reach him via another NID (172.16.3.4 at tcp0) which was closer to ilc6 than its elan NID. You should see a message on router dmesg that read: 172.16.101.6 at tcp, src 172.16.101.6 at tcp: Bad dest nid 4 at elan ...... I don''t think it''s a lnet bug. Isaac
Herb Wartens
2008-Jan-18 16:54 UTC
[Lustre-discuss] How do you make an MGS/OSS listen on 2 NICs?
Isaac Huang wrote:> On Thu, Jan 17, 2008 at 01:59:46PM -0800, Herb Wartens wrote: >> ...... >> Here is an example below of what I was referring to: >> >> Node1: >> ilc6 a lustre server that has two separate ethernet devices eth2 and eth3 >> >> # ilc6 /root > cat /etc/modprobe.conf >> options lnet networks="tcp0(eth2,eth3)" \ >> routes="elan0 172.16.3.[4-6]@tcp0" >> >> # ilc6 /root > lctl list_nids >> 172.16.101.6 at tcp >> >> Node2: >> adev4 is a lustre router that has two separate ethernet devices and and elan device >> >> # adev4 /root > cat /etc/modprobe.conf >> options lnet networks="tcp0(eth0,eth1),elan0" \ >> forwarding="enabled" >> >> # adev4 /root > lctl list_nids >> 172.16.3.4 at tcp >> 4 at elan >> >> Node3: >> adev3 is a lustre client with only an elan device >> >> # adev3 /root > lctl list_nids >> 3 at elan >> >> >> Now the actual problem here is that >> 1) ilc6 can only successfully issue an lctl ping to the tcp nid even though it knows >> how to get to the elan0 network. >> 2) adev3 can only successfully issue an lctl ping to the elan nid even though it knows >> how to get to the tcp0 network. >> >> FROM Node1: >> # ilc6 /root > lctl ping 172.16.3.4 at tcp0 >> 12345-0 at lo >> 12345-172.16.3.4 at tcp >> 12345-4 at elan >> >> # ilc6 /root > lctl ping 3 at elan >> 12345-0 at lo >> 12345-3 at elan >> >> ERROR: >> # ilc6 /root > lctl ping 4 at elan >> failed to ping 4 at elan: Input/output error >> > > The router rejected the ping request message because it believed that > ilc6 could reach him via another NID (172.16.3.4 at tcp0) which was > closer to ilc6 than its elan NID. > > You should see a message on router dmesg that read: > 172.16.101.6 at tcp, src 172.16.101.6 at tcp: Bad dest nid 4 at elan ...... > > I don''t think it''s a lnet bug.Correct that is how it works now, however I believe we have different ideas of what lctl ping should mean. In my opinion it should not matter if you can get to a node via another nid that is "closer." I feel that the lctl ping should show lnet connectivity regardless of the fact that there is a "closer" network. Maybe just me...=) -Herb> > Isaac
Lundgren, Andrew
2008-Jan-21 22:24 UTC
[Lustre-discuss] How to configure Redundant NICs over separate switches and subnets.
I am still unclear on how this should be configured. All of my clients, OSS and MGS servers have two nics, on different subnets, connected to different switches. This is done for network redundancy. We are also using aliased IP addresses for lustre in the 192.168 address space. We are not bonding interfaces. In my /etc/modprobe.conf file on all of the machines I have the following line: options lnet networks=tcp0(eth1:0),tcp1(eth0:0) When I format my OSTs, I have used: mkfs.lustre --fsname stage --ost --mgsnode=192.168.135.999 at tcp0 --param="failover.mode=failout" /dev/md6 Where 999 is the IP on the aliased nic. (Should I use both tcp0 and tcp1 with a comma?) When I mount the clients in my /etc/fstab, I have done this: 192.168.136.81 at tcp0,192.168.135.80 at tcp1:/stage /stage lustre defaults,_netdev 0 0 My intent is that if one path to the MGS fails, the second one will be used. Am I doing this correct, or am I off base here? Thanks! -- Andrew Lundgren
D. Marc Stearman
2008-Jan-22 17:03 UTC
[Lustre-discuss] How to configure Redundant NICs over separate switches and subnets.
As far as I know, LNET will use the shortest path on the network, so if you have two equivalent tcp networks, tcp0 and tcp1, LNET will just use the first one. If it fails, it should use the second one. If both NICs are in the same tcp network, LNET should use both. Whether you decide on one or two LNET networks is up to you. Regardless, your fstab entry is not correct. You should only list one server as the host:> 192.168.136.81 at tcp0:/stage /stage lustre > defaults,_netdev 0 0or> 192.168.135.80 at tcp1:/stage /stage lustre > defaults,_netdev 0 0If one NIC fails, while a client is not mounted, you would have to change the fstab to remount. If lustre is already mounted, it should just use the other LNET network. -Marc ---- D. Marc Stearman LC Lustre Systems Administrator marc at llnl.gov 925.423.9670 Pager: 1.888.203.0641 On Jan 21, 2008, at 2:24 PM, Lundgren, Andrew wrote:> I am still unclear on how this should be configured. > > All of my clients, OSS and MGS servers have two nics, on different > subnets, connected to different switches. This is done for network > redundancy. We are also using aliased IP addresses for lustre in > the 192.168 address space. We are not bonding interfaces. > > In my /etc/modprobe.conf file on all of the machines I have the > following line: > > options lnet networks=tcp0(eth1:0),tcp1(eth0:0) > > When I format my OSTs, I have used: > > mkfs.lustre --fsname stage --ost --mgsnode=192.168.135.999 at tcp0 -- > param="failover.mode=failout" /dev/md6 > > Where 999 is the IP on the aliased nic. (Should I use both tcp0 > and tcp1 with a comma?) > > When I mount the clients in my /etc/fstab, I have done this: > > 192.168.136.81 at tcp0,192.168.135.80 at tcp1:/stage /stage > lustre defaults,_netdev 0 0 > > My intent is that if one path to the MGS fails, the second one will > be used. > > Am I doing this correct, or am I off base here? > > Thanks! > > -- > Andrew Lundgren >
Lundgren, Andrew
2008-Jan-22 18:12 UTC
[Lustre-discuss] How to configure Redundant NICs over separate switches and subnets.
> > As far as I know, LNET will use the shortest path on the > network, so if you have two equivalent tcp networks, tcp0 and > tcp1, LNET will just use the first one. If it fails, it > should use the second one. > If both NICs are in the same tcp network, LNET should use both. > Whether you decide on one or two LNET networks is up to you.So setting up tcp0(eth1:0,eth0:0) and tcp0(eth1:0),tcp1(eth0:0) are functionally equivalent for what I am doing?> Regardless, your fstab entry is not correct. You should only > list one server as the host: > > > 192.168.136.81 at tcp0:/stage /stage lustre > > defaults,_netdev 0 0 > > or > > > 192.168.135.80 at tcp1:/stage /stage lustre > > defaults,_netdev 0 0 > > If one NIC fails, while a client is not mounted, you would > have to change the fstab to remount. If lustre is already > mounted, it should just use the other LNET network.Then as long as the box does not reboot while the network is down, the mount should still function, just over the secondary path? Thank you for the clarification. -- Andrew
D. Marc Stearman
2008-Jan-22 18:19 UTC
[Lustre-discuss] How to configure Redundant NICs over separate switches and subnets.
Functionally, it will work the same, but not performance wise. tcp0(eth1:0),tcp1(eth0:0) will create two LNET networks, and it will use the shorter of the two. If they are the same in terms of network hops from client to server, it will use the first one, and only the first one. This setup would create two NIDs on the servers, so you could use either fstab entry discussed before. tcp0(eth1:0,eth0:0) will create one LNET network, and use all interfaces between clients and servers. You would have double the bandwidth. This setup would create only one NID on the servers, and you would use the NID assiociated with eth1:0 in your fstab entries. If the NIC (or network) for that NID failed on the mgs/mds you would not be able to mount new clients, but your filesystem should still work, as it will mark that route down and use the other interface. -Marc ---- D. Marc Stearman LC Lustre Systems Administrator marc at llnl.gov 925.423.9670 Pager: 1.888.203.0641 On Jan 22, 2008, at 10:12 AM, Lundgren, Andrew wrote:>> >> As far as I know, LNET will use the shortest path on the >> network, so if you have two equivalent tcp networks, tcp0 and >> tcp1, LNET will just use the first one. If it fails, it >> should use the second one. >> If both NICs are in the same tcp network, LNET should use both. >> Whether you decide on one or two LNET networks is up to you. > > So setting up and are functionally equivalent for what I am doing? > >> Regardless, your fstab entry is not correct. You should only >> list one server as the host: >> >>> 192.168.136.81 at tcp0:/stage /stage lustre >>> defaults,_netdev 0 0 >> >> or >> >>> 192.168.135.80 at tcp1:/stage /stage lustre >>> defaults,_netdev 0 0 >> >> If one NIC fails, while a client is not mounted, you would >> have to change the fstab to remount. If lustre is already >> mounted, it should just use the other LNET network. > > Then as long as the box does not reboot while the network is down, > the mount should still function, just over the secondary path? > > Thank you for the clarification. > > > -- > Andrew >