Hi, I try to add new subnet to the Lustre server. I had tcp, via eth0, and IB connection and the lustre config to used tcp and IB network. I connect eth1 to the new subnet and used add interface in the lctl tools to add the new interface. When I run *interface_list *I see the first subnet and after some seconds the new subnet I add. But when I run *list_nids *I see only the first subnet and the IB network, and when I try to mount client vi the new subnet I get error connection. Is there any idea what can I do to solve the problem ? David
David, If you have both tcp and IB configured, I guess there is something like this in /etc/modprobe.conf: options lnet ''networks="tcp(eth0),o2ib(ib0)"'' Try to substitute tcp(eth0) with tcp0(eth0),tcp1(eth1) there. As for add_interface lctl command - this is for some special ksocklnd feature resembling ethernet-bonding. The effect is the same as tcp(eth0,eth1) in modprobe.conf. And tcp(eth0,eth1) is not equivalent to tcp0(eth0),tcp1(eth1) thoroughly. Sincerely, Maxim> Hi, > I try to add new subnet to the Lustre server. I had tcp, via eth0, and > IB connection and the lustre config to used tcp and IB network. I > connect eth1 to the new subnet and used add interface in the lctl tools > to add the new interface. When I run *interface_list *I see the first > subnet and after some seconds the new subnet I add. But when I run > *list_nids *I see only the first subnet and the IB network, and when I > try to mount client vi the new subnet I get error connection. Is there > any idea what can I do to solve the problem ? > > David > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
Is there a place where the networks options are thoroughly discussed? I have read though the manual and not found a lot on it. I would like to understand exactly what the different options and behaviors are. Thanks! -- Andrew> -----Original Message----- > From: lustre-discuss-bounces at lists.lustre.org > [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of > Maxim V. Patlasov > Sent: Monday, June 02, 2008 7:14 AM > To: David L.H. > Cc: lustre-discuss at lists.lustre.org > Subject: Re: [Lustre-discuss] lctl problem > > David, > > If you have both tcp and IB configured, I guess there is > something like > this in /etc/modprobe.conf: > > options lnet ''networks="tcp(eth0),o2ib(ib0)"'' > > Try to substitute tcp(eth0) with tcp0(eth0),tcp1(eth1) there. As for > add_interface lctl command - this is for some special ksocklnd feature > resembling ethernet-bonding. The effect is the same as > tcp(eth0,eth1) in > modprobe.conf. And tcp(eth0,eth1) is not equivalent to > tcp0(eth0),tcp1(eth1) thoroughly. > > Sincerely, > Maxim > > > Hi, > > I try to add new subnet to the Lustre server. I had tcp, > via eth0, and > > IB connection and the lustre config to used tcp and IB network. I > > connect eth1 to the new subnet and used add interface in > the lctl tools > > to add the new interface. When I run *interface_list *I see > the first > > subnet and after some seconds the new subnet I add. But when I run > > *list_nids *I see only the first subnet and the IB network, > and when I > > try to mount client vi the new subnet I get error > connection. Is there > > any idea what can I do to solve the problem ? > > > > David > > > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-discuss at lists.lustre.org > > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
Thanks so much for your e-mail, I did it for all the OSS and MDS and now I had 3 network when I run ''lctl list_nids'' I get: 10.xx.xx.xx at o2ib 10.xx.xx.xx at tcp 192.xx.xx.xx at tcp1 Now I need to config and mount the client, some clients connect to ib device, some to 10.xx and some to 192.xx. When I mount -t luster 10.xx.xx.xx at tcp or 10.xx.xx.xx at o2ib it''s look good and every thing work. I don''t know what to do with the client that suppose to connect to tcp1, what should be at /etc/modprob.conf and how I mount the lustre system, files, etc. 192.xx.xx.xx at tcp or 192.xx.xx.xx at tcp1 ? thanks david Maxim V. Patlasov wrote:> David, > > If you have both tcp and IB configured, I guess there is something > like this in /etc/modprobe.conf: > > options lnet ''networks="tcp(eth0),o2ib(ib0)"'' > > Try to substitute tcp(eth0) with tcp0(eth0),tcp1(eth1) there. As for > add_interface lctl command - this is for some special ksocklnd feature > resembling ethernet-bonding. The effect is the same as tcp(eth0,eth1) > in modprobe.conf. And tcp(eth0,eth1) is not equivalent to > tcp0(eth0),tcp1(eth1) thoroughly. > > Sincerely, > Maxim > >> Hi, >> I try to add new subnet to the Lustre server. I had tcp, via eth0, >> and IB connection and the lustre config to used tcp and IB network. >> I connect eth1 to the new subnet and used add interface in the lctl >> tools to add the new interface. When I run *interface_list *I see the >> first subnet and after some seconds the new subnet I add. But when I >> run *list_nids *I see only the first subnet and the IB network, and >> when I try to mount client vi the new subnet I get error connection. >> Is there any idea what can I do to solve the problem ? >> >> David >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> > >