Hi, I learn about LNet and I also have read one of the recent post with subject ''Adding IB to tcp only cluster''. Interesting is how to use multiple interfaces on the same server in Lustre/LNet. My understanding is that TCP(ksocklnd) can manage multiple physical interfaces as one LNet interface with one unique NID. Is that still correct and recommended to use? Or is it better to setup Ethernet bonding(under Linux) and bind that bonding interface to LNet? Beside of TCP it is only possible to use multiple interfaces on the same node with o2ib, right? With ko2iblnd one can setup several Lustre networks for each IB interface. In fact you must setup several Lustre networks otherwise only the 1st IB interface is used, correct? It is not clear for me how MGS, MDS, OSS and Client choose a NID for communication. I mean I know that LNet choose the best one, but who provides the list with all available NIDs for a server? Or does it work somehow different? I''am aware of ''lctl list_nids'' and ''lctl which_nid <list of nids>''. That is fine. I would like to know what a Lustre server or a client does. Furthermore I don''t get the MGS target information kept in MDT and OST devices. For what and how is it used? What happens during mount of MDT or OST and who talks to each other and how? I''am looking forward to get some better understanding. Thank you and Best regards, Danny -- Danny Sternkopf http://www.nec.de/hpc dsternkopf at hpce.nec.com HPCE Division Germany phone: +49-711-68770-35 fax: +49-711-6877145 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ NEC Deutschland GmbH, Hansaallee 101, 40549 D?sseldorf Gesch?ftsf?hrer Yuya Momose Handelsregister D?sseldorf HRB 57941; VAT ID DE129424743
On Mon, Oct 13, 2008 at 05:29:14PM +0200, Danny Sternkopf wrote:> ...... > Interesting is how to use multiple interfaces on the same server in > Lustre/LNet. My understanding is that TCP(ksocklnd) can manage multiple > physical interfaces as one LNet interface with one unique NID. Is that > still correct and recommended to use? Or is it better to setup Ethernet > bonding(under Linux) and bind that bonding interface to LNet?Yes, ksocklnd is capable of load-balancing over multiple physical interfaces under one LNet interface, but the Linux Ethernet bonding is the recommended way to go.> Beside of TCP it is only possible to use multiple interfaces on the same > node with o2ib, right? With ko2iblnd one can setup several Lustre > networks for each IB interface. In fact you must setup several Lustre > networks otherwise only the 1st IB interface is used, correct?Yes, ksocklnd and ko2iblnd are the only LNDs that support multiple instances, i.e. LNet interfaces. Unlike ksocklnd, each ko2iblnd interface only sits on one IB interface, and multiple ko2iblnd interfaces can sit on a same IB interface.> It is not clear for me how MGS, MDS, OSS and Client choose a NID for > communication. I mean I know that LNet choose the best one, but who > provides the list with all available NIDs for a server? Or does it work > somehow different?To be precise, for outgoing messages, LNet only chooses a local NID to use, which is determined by the remote NID specified by PTLRPC (and ultimately by Lustre). Though it''s possible to specify a local NID (e.g. the ''self'' parameter of LNetPut), it is in fact used only for sanity checks. Isaac
Hi, thank your for clarification!>> Beside of TCP it is only possible to use multiple interfaces on the same >> node with o2ib, right? With ko2iblnd one can setup several Lustre >> networks for each IB interface. In fact you must setup several Lustre >> networks otherwise only the 1st IB interface is used, correct? > > Yes, ksocklnd and ko2iblnd are the only LNDs that support multiple > instances, i.e. LNet interfaces. Unlike ksocklnd, each ko2iblnd > interface only sits on one IB interface, and multiple ko2iblnd > interfaces can sit on a same IB interface.That''s interesting as well. Does it mean I can configure two LNet networks over the same IB interface like this?: options lnet networks=o2ib0(ib0),o2ib1(ib0)> >> It is not clear for me how MGS, MDS, OSS and Client choose a NID for >> communication. I mean I know that LNet choose the best one, but who >> provides the list with all available NIDs for a server? Or does it work >> somehow different? > > To be precise, for outgoing messages, LNet only chooses a local NID to > use, which is determined by the remote NID specified by PTLRPC (and > ultimately by Lustre). Though it''s possible to specify a local NID > (e.g. the ''self'' parameter of LNetPut), it is in fact used only for > sanity checks.Does MGS provide a list of NIDs to servers and clients? And then the node chooses the best one. Who knows more about this? Best regards, Danny -- Danny Sternkopf http://www.nec.de/hpc dsternkopf at hpce.nec.com HPCE Division Germany phone: +49-711-68770-35 fax: +49-711-6877145 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ NEC Deutschland GmbH, Hansaallee 101, 40549 D?sseldorf Gesch?ftsf?hrer Yuya Momose Handelsregister D?sseldorf HRB 57941; VAT ID DE129424743
On Mon, Oct 20, 2008 at 01:20:46PM +0200, Danny Sternkopf wrote:> >> Beside of TCP it is only possible to use multiple interfaces on the same > >> node with o2ib, right? With ko2iblnd one can setup several Lustre > >> networks for each IB interface. In fact you must setup several Lustre > >> networks otherwise only the 1st IB interface is used, correct? > > > > Yes, ksocklnd and ko2iblnd are the only LNDs that support multiple > > instances, i.e. LNet interfaces. Unlike ksocklnd, each ko2iblnd > > interface only sits on one IB interface, and multiple ko2iblnd > > interfaces can sit on a same IB interface. > > That''s interesting as well. Does it mean I can configure two LNet > networks over the same IB interface like this?: > options lnet networks=o2ib0(ib0),o2ib1(ib0)Yes, but such config has only been used for testing, e.g. testing routers without multi-homed nodes. Isaac