Hi all, I have two cases that I want to make sure that I handle correctly. Both involve a machine with two (or more) Myrinet NICs. The first case is that I want to sysadmin to be able to specify which NIC to use. The second case is to use both NICs (e.g. for routing). For the first case, I have a module parameter that allows the caller to set the board ID (0, 1, 2, etc.). That is simple enough. The sysadmin can then modify /etc/modprobe.conf (or /etc/modprobe.d/ kmxlnd) to add "options kmxlnd board=1". For the second case, I can''t rely on the above since both instances will be passed the same board number. How do other LNDs handle this (or do they)? Scott -- Scott Atchley Myricom Inc. http://www.myri.com
Ping. Can other LNDs handle multiple NICs of the same type (e.g. IB or Quadrics)? If so, how does LNET specify which NIC an instance uses? Scott On May 24, 2006, at 4:53 PM, Scott Atchley wrote:> Hi all, > > I have two cases that I want to make sure that I handle correctly. > Both involve a machine with two (or more) Myrinet NICs. The first > case is that I want to sysadmin to be able to specify which NIC to > use. The second case is to use both NICs (e.g. for routing). > > For the first case, I have a module parameter that allows the > caller to set the board ID (0, 1, 2, etc.). That is simple enough. > The sysadmin can then modify /etc/modprobe.conf (or /etc/modprobe.d/ > kmxlnd) to add "options kmxlnd board=1". > > For the second case, I can''t rely on the above since both instances > will be passed the same board number. How do other LNDs handle this > (or do they)? > > Scott > > -- > Scott Atchley > Myricom Inc. > http://www.myri.com > > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel@clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-devel
Hello, I figure I''d get my feet wet and join in on the discussion. My name''s Duane Cloud and I work at the Army High Performance Computing Research Center where I''m working on deploying Lustre. I''m interested in hearing anyone''s thoughts on Scott''s question, as I''d like to learn a bit more with regards to how the mapping, from NID to which interface and low level addressing to use, is performed. Is each interface identified by an "lnet_ni" structure, with its own NID, driver, etc, similar to how interfaces are presented to IP? In other words, each remote node would be identified by X NIDs, which map to specific hardware interfaces, where X represents the number of interfaces (e.g. X = 4, if 2 IP interfaces and 2 IB interfaces are present). So which NIC is used would be identified by the lnet_ni associated with that peer''s NID...And control of which NIC to use would be via which NID is used when talking to that peer. Or is each LND identified by one lnet_ni and each implements it''s own decision logic to control multiple interfaces if/when present? I see where there''s been some performance measurements using "socknal load balancing" and was wondering if there''s some detailed information available as to how it works. Any thoughts floating around about assigning one NID to each peer and then having a "list of paths" that can be used to get to that peer (e.g. peer NID001 can be reached using the SOCKLND with IP address "a.b.c.d", or using the MXLND with address "whatever is meaningful to it")? -- Thank you, Duane Cloud Systems Programmer Network Computing Services, Inc. Army High Performance Computing Research Center (AHPCRC) cloud@ahpcrc.org, 612-337-3407 Desk Scott Atchley wrote:> Ping. > > Can other LNDs handle multiple NICs of the same type (e.g. IB or > Quadrics)? If so, how does LNET specify which NIC an instance uses? > > Scott > > On May 24, 2006, at 4:53 PM, Scott Atchley wrote: > >> Hi all, >> >> I have two cases that I want to make sure that I handle correctly. >> Both involve a machine with two (or more) Myrinet NICs. The first case >> is that I want to sysadmin to be able to specify which NIC to use. The >> second case is to use both NICs (e.g. for routing). >> >> For the first case, I have a module parameter that allows the caller >> to set the board ID (0, 1, 2, etc.). That is simple enough. The >> sysadmin can then modify /etc/modprobe.conf (or >> /etc/modprobe.d/kmxlnd) to add "options kmxlnd board=1". >> >> For the second case, I can''t rely on the above since both instances >> will be passed the same board number. How do other LNDs handle this >> (or do they)? >> >> Scott >> >> -- >> Scott Atchley >> Myricom Inc. >> http://www.myri.com >> >> >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel@clusterfs.com >> https://mail.clusterfs.com/mailman/listinfo/lustre-devel > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel@clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-devel >
Duane, Scott,
I''m sorry not to contribute anything sensible right now, but
I''m a bit
swamped. I''ll get back ASAP....
Cheers,
Eric
> -----Original Message-----
> From: lustre-devel-bounces@clusterfs.com
> [mailto:lustre-devel-bounces@clusterfs.com] On Behalf Of Duane Cloud
> Sent: 30 May 2006 9:07 PM
> To: Scott Atchley
> Cc: Lustre-devel@clusterfs.com; Eric Barton
> Subject: Re: [Lustre-devel] supporting multiple NICs
>
> Hello,
>
> I figure I''d get my feet wet and join in on the discussion.
> My name''s
> Duane Cloud and I work at the Army High Performance Computing
> Research
> Center where I''m working on deploying Lustre.
>
> I''m interested in hearing anyone''s thoughts on
Scott''s
> question, as I''d
> like to learn a bit more with regards to how the mapping, from NID to
> which interface and low level addressing to use, is performed.
>
> Is each interface identified by an "lnet_ni" structure, with its
own
> NID, driver, etc, similar to how interfaces are presented to IP? In
> other words, each remote node would be identified by X NIDs,
> which map
> to specific hardware interfaces, where X represents the number of
> interfaces (e.g. X = 4, if 2 IP interfaces and 2 IB interfaces are
> present). So which NIC is used would be identified by the lnet_ni
> associated with that peer''s NID...And control of which NIC to
> use would
> be via which NID is used when talking to that peer.
>
> Or is each LND identified by one lnet_ni and each implements it''s
own
> decision logic to control multiple interfaces if/when present? I see
> where there''s been some performance measurements using
"socknal load
> balancing" and was wondering if there''s some detailed
information
> available as to how it works.
>
> Any thoughts floating around about assigning one NID to each peer and
> then having a "list of paths" that can be used to get to that
> peer (e.g.
> peer NID001 can be reached using the SOCKLND with IP address
> "a.b.c.d",
> or using the MXLND with address "whatever is meaningful to it")?
>
>
> --
>
> Thank you,
>
> Duane Cloud
> Systems Programmer
>
> Network Computing Services, Inc.
> Army High Performance Computing Research Center (AHPCRC)
>
> cloud@ahpcrc.org, 612-337-3407 Desk
>
>
> Scott Atchley wrote:
> > Ping.
> >
> > Can other LNDs handle multiple NICs of the same type (e.g. IB or
> > Quadrics)? If so, how does LNET specify which NIC an instance uses?
> >
> > Scott
> >
> > On May 24, 2006, at 4:53 PM, Scott Atchley wrote:
> >
> >> Hi all,
> >>
> >> I have two cases that I want to make sure that I handle correctly.
> >> Both involve a machine with two (or more) Myrinet NICs.
> The first case
> >> is that I want to sysadmin to be able to specify which NIC
> to use. The
> >> second case is to use both NICs (e.g. for routing).
> >>
> >> For the first case, I have a module parameter that allows
> the caller
> >> to set the board ID (0, 1, 2, etc.). That is simple enough. The
> >> sysadmin can then modify /etc/modprobe.conf (or
> >> /etc/modprobe.d/kmxlnd) to add "options kmxlnd board=1".
> >>
> >> For the second case, I can''t rely on the above since both
> instances
> >> will be passed the same board number. How do other LNDs
> handle this
> >> (or do they)?
> >>
> >> Scott
> >>
> >> --
> >> Scott Atchley
> >> Myricom Inc.
> >> http://www.myri.com
> >>
> >>
> >> _______________________________________________
> >> Lustre-devel mailing list
> >> Lustre-devel@clusterfs.com
> >> https://mail.clusterfs.com/mailman/listinfo/lustre-devel
> >
> > _______________________________________________
> > Lustre-devel mailing list
> > Lustre-devel@clusterfs.com
> > https://mail.clusterfs.com/mailman/listinfo/lustre-devel
> >
>
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel@clusterfs.com
> https://mail.clusterfs.com/mailman/listinfo/lustre-devel