Mr. Mark L. Dotson (Contractor)
2013-May-07 16:16 UTC
[Lustre-discuss] Lustre On Two Clusters
I have Lustre installed and working on 1 cluster. Everything is IB. I can mount clients in this cluster with no problems. I want to mount this Lustre FS on another cluster that is attached to a separate IB switch. What''s the best way to do this? Does it require a separate subnet for the IB interfaces, or does it matter? Currently, my /etc/modprobe.d/lustre.conf has the following: options lnet networks="o2ib0(ib0)" Lustre version is 2.3 OS''s are CentOS 6.4. Any help would be much appreciated. Thanks. Mark -- Mark Dotson Systems Administrator Lockheed-Martin dotsonml.ctr at afrl.hpc.mil
> -----Original Message----- > From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss- > bounces at lists.lustre.org] On Behalf Of Mr. Mark L. Dotson (Contractor) > Sent: Tuesday, May 07, 2013 9:16 AM > To: lustre-discuss at lists.lustre.org > Subject: [Lustre-discuss] Lustre On Two Clusters > > I have Lustre installed and working on 1 cluster. Everything is IB. I can mount > clients in this cluster with no problems. I want to mount this Lustre FS on > another cluster that is attached to a separate IB switch. > What''s the best way to do this? Does it require a separate subnet for the IB > interfaces, or does it matter?Hi Mark, Good to hear from you on the list. Regarding your question, a couple options jump out at me. 1. Add additional interfaces to the servers. This will allow the Lustre servers to be on both IB networks, and able to directly serve the file system to the clients. 2. Use LNet router(s), the basics of which is documented in the operations manuals. Either way, you''ll need to perform some network configuration in (at least) the servers "lustre.conf". -Brett> > Currently, my /etc/modprobe.d/lustre.conf has the following: > > options lnet networks="o2ib0(ib0)" > > Lustre version is 2.3 > OS''s are CentOS 6.4. > > Any help would be much appreciated. Thanks. > > Mark > > > -- > Mark Dotson > Systems Administrator > Lockheed-Martin > dotsonml.ctr at afrl.hpc.mil > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss-- Brett Lee Sr. Systems Engineer Intel High Performance Data Division
Mr. Mark L. Dotson (Contractor)
2013-May-09 15:15 UTC
[Lustre-discuss] Lustre On Two Clusters
Thanks, Lee. Has anyone done any work with Lustre and IB WAN extenders? I need help with my configuration. Thanks, Mark On 05/08/13 11:03, Lee, Brett wrote:>> -----Original Message----- >> From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss- >> bounces at lists.lustre.org] On Behalf Of Mr. Mark L. Dotson (Contractor) >> Sent: Tuesday, May 07, 2013 9:16 AM >> To: lustre-discuss at lists.lustre.org >> Subject: [Lustre-discuss] Lustre On Two Clusters >> >> I have Lustre installed and working on 1 cluster. Everything is IB. I can mount >> clients in this cluster with no problems. I want to mount this Lustre FS on >> another cluster that is attached to a separate IB switch. >> What''s the best way to do this? Does it require a separate subnet for the IB >> interfaces, or does it matter? > > Hi Mark, > > Good to hear from you on the list. > > Regarding your question, a couple options jump out at me. > > 1. Add additional interfaces to the servers. This will allow the Lustre servers to be on both IB networks, and able to directly serve the file system to the clients. > 2. Use LNet router(s), the basics of which is documented in the operations manuals. > > Either way, you''ll need to perform some network configuration in (at least) the servers "lustre.conf". > > -Brett > >> >> Currently, my /etc/modprobe.d/lustre.conf has the following: >> >> options lnet networks="o2ib0(ib0)" >> >> Lustre version is 2.3 >> OS''s are CentOS 6.4. >> >> Any help would be much appreciated. Thanks. >> >> Mark >> >> >> -- >> Mark Dotson >> Systems Administrator >> Lockheed-Martin >> dotsonml.ctr at afrl.hpc.mil >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > -- > Brett Lee > Sr. Systems Engineer > Intel High Performance Data Division >-- Mark Dotson Systems Administrator Lockheed-Martin dotsonml.ctr at afrl.hpc.mil
Hi Mark, I remember that the NRL used them. They had a couple of presentations at the Lustre User Group. Here is some pretty old stuff: http://wiki.lustre.org/images/3/3a/JamesHoffman.pdf Regards, Michael Am 09.05.2013 17:15, schrieb Mr. Mark L. Dotson (Contractor):> Thanks, Lee. > > Has anyone done any work with Lustre and IB WAN extenders? I need help > with my configuration. > > Thanks, > > Mark > > On 05/08/13 11:03, Lee, Brett wrote: >>> -----Original Message----- >>> From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss- >>> bounces at lists.lustre.org] On Behalf Of Mr. Mark L. Dotson (Contractor) >>> Sent: Tuesday, May 07, 2013 9:16 AM >>> To: lustre-discuss at lists.lustre.org >>> Subject: [Lustre-discuss] Lustre On Two Clusters >>> >>> I have Lustre installed and working on 1 cluster. Everything is IB. I can mount >>> clients in this cluster with no problems. I want to mount this Lustre FS on >>> another cluster that is attached to a separate IB switch. >>> What''s the best way to do this? Does it require a separate subnet for the IB >>> interfaces, or does it matter? >> >> Hi Mark, >> >> Good to hear from you on the list. >> >> Regarding your question, a couple options jump out at me. >> >> 1. Add additional interfaces to the servers. This will allow the Lustre servers to be on both IB networks, and able to directly serve the file system to the clients. >> 2. Use LNet router(s), the basics of which is documented in the operations manuals. >> >> Either way, you''ll need to perform some network configuration in (at least) the servers "lustre.conf". >> >> -Brett >> >>> >>> Currently, my /etc/modprobe.d/lustre.conf has the following: >>> >>> options lnet networks="o2ib0(ib0)" >>> >>> Lustre version is 2.3 >>> OS''s are CentOS 6.4. >>> >>> Any help would be much appreciated. Thanks. >>> >>> Mark >>> >>> >>> -- >>> Mark Dotson >>> Systems Administrator >>> Lockheed-Martin >>> dotsonml.ctr at afrl.hpc.mil >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> >> >> -- >> Brett Lee >> Sr. Systems Engineer >> Intel High Performance Data Division >> > >
Mr. Mark L. Dotson (Contractor)
2013-May-13 19:52 UTC
[Lustre-discuss] Lustre On Two Clusters
Thank you all very much. I discovered that I needed to use the same subnet throughout my IB configuration in order for the IB WAN extenders to work properly. I was able to mount Lustre across the WAN and everything is now working fine. Thanks again, Mark On 05/13/13 15:25, Jeremy Filizetti wrote:> We at NRL can probably give you a hand. Do you have Obsidian or Bays? > Which versions of lustre? What''s your ko2ibld module parameter settings. > > Ccing a few others and my work email since I don''t watch my personal > email too closely > > Jeremy > > On May 13, 2013 2:44 PM, "Mr. Mark L. Dotson (Contractor)" > <dotsonml.ctr at afrl.hpc.mil <mailto:dotsonml.ctr at afrl.hpc.mil>> wrote: > > Thanks, Lee. > > Has anyone done any work with Lustre and IB WAN extenders? I need help > with my configuration. > > Thanks, > > Mark > > On 05/08/13 11:03, Lee, Brett wrote: > >> -----Original Message----- > >> From: lustre-discuss-bounces at lists.lustre.org > <mailto:lustre-discuss-bounces at lists.lustre.org> > [mailto:lustre-discuss- <mailto:lustre-discuss-> > >> bounces at lists.lustre.org <mailto:bounces at lists.lustre.org>] On > Behalf Of Mr. Mark L. Dotson (Contractor) > >> Sent: Tuesday, May 07, 2013 9:16 AM > >> To: lustre-discuss at lists.lustre.org > <mailto:lustre-discuss at lists.lustre.org> > >> Subject: [Lustre-discuss] Lustre On Two Clusters > >> > >> I have Lustre installed and working on 1 cluster. Everything is > IB. I can mount > >> clients in this cluster with no problems. I want to mount this > Lustre FS on > >> another cluster that is attached to a separate IB switch. > >> What''s the best way to do this? Does it require a separate > subnet for the IB > >> interfaces, or does it matter? > > > > Hi Mark, > > > > Good to hear from you on the list. > > > > Regarding your question, a couple options jump out at me. > > > > 1. Add additional interfaces to the servers. This will allow > the Lustre servers to be on both IB networks, and able to directly > serve the file system to the clients. > > 2. Use LNet router(s), the basics of which is documented in the > operations manuals. > > > > Either way, you''ll need to perform some network configuration in > (at least) the servers "lustre.conf". > > > > -Brett > > > >> > >> Currently, my /etc/modprobe.d/lustre.conf has the following: > >> > >> options lnet networks="o2ib0(ib0)" > >> > >> Lustre version is 2.3 > >> OS''s are CentOS 6.4. > >> > >> Any help would be much appreciated. Thanks. > >> > >> Mark > >> > >> > >> -- > >> Mark Dotson > >> Systems Administrator > >> Lockheed-Martin > >> dotsonml.ctr at afrl.hpc.mil <mailto:dotsonml.ctr at afrl.hpc.mil> > >> _______________________________________________ > >> Lustre-discuss mailing list > >> Lustre-discuss at lists.lustre.org > <mailto:Lustre-discuss at lists.lustre.org> > >> http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > > > > -- > > Brett Lee > > Sr. Systems Engineer > > Intel High Performance Data Division > > > > > -- > Mark Dotson > Systems Administrator > Lockheed-Martin > dotsonml.ctr at afrl.hpc.mil <mailto:dotsonml.ctr at afrl.hpc.mil> > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org <mailto:Lustre-discuss at lists.lustre.org> > http://lists.lustre.org/mailman/listinfo/lustre-discuss >-- Mark Dotson Systems Administrator Lockheed-Martin dotsonml.ctr at afrl.hpc.mil
We did this at CSCS: http://www.hpcadvisorycouncil.com/events/2012/Switzerland-Workshop/Presentations/Day_2/2_CSCS.pdf Good luck. Jason On 05/13/2013 09:14 PM, Michael Kluge wrote:> Hi Mark, > > I remember that the NRL used them. They had a couple of presentations at > the Lustre User Group. Here is some pretty old stuff: > http://wiki.lustre.org/images/3/3a/JamesHoffman.pdf > > > Regards, Michael > > > Am 09.05.2013 17:15, schrieb Mr. Mark L. Dotson (Contractor): >> Thanks, Lee. >> >> Has anyone done any work with Lustre and IB WAN extenders? I need help >> with my configuration. >> >> Thanks, >> >> Mark >> >> On 05/08/13 11:03, Lee, Brett wrote: >>>> -----Original Message----- >>>> From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss- >>>> bounces at lists.lustre.org] On Behalf Of Mr. Mark L. Dotson (Contractor) >>>> Sent: Tuesday, May 07, 2013 9:16 AM >>>> To: lustre-discuss at lists.lustre.org >>>> Subject: [Lustre-discuss] Lustre On Two Clusters >>>> >>>> I have Lustre installed and working on 1 cluster. Everything is IB. I can mount >>>> clients in this cluster with no problems. I want to mount this Lustre FS on >>>> another cluster that is attached to a separate IB switch. >>>> What''s the best way to do this? Does it require a separate subnet for the IB >>>> interfaces, or does it matter? >>> Hi Mark, >>> >>> Good to hear from you on the list. >>> >>> Regarding your question, a couple options jump out at me. >>> >>> 1. Add additional interfaces to the servers. This will allow the Lustre servers to be on both IB networks, and able to directly serve the file system to the clients. >>> 2. Use LNet router(s), the basics of which is documented in the operations manuals. >>> >>> Either way, you''ll need to perform some network configuration in (at least) the servers "lustre.conf". >>> >>> -Brett >>> >>>> Currently, my /etc/modprobe.d/lustre.conf has the following: >>>> >>>> options lnet networks="o2ib0(ib0)" >>>> >>>> Lustre version is 2.3 >>>> OS''s are CentOS 6.4. >>>> >>>> Any help would be much appreciated. Thanks. >>>> >>>> Mark >>>> >>>> >>>> -- >>>> Mark Dotson >>>> Systems Administrator >>>> Lockheed-Martin >>>> dotsonml.ctr at afrl.hpc.mil >>>> _______________________________________________ >>>> Lustre-discuss mailing list >>>> Lustre-discuss at lists.lustre.org >>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>> >>> -- >>> Brett Lee >>> Sr. Systems Engineer >>> Intel High Performance Data Division >>> >> > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss