We currently use lustre over an infiniband network. We would like to allow ethernet clients (ie. no infiniband) to use the same lustre FS. The MDT and OSTs were all created using only the MGS infiniband interface (ie. mkfs.lustre --mgsnode=10.2.1.30 at ib0). We have modified the OSTs and MDS to route over ib & tcp networks via lustre lnet module flags: >options lnet networks=o2ib0(ib0),tcp(eth0) For the ethernet clients, we specify only the tcp network for the lnet module: >options lnet network=tcp(eth0) For the infiniband clients, we use only the ib network in lnet module: >options lnet networks=o2ib(ib0) From an infiniband client, we mount lustre using the MDS infiniband IP address. eg) mount -t lustre 10.2.1.30 at ib0:/lustre0 /lustre0 On an ethernet client, mounting lustre using the MDS tcp interface fails. eg) mount -t lustre 10.1.1.30 at tcp0:/lustre0 /lustre0 From the syslog messages, it appears the MGS tells the ethernet client to use the ib0 interface (see attached syslog messages) to find the MDT. So I am missing some configuration for the tcp network. My best guess we need to use the "tunefs.lustre" command to add the mgs tcp NID to the MDT. However this command won''t execute while lustre is mounted. Any suggestions how to add the tcp network to our existing lustre ib network ? Thanks, --Chris -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: bdk-lustre-tcp-syslog.txt Url: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090107/009a1f4d/attachment.txt
On Jan 07, 2009 18:16 -0500, Chris Hunter wrote:> We currently use lustre over an infiniband network. We would like to > allow ethernet clients (ie. no infiniband) to use the same lustre FS. > > The MDT and OSTs were all created using only the MGS infiniband > interface (ie. mkfs.lustre --mgsnode=10.2.1.30 at ib0). > > So I am missing some configuration for the tcp network. My best guess we > need to use the "tunefs.lustre" command to add the mgs tcp NID to the > MDT. However this command won''t execute while lustre is mounted. > > Any suggestions how to add the tcp network to our existing lustre ib > network ?I believe this is documented in the Lustre manual at http://manual.lustre.org/ Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
Andreas Dilger wrote:> On Jan 07, 2009 18:16 -0500, Chris Hunter wrote: >> We currently use lustre over an infiniband network. We would like to >> allow ethernet clients (ie. no infiniband) to use the same lustre FS. >> >> The MDT and OSTs were all created using only the MGS infiniband >> interface (ie. mkfs.lustre --mgsnode=10.2.1.30 at ib0). >> >> So I am missing some configuration for the tcp network. My best guess we >> need to use the "tunefs.lustre" command to add the mgs tcp NID to the >> MDT. However this command won''t execute while lustre is mounted. >> >> Any suggestions how to add the tcp network to our existing lustre ib >> network ? > > I believe this is documented in the Lustre manual at http://manual.lustre.org/Section 7 of the operations manual does discuss multihomed lustre servers. The example given is destructive -- lustre needs to be reformatted (mkfs.lustre) for multihoming. Can we setup multihoming with our re-formating our lustre FS ? Thanks, --Chris
On Jan 09, 2009 18:31 -0500, Chris Hunter wrote:> Andreas Dilger wrote: >> On Jan 07, 2009 18:16 -0500, Chris Hunter wrote: >>> We currently use lustre over an infiniband network. We would like to >>> allow ethernet clients (ie. no infiniband) to use the same lustre FS. >>> >>> The MDT and OSTs were all created using only the MGS infiniband >>> interface (ie. mkfs.lustre --mgsnode=10.2.1.30 at ib0). >>> >>> So I am missing some configuration for the tcp network. My best guess >>> we need to use the "tunefs.lustre" command to add the mgs tcp NID to >>> the MDT. However this command won''t execute while lustre is mounted. >>> >>> Any suggestions how to add the tcp network to our existing lustre ib >>> network ? >> >> I believe this is documented in the Lustre manual at http://manual.lustre.org/ > > Section 7 of the operations manual does discuss multihomed lustre > servers. The example given is destructive -- lustre needs to be > reformatted (mkfs.lustre) for multihoming. Can we setup multihoming with > our re-formating our lustre FS ?Yes, definitely. It should only require a --write-conf... I think this has been answered previously on this list, but if not Isaac should get it into the manual. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
Is there a way to backup the configuration information on the target disk before changing the MGS NID (ie. adding second multi-homed network) ? These config logs are created when the filesystem is created (ie. mkfs.lustre) and not regenerated each time lustre is mounted ? To add the second network, we plan to execute the command: tune2fs.lustre --verbose --erase-param --mgsnode=10.2.1.30 at o2ib,10.1.1.30 at tcp --writeconf /dev/sdb PS It would be helpful to add this example to the manual, assuming it works and is non-destructive. The current example for multi-homed networks (mkfs.lustre) is destructive. Thank-you in advance, Chris Hunter Andreas Dilger wrote:> On Jan 09, 2009 18:31 -0500, Chris Hunter wrote: >> Andreas Dilger wrote: >>> On Jan 07, 2009 18:16 -0500, Chris Hunter wrote: >>>> We currently use lustre over an infiniband network. We would like to >>>> allow ethernet clients (ie. no infiniband) to use the same lustre FS. >>>> >>>> The MDT and OSTs were all created using only the MGS infiniband >>>> interface (ie. mkfs.lustre --mgsnode=10.2.1.30 at ib0). >>>> >>>> So I am missing some configuration for the tcp network. My best guess >>>> we need to use the "tunefs.lustre" command to add the mgs tcp NID to >>>> the MDT. However this command won''t execute while lustre is mounted. >>>> >>>> Any suggestions how to add the tcp network to our existing lustre ib >>>> network ? >>> I believe this is documented in the Lustre manual at http://manual.lustre.org/ >> Section 7 of the operations manual does discuss multihomed lustre >> servers. The example given is destructive -- lustre needs to be >> reformatted (mkfs.lustre) for multihoming. Can we setup multihoming with >> our re-formating our lustre FS ? > > Yes, definitely. It should only require a --write-conf... I think this > has been answered previously on this list, but if not Isaac should get > it into the manual. > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc.