Hello, I am new to this list and Lustre, please excuse me if I am asking an already-answered question (I searched the archives without success). I''d like to use Lustre on a cluster where MGS/MDT and OSTs hosts can access some logical volumes (one for each host) exported from a SAN through fiber channel links, and available under /dev/mapper/mpathN. Is this configuration possible? Are there any known issues with it? Thanks in advance Regards -- http://digilander.libero.it/mabozzo/
Marco, This configuration is very much supported. You need not do anything special to make lustre work on mpath devices. Thanks Anand On Oct 23, 2007, at 3:54 AM, Marco wrote:> Hello, > > I am new to this list and Lustre, please excuse me if I am asking an > already-answered question (I searched the archives without success). > > I''d like to use Lustre on a cluster where MGS/MDT and OSTs hosts can > access some logical volumes (one for each host) exported from a SAN > through fiber channel links, and available under /dev/mapper/mpathN. > > Is this configuration possible? Are there any known issues with it? > > > Thanks in advance > Regards > > -- > http://digilander.libero.it/mabozzo/ > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
Does device multi-pathing have to be configured before the file system has been created? Is there any documentation I should be referring to on the Lustre side to get this set up? thanks, Klaus> > On Oct 23, 2007, at 3:54 AM, Marco wrote: > >> Hello, >> >> I am new to this list and Lustre, please excuse me if I am asking an >> already-answered question (I searched the archives without success). >> >> I''d like to use Lustre on a cluster where MGS/MDT and OSTs hosts can >> access some logical volumes (one for each host) exported from a SAN >> through fiber channel links, and available under /dev/mapper/mpathN. >> >> Is this configuration possible? Are there any known issues with it? >> >> >> Thanks in advance >> Regards >> >> -- >> http://digilander.libero.it/mabozzo/ >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at clusterfs.com >> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
Klaus Steden wrote:> Does device multi-pathing have to be configured before the file system has > been created? >Lustre needs a consistent device name, so I would say yes.> Is there any documentation I should be referring to on the Lustre side to > get this set up?No, as long as you give Lustre something that looks like a block device it''s fine. There is no special setup for multipath devices. Which is why the consistent device name is important. cliffw> > thanks, > Klaus > >> On Oct 23, 2007, at 3:54 AM, Marco wrote: >> >>> Hello, >>> >>> I am new to this list and Lustre, please excuse me if I am asking an >>> already-answered question (I searched the archives without success). >>> >>> I''d like to use Lustre on a cluster where MGS/MDT and OSTs hosts can >>> access some logical volumes (one for each host) exported from a SAN >>> through fiber channel links, and available under /dev/mapper/mpathN. >>> >>> Is this configuration possible? Are there any known issues with it? >>> >>> >>> Thanks in advance >>> Regards >>> >>> -- >>> http://digilander.libero.it/mabozzo/ >>> >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss at clusterfs.com >>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
What about the FS label, i.e. LUSTRE-OST0001 (which is what''s actually in my /etc/fstab, even though Lustre mounts a specific device)? Does a multipathed setup automatically address failover, or is STONITH configuration still required to automatically remap devices normally served by downed servers? thanks, Klaus On 10/24/07 4:09 PM, "Cliff White" <Cliff.White at Sun.COM>did etch on stone tablets:> Klaus Steden wrote: >> Does device multi-pathing have to be configured before the file system has >> been created? >> > Lustre needs a consistent device name, so I would say yes. > >> Is there any documentation I should be referring to on the Lustre side to >> get this set up? > > No, as long as you give Lustre something that looks like a block device > it''s fine. There is no special setup for multipath devices. Which is why > the consistent device name is important. > > cliffw > >> >> thanks, >> Klaus >> >>> On Oct 23, 2007, at 3:54 AM, Marco wrote: >>> >>>> Hello, >>>> >>>> I am new to this list and Lustre, please excuse me if I am asking an >>>> already-answered question (I searched the archives without success). >>>> >>>> I''d like to use Lustre on a cluster where MGS/MDT and OSTs hosts can >>>> access some logical volumes (one for each host) exported from a SAN >>>> through fiber channel links, and available under /dev/mapper/mpathN. >>>> >>>> Is this configuration possible? Are there any known issues with it? >>>> >>>> >>>> Thanks in advance >>>> Regards >>>> >>>> -- >>>> http://digilander.libero.it/mabozzo/ >>>> >>>> _______________________________________________ >>>> Lustre-discuss mailing list >>>> Lustre-discuss at clusterfs.com >>>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at clusterfs.com >> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >
Sorry, I should have made that first sentence more clear: Does an FS label (i.e. LUSTRE-OST0001) provide the same functionality as the consistent device name that the multipath device-mapper provides? thanks, Klaus On 10/24/07 5:43 PM, "Klaus Steden" <klaus.steden at thomson.net>did etch on stone tablets:> > What about the FS label, i.e. LUSTRE-OST0001 (which is what''s actually in my > /etc/fstab, even though Lustre mounts a specific device)? > > Does a multipathed setup automatically address failover, or is STONITH > configuration still required to automatically remap devices normally served > by downed servers? > > thanks, > Klaus > > On 10/24/07 4:09 PM, "Cliff White" <Cliff.White at Sun.COM>did etch on stone > tablets: > >> Klaus Steden wrote: >>> Does device multi-pathing have to be configured before the file system has >>> been created? >>> >> Lustre needs a consistent device name, so I would say yes. >> >>> Is there any documentation I should be referring to on the Lustre side to >>> get this set up? >> >> No, as long as you give Lustre something that looks like a block device >> it''s fine. There is no special setup for multipath devices. Which is why >> the consistent device name is important. >> >> cliffw >> >>> >>> thanks, >>> Klaus >>> >>>> On Oct 23, 2007, at 3:54 AM, Marco wrote: >>>> >>>>> Hello, >>>>> >>>>> I am new to this list and Lustre, please excuse me if I am asking an >>>>> already-answered question (I searched the archives without success). >>>>> >>>>> I''d like to use Lustre on a cluster where MGS/MDT and OSTs hosts can >>>>> access some logical volumes (one for each host) exported from a SAN >>>>> through fiber channel links, and available under /dev/mapper/mpathN. >>>>> >>>>> Is this configuration possible? Are there any known issues with it? >>>>> >>>>> >>>>> Thanks in advance >>>>> Regards >>>>> >>>>> -- >>>>> http://digilander.libero.it/mabozzo/ >>>>> >>>>> _______________________________________________ >>>>> Lustre-discuss mailing list >>>>> Lustre-discuss at clusterfs.com >>>>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >>> >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss at clusterfs.com >>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >> > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
On Oct 24, 2007 17:49 -0700, Klaus Steden wrote:> Does an FS label (i.e. LUSTRE-OST0001) provide the same functionality as the > consistent device name that the multipath device-mapper provides?That depends... It depends on what "blkid -t LABEL={fsname}-OST0001" returns. It _should_ be smart enough to return the DM device, but it is prudent to make sure of this. There shouldn''t be any problem with mounting the Lustre filesystems by LABEL= (which is one reason we moved to a mount-based setup).> On 10/24/07 5:43 PM, "Klaus Steden" <klaus.steden at thomson.net>did etch on > stone tablets: > > Does a multipathed setup automatically address failover, or is STONITH > > configuration still required to automatically remap devices normally served > > by downed servers?No, multipath doesn''t help failover at all, it only prevents device access problems for the one node accessing the OST filesystem.> > On 10/24/07 4:09 PM, "Cliff White" <Cliff.White at Sun.COM>did etch on stone > > tablets: > > > >> Klaus Steden wrote: > >>> Does device multi-pathing have to be configured before the file system has > >>> been created? > >>> > >> Lustre needs a consistent device name, so I would say yes. > >> > >>> Is there any documentation I should be referring to on the Lustre side to > >>> get this set up? > >> > >> No, as long as you give Lustre something that looks like a block device > >> it''s fine. There is no special setup for multipath devices. Which is why > >> the consistent device name is important. > >> > >> cliffw > >> > >>> > >>> thanks, > >>> Klaus > >>> > >>>> On Oct 23, 2007, at 3:54 AM, Marco wrote: > >>>> > >>>>> Hello, > >>>>> > >>>>> I am new to this list and Lustre, please excuse me if I am asking an > >>>>> already-answered question (I searched the archives without success). > >>>>> > >>>>> I''d like to use Lustre on a cluster where MGS/MDT and OSTs hosts can > >>>>> access some logical volumes (one for each host) exported from a SAN > >>>>> through fiber channel links, and available under /dev/mapper/mpathN. > >>>>> > >>>>> Is this configuration possible? Are there any known issues with it? > >>>>> > >>>>> > >>>>> Thanks in advance > >>>>> Regards > >>>>> > >>>>> -- > >>>>> http://digilander.libero.it/mabozzo/ > >>>>> > >>>>> _______________________________________________ > >>>>> Lustre-discuss mailing list > >>>>> Lustre-discuss at clusterfs.com > >>>>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > >>> > >>> _______________________________________________ > >>> Lustre-discuss mailing list > >>> Lustre-discuss at clusterfs.com > >>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > >> > > > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-discuss at clusterfs.com > > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discussCheers, Andreas -- Andreas Dilger Sr. Software Engineer, Lustre Group Sun Microsystems of Canada, Inc.
> That depends... It depends on what "blkid -t LABEL={fsname}-OST0001" > returns. It _should_ be smart enough to return the DM device, but > it is prudent to make sure of this. There shouldn''t be any problem > with mounting the Lustre filesystems by LABEL= (which is one reason we > moved to a mount-based setup). >Hi Andreas, I just checked out my local system and it''s returning a regular device name (/dev/sdh, or /dev/sdi, depending on the label) and a unique UUID, but then I didn''t enable multipath when I built the FS. If I avoid using the /dev name, is it still possible to build failover properly if I''m not using the multi-path framework? If it''s not, is a rebuild of the filesystem required in order to enable multi-path support? Sorry to keep firing questions at you, I''m trying to make sure I''ve got all the bases covered for failover. thanks again, Klaus
What I''ve done on our multipath is to specify an alias in /etc/multipath.conf that has the wwid of the LUN and then I give it a nice name like ldiska, ldiskb, etc and since I was having some troubles with _netdev and then multipath didn''t settle before kpartx ran with udev, I created an init script that makes sure kpartx creates the dev devices for the LUNs at /dev/mapper/ldisk[ab]. Then heartbeat mounts the Lustre volumes and we are good to go. May be clunky, but it works, the key is use multipath to create a nice alias, then use that and not the volume label, multipath will make sure there is a good path to the volume. Robert On 10/25/07 2:37 PM, "Klaus Steden" <klaus.steden at thomson.net> wrote:> >> That depends... It depends on what "blkid -t LABEL={fsname}-OST0001" >> returns. It _should_ be smart enough to return the DM device, but >> it is prudent to make sure of this. There shouldn''t be any problem >> with mounting the Lustre filesystems by LABEL= (which is one reason we >> moved to a mount-based setup). >> > Hi Andreas, > > I just checked out my local system and it''s returning a regular device name > (/dev/sdh, or /dev/sdi, depending on the label) and a unique UUID, but then > I didn''t enable multipath when I built the FS. > > If I avoid using the /dev name, is it still possible to build failover > properly if I''m not using the multi-path framework? If it''s not, is a > rebuild of the filesystem required in order to enable multi-path support? > > Sorry to keep firing questions at you, I''m trying to make sure I''ve got all > the bases covered for failover. > > thanks again, > Klaus > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >Robert LeBlanc College of Life Sciences Computer Support Brigham Young University leblanc at byu.edu (801)422-1882
On Oct 25, 2007 13:37 -0700, Klaus Steden wrote:> > > That depends... It depends on what "blkid -t LABEL={fsname}-OST0001" > > returns. It _should_ be smart enough to return the DM device, but > > it is prudent to make sure of this. There shouldn''t be any problem > > with mounting the Lustre filesystems by LABEL= (which is one reason we > > moved to a mount-based setup). > > I just checked out my local system and it''s returning a regular device name > (/dev/sdh, or /dev/sdi, depending on the label) and a unique UUID, but then > I didn''t enable multipath when I built the FS.If those devices (sdh, sdi) are actually part of the multipath device, then you should NOT use mount-by-label, because the "blkid" tool is using the same mechanism for locating devices as "mount" will be.> If I avoid using the /dev name, is it still possible to build failover > properly if I''m not using the multi-path framework? If it''s not, is a > rebuild of the filesystem required in order to enable multi-path support?The use of Lustre failover is completely orthogonal to multipath. The multipath is a local-node mechanism, and failover is between different nodes. As for changing the mount device - this should work, but you might need to fix up the lustre config on the OST/MDT. Cliff? Cheers, Andreas -- Andreas Dilger Sr. Software Engineer, Lustre Group Sun Microsystems of Canada, Inc.
Is kpartx destructive? I''ve already got a live file system constructed, and I need to know ahead of time if I''ve need to restore if I make configuration changes, or if I''m risking corruption during setup. Klaus On 10/25/07 1:59 PM, "Robert LeBlanc" <robert at leblancnet.us>did etch on stone tablets:> What I''ve done on our multipath is to specify an alias in > /etc/multipath.conf that has the wwid of the LUN and then I give it a nice > name like ldiska, ldiskb, etc and since I was having some troubles with > _netdev and then multipath didn''t settle before kpartx ran with udev, I > created an init script that makes sure kpartx creates the dev devices for > the LUNs at /dev/mapper/ldisk[ab]. Then heartbeat mounts the Lustre volumes > and we are good to go. May be clunky, but it works, the key is use multipath > to create a nice alias, then use that and not the volume label, multipath > will make sure there is a good path to the volume. > > Robert > > > On 10/25/07 2:37 PM, "Klaus Steden" <klaus.steden at thomson.net> wrote: > >> >>> That depends... It depends on what "blkid -t LABEL={fsname}-OST0001" >>> returns. It _should_ be smart enough to return the DM device, but >>> it is prudent to make sure of this. There shouldn''t be any problem >>> with mounting the Lustre filesystems by LABEL= (which is one reason we >>> moved to a mount-based setup). >>> >> Hi Andreas, >> >> I just checked out my local system and it''s returning a regular device name >> (/dev/sdh, or /dev/sdi, depending on the label) and a unique UUID, but then >> I didn''t enable multipath when I built the FS. >> >> If I avoid using the /dev name, is it still possible to build failover >> properly if I''m not using the multi-path framework? If it''s not, is a >> rebuild of the filesystem required in order to enable multi-path support? >> >> Sorry to keep firing questions at you, I''m trying to make sure I''ve got all >> the bases covered for failover. >> >> thanks again, >> Klaus >> >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at clusterfs.com >> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >> > > > Robert LeBlanc > College of Life Sciences Computer Support > Brigham Young University > leblanc at byu.edu > (801)422-1882 > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
Klaus Steden wrote:>> That depends... It depends on what "blkid -t LABEL={fsname}-OST0001" >> returns. It _should_ be smart enough to return the DM device, but >> it is prudent to make sure of this. There shouldn''t be any problem >> with mounting the Lustre filesystems by LABEL= (which is one reason we >> moved to a mount-based setup). >> > Hi Andreas, > > I just checked out my local system and it''s returning a regular device name > (/dev/sdh, or /dev/sdi, depending on the label) and a unique UUID, but then > I didn''t enable multipath when I built the FS. > > If I avoid using the /dev name, is it still possible to build failover > properly if I''m not using the multi-path framework? If it''s not, is a > rebuild of the filesystem required in order to enable multi-path support?Hmm. Let''s be clear here. You are talking about node failover, which would be managed by some sort of HA framework (Heartbeat, etc) For node failure, again you only have to be sure the disk is reachable by both nodes (shared) and that each node has a consistent method for identifying the shared spindle. Mulitpath shouldn''t matter to Heartbeat, it will use the Lustre mount command which supports labels. cliffw> > Sorry to keep firing questions at you, I''m trying to make sure I''ve got all > the bases covered for failover. > > thanks again, > Klaus > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss