What''s the recommended way of getting all my OSTs and things mounted at boot time of the machines which make up the cluster fs? I''m commissioning a test cluster fs for my engineering group to use, and I want everything to start up by itself in case of power failures, random reboots etc. Is there any reason not to put stuff in fstab? Or am I better off to put it all in a script which can be run later on? Or something else? How are y''all handling this? Thanks in advance...
On Wed, 30 Aug 2006, jrd@jrd.org wrote:> What''s the recommended way of getting all my OSTs and things mounted at boot > time of the machines which make up the cluster fs? > > I''m commissioning a test cluster fs for my engineering group to use, and I > want everything to start up by itself in case of power failures, random > reboots etc. Is there any reason not to put stuff in fstab? Or am I better > off to put it all in a script which can be run later on? Or something else? > How are y''all handling this?I''m assuming you''re talking about Lustre 1.6 here? Since you must have networking up you may or may not be able to have stuff mounted automatically in the traditional mount-at-boot-way. I''d go for the following: * Put stuff in fstab using the noauto mount option * Use a startup script that''s run after networking etc. is set up. The recommended way seems to be to mount by label to not be burned by changed device numbers when devices are missing, but since e2label can''t read the labels of lustre ost devices on Ubuntu Dapper I''m not entirely happy with this. /Nikke -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke@hpc2n.umu.se --------------------------------------------------------------------------- Black holes resulted when MS tried to beat a deadline =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
On Aug 31, 2006 09:21 +0200, Niklas Edmundsson wrote:> The recommended way seems to be to mount by label to not be burned by > changed device numbers when devices are missing, but since e2label > can''t read the labels of lustre ost devices on Ubuntu Dapper I''m not > entirely happy with this.That seems exceedingly odd. Is it because the e2fsprogs do not understand the EXTENTS feature flag and are simply refusing to open the filesystem? You should upgrade to the CFS e2fsprogs-1.39. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
On Thu, 31 Aug 2006, Andreas Dilger wrote:> On Aug 31, 2006 09:21 +0200, Niklas Edmundsson wrote: >> The recommended way seems to be to mount by label to not be burned by >> changed device numbers when devices are missing, but since e2label >> can''t read the labels of lustre ost devices on Ubuntu Dapper I''m not >> entirely happy with this. > > That seems exceedingly odd. Is it because the e2fsprogs do not understand > the EXTENTS feature flag and are simply refusing to open the filesystem? > You should upgrade to the CFS e2fsprogs-1.39.Ah. Ubuntu Dapper has e2fsprogs 1.38-2ubuntu2 which says: # e2label /dev/sdb e2label: Filesystem has unsupported feature(s) while trying to open /dev/sdb Couldn''t find valid filesystem superblock. I''ll look into installing the CFS 1.39 then. /Nikke -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke@hpc2n.umu.se --------------------------------------------------------------------------- Adam''s Rib: The original bone of contention. =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
From: Niklas Edmundsson <Niklas.Edmundsson@hpc2n.umu.se>
Date: Thu, 31 Aug 2006 09:21:49 +0200 (MEST)
[...]
I''m assuming you''re talking about Lustre 1.6 here?
Sorry, yes.
Since you must have networking up you may or may not be able to have
stuff mounted automatically in the traditional mount-at-boot-way.
I''d go for the following:
* Put stuff in fstab using the noauto mount option
* Use a startup script that''s run after networking etc. is set up.
Ok, sounds plausible.
The recommended way seems to be to mount by label to not be burned by
changed device numbers when devices are missing,
Yeah, I''m not too concerned about that problem. This isn''t
meant to
be a production system, it''s just going to be beaten on by a horde of
engineers doing testing. I''m really mostly trying avoid the issue of
everybody coming and asking me how to restart the filesystem by hand
when they''ve decided to restart the systems for one reason or another.
Thanks...
John R. Dunning wrote:> From: Niklas Edmundsson <Niklas.Edmundsson@hpc2n.umu.se> > Date: Thu, 31 Aug 2006 09:21:49 +0200 (MEST) > [...] > > I''m assuming you''re talking about Lustre 1.6 here? > > Sorry, yes. > > Since you must have networking up you may or may not be able to have > stuff mounted automatically in the traditional mount-at-boot-way. >Exactly, depending on your networking setup. I''ve had no problem with starting a TCP-based Lustre directly from fstab, but other network types might start late in the game.> > I''d go for the following: > * Put stuff in fstab using the noauto mount option > * Use a startup script that''s run after networking etc. is set up. > > Ok, sounds plausible. >Indeed. This is where your HA software would determine if this node is the correct one to start the service on in a failover situation.
Thus spake Andreas Dilger:> On Aug 31, 2006 09:21 +0200, Niklas Edmundsson wrote: > > The recommended way seems to be to mount by label to not be burned by > > changed device numbers when devices are missing, but since e2label > > can''t read the labels of lustre ost devices on Ubuntu Dapper I''m not > > entirely happy with this. > > That seems exceedingly odd. Is it because the e2fsprogs do not understand > the EXTENTS feature flag and are simply refusing to open the filesystem? > You should upgrade to the CFS e2fsprogs-1.39.Hi, I''m new to lustre, but not to linux or filesystems. Is there any reason why someone would not recommend using autofs to mount lustre filesystesm. It seems to be acceptable for usb-devices to nfs-mounts and everything in between. Jeremy -- Jeremy S. Brand JBrand@lbl.gov +1.925.296.5686 Unix Systems Administrator - U.S. Department of Energy Joint Genome Institute http://www.jgi.doe.gov Lawrence Berkeley National Lab http://www.lbl.gov PGP Fingerprint 1F4F 6404 9D23 D71F E6B4 51BF 2834 93CD C537 954C
From: Nathaniel Rutman <nathan@clusterfs.com>
Date: Thu, 31 Aug 2006 08:48:23 -0700
Exactly, depending on your networking setup. I''ve had no problem
with
starting a
TCP-based Lustre directly from fstab, but other network types might
start late in the game.
Ah, I think I may have been more confusing than necessary. I wasn''t
talking about clients mounting the lustre fs on those servers over
there, I was talking about the servers mounting the mdt and ost
pieces as they''re coming up.
Though the other is interesting as well, as when my users start using
this thing, they will likely want to stick it in their fstabs.
Indeed. This is where your HA software would determine if this node is
the correct one
to start the service on in a failover situation.
In this case, there is no HA software involved, but I understand the
point. When we get closer to real deployment, I''m sure that''s
what
we''ll be using.
On Thu, 31 Aug 2006, John R. Dunning wrote:> From: Nathaniel Rutman <nathan@clusterfs.com> > Date: Thu, 31 Aug 2006 08:48:23 -0700 > > Exactly, depending on your networking setup. I''ve had no problem with > starting a > TCP-based Lustre directly from fstab, but other network types might > start late in the game. > > Ah, I think I may have been more confusing than necessary. I wasn''t > talking about clients mounting the lustre fs on those servers over > there, I was talking about the servers mounting the mdt and ost > pieces as they''re coming up.Even on servers, you want to start Lustre after the network has been configured, since LNET (the Luster networking layer) needs to discover which network interfaces can be used (search for "LNI" in kernel logs). By the way, you may find it useful to use the "_netdev" option in fstab, on some distributions this lets the boot scripts postpone the mount until after the network is configured (see mount(8) for details). I think it works on RHEL and Debian. -- Jean-Marc Saffroy - jean-marc.saffroy@ext.bull.net
John,> I''d go for the following: > * Put stuff in fstab using the noauto mount option > * Use a startup script that''s run after networking etc. is set up. > > Ok, sounds plausible.The Global Filesystem (GFS) from Redhat does something similar as it needs networking up for locking. They put the filesystem in /etc/fstab but use and init.d script to actually do the mounting long after the network and other cluster apps have initialised. Perhaps CFS should still supply a basic init.d script with Lustre 1.6 to do this (or someone wants to contribute one)? I was going to write one for use with Redhat''s Cluster Suite and include a "status" option which looks at /proc/fs/lustre/health_check which can be useful for monitoring a bad lustre server that hasn''t lost its network completely. I beleieve this was present in the init.d script for Lustre 1.4.x. Regards, Daire