Lustre discuss - Sep 2009 - Unable to write to filesystem (device full)

Hi Everyone,

 RHEL5.3 - Lustre 1.6.7.2 - filesystem mounted as /data.

 I''ve got another odd problem this morning. Suddenly the file system
was
reporting "no space left on device" failing every write attempt. A df
showed still 1.7TB free. 

 I first tried unmounting and remounting the /data filesystem. However
on remount I got the error:

# mount /data 
mount.lustre: mount idbn1 at tcp1:/clients at /data failed: Cannot send
after transport endpoint shutdown

 I then realized top was reporting one of the Lustre threads on the MDT
at 100% CPU. So I unmounted and remounted the MDT partition on that
server, and could then remount the /data filesystem on the client node. 

 Now everything seems to be back to normal operation. Any idea what was
going on there? I''ve got absolutely nothing from any of the log files. 

Thanks for any help,
-Nick


-  
Nick Jennings
Technical Director
Creative Motion Design
www.creativemotiondesign.com 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090929/f616a210/attachment-0001.bin

Hi Again,

 I guess I spoke too soon. Looks like the MDS is full, even though it
still has plenty of space, it''s reporting that it''s reached
it''s inode
limit. Not sure how this is fixed. I am also having a hard time finding
information about this (I wish PDF text was searchable). 

# lfs df -h
UUID                     bytes      Used Available  Use% Mounted on
clients-MDT0000_UUID     35.0G      9.4G     23.6G   26% /data[MDT:0]
clients-OST0000_UUID      2.0T    319.5G      1.6T   15% /data[OST:0]

filesystem summary:       2.0T    319.5G      1.6T   15% /data

# lfs df -i
UUID                    Inodes     IUsed     IFree IUse% Mounted on
clients-MDT0000_UUID  10485760  10409292     76468   99% /data[MDT:0]
clients-OST0000_UUID 134217728   8082612 126135116    6% /data[OST:0]

filesystem summary:   10485760  10409292     76468   99% /data



I tried to grow the MDT +50G with the following:

# lvextend  -L +50G /dev/vg-localdisk/mdt 
  Extending logical volume mdt to 90.00 GB
  Logical volume mdt successfully resized

# resize2fs -p /dev/vg-localdisk/mdt 
resize2fs 1.40.11.sun1 (17-June-2008)
resize2fs: Filesystem has unsupported read-only feature(s) :-
uninit_groups


Didn''t seem to work. So I''ve got a 90G LVM slice allocated to
the MDT
which is only using 35G of that (can''t get it to grow to use the rest).
Of that 35G, 24G is still free but my inode limit is at 99% and I''m not
sure how to extend that. 

Thanks for any help.
-Nick


On Tue, 2009-09-29 at 11:47 +0200, Nick Jennings wrote:
> Hi Everyone,
> 
>  RHEL5.3 - Lustre 1.6.7.2 - filesystem mounted as /data.
> 
>  I''ve got another odd problem this morning. Suddenly the file
system was
> reporting "no space left on device" failing every write attempt.
A df
> showed still 1.7TB free. 
> 
>  I first tried unmounting and remounting the /data filesystem. However
> on remount I got the error:
> 
> # mount /data 
> mount.lustre: mount idbn1 at tcp1:/clients at /data failed: Cannot send
> after transport endpoint shutdown
> 
>  I then realized top was reporting one of the Lustre threads on the MDT
> at 100% CPU. So I unmounted and remounted the MDT partition on that
> server, and could then remount the /data filesystem on the client node. 
> 
>  Now everything seems to be back to normal operation. Any idea what was
> going on there? I''ve got absolutely nothing from any of the log
files.
> 
> Thanks for any help,
> -Nick
> 
> 
> -  
> Nick Jennings
> Technical Director
> Creative Motion Design
> www.creativemotiondesign.com 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090929/5bd9ac00/attachment.bin

On Tue, 2009-09-29 at 12:40 +0200, Nick Jennings wrote:
> Hi Again,
> 
>  I guess I spoke too soon. Looks like the MDS is full, even though it
> still has plenty of space, it''s reporting that it''s
reached it''s inode
> limit. Not sure how this is fixed. I am also having a hard time finding
> information about this (I wish PDF text was searchable). 
 I had a momentary lapse and forgot I could, in fact, search PDFs :P

> # lfs df -h
> UUID                     bytes      Used Available  Use% Mounted on
> clients-MDT0000_UUID     35.0G      9.4G     23.6G   26% /data[MDT:0]
> clients-OST0000_UUID      2.0T    319.5G      1.6T   15% /data[OST:0]
> 
> filesystem summary:       2.0T    319.5G      1.6T   15% /data
> 
> # lfs df -i
> UUID                    Inodes     IUsed     IFree IUse% Mounted on
> clients-MDT0000_UUID  10485760  10409292     76468   99% /data[MDT:0]
> clients-OST0000_UUID 134217728   8082612 126135116    6% /data[OST:0]
> 
> filesystem summary:   10485760  10409292     76468   99% /data
> 
> 
> 
> I tried to grow the MDT +50G with the following:
> 
> # lvextend  -L +50G /dev/vg-localdisk/mdt 
>   Extending logical volume mdt to 90.00 GB
>   Logical volume mdt successfully resized
> 
> # resize2fs -p /dev/vg-localdisk/mdt 
> resize2fs 1.40.11.sun1 (17-June-2008)
> resize2fs: Filesystem has unsupported read-only feature(s) :-
> uninit_groups
> 
 After doing more digging it seems my only option is to migrate the MDT
to a new partition (with smaller number of bytes per inode). Now I wish
I hadn''t added that 50G to the existing MDT slice, because it seems
like
I can''t incorporate it into the MDT filesystem, and can''t
remove it from
the Logical Volume (warns of data loss and I don''t want to take that
chance).

 So I''ve got about 80G left on this server, Ideally I''d like
to use both
partitions together 80G+90G for the new MDT slice, and format it using
2096 bytes for each inode. However since I can''t seem to grow an MDT
filesystem I don''t know how I''m going to do this...

 I found some docs on migrating and MDT to a new partition a little
while ago but now can''t seem to find it. If anyone can point me in the
right direction that would be great. In the meantime I''ve removed as
many files as I could from the filesystem to free up inodes. 

 If I use half the default recommended setting of 2096 bytes per inode,
will performance degrade substantially?

cheers,
-nick
> Didn''t seem to work. So I''ve got a 90G LVM slice
allocated to the MDT
> which is only using 35G of that (can''t get it to grow to use the
rest).
> Of that 35G, 24G is still free but my inode limit is at 99% and
I''m not
> sure how to extend that. 
> 
> Thanks for any help.
> -Nick
> 
> 
> On Tue, 2009-09-29 at 11:47 +0200, Nick Jennings wrote:
> > Hi Everyone,
> > 
> >  RHEL5.3 - Lustre 1.6.7.2 - filesystem mounted as /data.
> > 
> >  I''ve got another odd problem this morning. Suddenly the file
system was
> > reporting "no space left on device" failing every write
attempt. A df
> > showed still 1.7TB free. 
> > 
> >  I first tried unmounting and remounting the /data filesystem. However
> > on remount I got the error:
> > 
> > # mount /data 
> > mount.lustre: mount idbn1 at tcp1:/clients at /data failed: Cannot
send
> > after transport endpoint shutdown
> > 
> >  I then realized top was reporting one of the Lustre threads on the
MDT
> > at 100% CPU. So I unmounted and remounted the MDT partition on that
> > server, and could then remount the /data filesystem on the client
node.
> > 
> >  Now everything seems to be back to normal operation. Any idea what
was
> > going on there? I''ve got absolutely nothing from any of the
log files.
> > 
> > Thanks for any help,
> > -Nick
> > 
> > 
> > -  
> > Nick Jennings
> > Technical Director
> > Creative Motion Design
> > www.creativemotiondesign.com 
> > 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090929/f78eb1c2/attachment.bin

On Tue, 2009-09-29 at 12:40 +0200, Nick Jennings wrote:
> Hi Again,
Hi,
>  I guess I spoke too soon. Looks like the MDS is full, even though it
> still has plenty of space, it''s reporting that it''s
reached it''s inode
> limit. Not sure how this is fixed.
Hrm.  Well, definitely, creating a new, bigger MDS is one sure fire way
to go.  This includes a backup, re-create (of the existing MDS) and
restore operation.  Be careful with it though.  It seems lots of people
have trouble with it.  Measure twice and cut once at every step.
> I am also having a hard time finding
> information about this (I wish PDF text was searchable). 
PDF is searchable.  It is in evince at least.  Why do you need to search
PDF to get information about this?
> I tried to grow the MDT +50G with the following:
> 
> # lvextend  -L +50G /dev/vg-localdisk/mdt 
>   Extending logical volume mdt to 90.00 GB
>   Logical volume mdt successfully resized
Oh, it''s on an LV?  Good.
> # resize2fs -p /dev/vg-localdisk/mdt 
> resize2fs 1.40.11.sun1 (17-June-2008)
> resize2fs: Filesystem has unsupported read-only feature(s) :-
> uninit_groups
Hrm.  I don''t recall if I''ve tried to resize a recently
created lustre
target.  I know I resized one several times not that long ago, but it''s
lineage was ancient so it wouldn''t have had new features on it like
uninit groups.

I also don''t recall if resize2fs actually adds inodes or not.
> Didn''t seem to work. So I''ve got a 90G LVM slice
allocated to the MDT
> which is only using 35G of that
Well, it doesn''t have to be 90G.  You can lvreduce it again to get back
to whatever it was before you lvextended it.  Maybe slightly bigger if
you are paranoid.
> (can''t get it to grow to use the rest).
> Of that 35G, 24G is still free but my inode limit is at 99% and
I''m not
> sure how to extend that. 
Well, short of hunting down the issue with resize2fs and uninit_groups,
you could create a new LV for a new MDT and *carefully* copy your
existing MDT to it.  Be sure to copy EAs!  This stuff is all in the
manual and have been on this list many, many times.  The archives are
your friend.

b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090929/cbca116a/attachment.bin

On Tue, 2009-09-29 at 13:30 +0200, Nick Jennings wrote:
> Now I wish
> I hadn''t added that 50G to the existing MDT slice, because it
seems like
> I can''t incorporate it into the MDT filesystem, and can''t
remove it from
> the Logical Volume (warns of data loss and I don''t want to take
that
> chance).
The data loss it''s warning about is data that would be in the portion
you are discarding in the reduce operation.  Obviously if you have not
written anything there, there is nothing to lose.  But indeed, it''s
your
call.
>  So I''ve got about 80G left on this server, Ideally I''d
like to use both
> partitions together 80G+90G for the new MDT slice, and format it using
> 2096 bytes for each inode.
2048 or 4096?
>  If I use half the default recommended setting of 2096 bytes per inode,
> will performance degrade substantially?
That depends on how well your current inode allocation is suiting your
filesystem usage.  Ideally you can analyze your current usage and
determine your optimum inode size, assuming your usage is going to
continue along the same lines.  Ultimately, for best performance, you
want to make your inodes big enough to fill a fully striped file -- at a
cost of space.  But on the other hand, if only a small percentage of
files were widely striped, you might decide that for the perhaps only
few files that are, they can suffer a bit of performance for the overall
space savings.  Again, a call you have to make yourself.

b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090929/d7612fbc/attachment.bin

On Tue, 2009-09-29 at 07:45 -0400, Brian J. Murrell
wrote:
> On Tue, 2009-09-29 at 12:40 +0200, Nick Jennings wrote:
> > Hi Again,
> 
> Hi,
> 
> >  I guess I spoke too soon. Looks like the MDS is full, even though it
> > still has plenty of space, it''s reporting that it''s
reached it''s inode
> > limit. Not sure how this is fixed.
> 
> Hrm.  Well, definitely, creating a new, bigger MDS is one sure fire way
> to go.  This includes a backup, re-create (of the existing MDS) and
> restore operation.  Be careful with it though.  It seems lots of people
> have trouble with it.  Measure twice and cut once at every step.
> 
> > I am also having a hard time finding
> > information about this (I wish PDF text was searchable). 
> 
> PDF is searchable.  It is in evince at least.  Why do you need to search
> PDF to get information about this?
 Yeah, just forgot it was there. Flashback to the old days :)

> > I tried to grow the MDT +50G with the following:
> > 
> > # lvextend  -L +50G /dev/vg-localdisk/mdt 
> >   Extending logical volume mdt to 90.00 GB
> >   Logical volume mdt successfully resized
> 
> Oh, it''s on an LV?  Good.
> 
> > # resize2fs -p /dev/vg-localdisk/mdt 
> > resize2fs 1.40.11.sun1 (17-June-2008)
> > resize2fs: Filesystem has unsupported read-only feature(s) :-
> > uninit_groups
> 
> Hrm.  I don''t recall if I''ve tried to resize a recently
created lustre
> target.  I know I resized one several times not that long ago, but
it''s
> lineage was ancient so it wouldn''t have had new features on it
like
> uninit groups.
> 
> I also don''t recall if resize2fs actually adds inodes or not.
> 
> > Didn''t seem to work. So I''ve got a 90G LVM slice
allocated to the MDT
> > which is only using 35G of that
> 
> Well, it doesn''t have to be 90G.  You can lvreduce it again to get
back
> to whatever it was before you lvextended it.  Maybe slightly bigger if
> you are paranoid.
 Ok, I will shrink it down with a little extra room just to be safe.

> > (can''t get it to grow to use the rest).
> > Of that 35G, 24G is still free but my inode limit is at 99% and
I''m not
> > sure how to extend that. 
> 
> Well, short of hunting down the issue with resize2fs and uninit_groups,
> you could create a new LV for a new MDT and *carefully* copy your
> existing MDT to it.  Be sure to copy EAs!  This stuff is all in the
> manual and have been on this list many, many times.  The archives are
> your friend.
 I''m having trouble finding concrete examples searching for "MDT
migration" "Moving MDT" and other variations. Anywhere you could
point
me to specifically? I also can''t seem to find the relevant spot in the
manual. 

Thanks for your help Brian!
-Nick

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090929/fc96d960/attachment.bin

On Tue, 2009-09-29 at 14:22 +0200, Nick Jennings wrote:
> On Tue, 2009-09-29 at 07:45 -0400, Brian J. Murrell wrote:
> > Well, short of hunting down the issue with resize2fs and
uninit_groups,
> > you could create a new LV for a new MDT and *carefully* copy your
> > existing MDT to it.  Be sure to copy EAs!  This stuff is all in the
> > manual and have been on this list many, many times.  The archives are
> > your friend.
> 
>  I''m having trouble finding concrete examples searching for
"MDT
> migration" "Moving MDT" and other variations. Anywhere you
could point
> me to specifically? I also can''t seem to find the relevant spot in
the
> manual. 
> 
 The relevant section in the manual (15.1.3.1) I can''t even get past
the
first step, when I mount the mdt at /mnt/lustre/mdt I can''t chdir to
it:

# mount
...<snip>...
/dev/vg-localdisk/mdt on /mnt/lustre/mdt type lustre (rw,noauto,_netdev)
# cd /mnt/lustre/mdt/
-bash: cd: /mnt/lustre/mdt/: Not a directory


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090929/19f9be44/attachment-0001.bin

> # mount
> ...<snip>...
> /dev/vg-localdisk/mdt on /mnt/lustre/mdt type lustre (rw,noauto,_netdev)
> # cd /mnt/lustre/mdt/
> -bash: cd: /mnt/lustre/mdt/: Not a directory
> 
Seems you didn''t mount with -t ldiskfs as described (but with -t
lustre)

The latter starts lustre services and then blocks acces to the
mountpoint.

On Sep 29, 2009  13:30 +0200, Nick Jennings wrote:
> > I tried to grow the MDT +50G with the following:
> > 
> > # lvextend  -L +50G /dev/vg-localdisk/mdt 
> >   Extending logical volume mdt to 90.00 GB
> >   Logical volume mdt successfully resized
> > 
> > # resize2fs -p /dev/vg-localdisk/mdt 
> > resize2fs 1.40.11.sun1 (17-June-2008)
> > resize2fs: Filesystem has unsupported read-only feature(s) :-
> > uninit_groups
Try with a newer release of resize2fs.

I strongly recommend making a backup of the MDT device before doing
anything like this.  Even if you don''t need it, then you have a
backup :-).

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.