Lustre discuss - Mar 2007 - again unexpected "no space left on device"

Hi,

I have lustre 1.5.95 running with 3 ost on one machine and and mds and mdt on 
another.
It works great, but the problem is, that I have to produce a lot of root 
(root.cern.ch)  trees and this files start with 0 size and grow up to 2 GB.

Unfortunately these files are created more likely on the ost which is more or 
less full then on the other 2 (one is less then 60% full).

Therefore I can not use the full disk space.
The easiest way to do so is if one would be able to give a tune parameter 
where new files are never created on ost with less then 2 GB free disk space.

Is this possible?

In an older discussion I found the hint, that I just can deactivate the
device on mds and then just no new file can be created but read access is 
still possible.
Is this also possible if mds and mdt is on the same file system?

Are there better solutions?

Many thanks in advance
Harald

On Mar 05, 2007  15:41 +0100, Harald van Pee wrote:
> I have lustre 1.5.95 running with 3 ost on one machine and and mds and mdt
on
> another.
> 
> Unfortunately these files are created more likely on the ost which is more
or
> less full then on the other 2 (one is less then 60% full).
> 
> Therefore I can not use the full disk space.
This is a bug in the free space management in the older 1.5.x betas that
was fixed recently.  You should try out 1.5.97 and it should prefer to
allocate new files on the least full OST.

One caveat - this decision is made at the time the file is first opened.
If you create a lot of small files, say with OST1 nearly full and OST3
nearly empty then a lot more files will be created on OST3.  If these
files grow slowly over time then it may be that OST1 becomes overly full
(because of the many more files created there) when it was less full.
> The easiest way to do so is if one would be able to give a tune parameter 
> where new files are never created on ost with less then 2 GB free disk
space.
That is already done to some extent - when only 0.1% of the filesystem
space is left the OST will not allow new allocations there.  There is
no tunable for this, though almost anyone could implement this quite
easily.  Add a new tunable (obdfilter/lproc_obdfilter.c) and use it for
the check in filter_precreate() (obdfilter/filter.c).

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

On Monday 05 March 2007 04:13 pm, Andreas Dilger wrote:
> On Mar 05, 2007  15:41 +0100, Harald van Pee wrote:
> > I have lustre 1.5.95 running with 3 ost on one machine and and mds and
> > mdt on another.
> >
> > Unfortunately these files are created more likely on the ost which is
> > more or less full then on the other 2 (one is less then 60% full).
> >
> > Therefore I can not use the full disk space.
>
> This is a bug in the free space management in the older 1.5.x betas that
> was fixed recently.  You should try out 1.5.97 and it should prefer to
> allocate new files on the least full OST.
o.k. thats nice to here.
I will try to update as soon as possible, but not before the end of next week.
In the meanwhile I have switched off the full osc, which seems to work without 
any problems.
>
> One caveat - this decision is made at the time the file is first opened.
> If you create a lot of small files, say with OST1 nearly full and OST3
> nearly empty then a lot more files will be created on OST3.  If these
> files grow slowly over time then it may be that OST1 becomes overly full
> (because of the many more files created there) when it was less full.
this can be a problem for our production cluster, but of course only if we run 
out of diskspace. 
But still the situation will be much improved, at the moment I have 160 GB 
free on one ost0, 70 GB on ost2 and just 100 MB on ost1, and the files will 
be created at least as often on ost1 as on the other two.