Lustre devel - Aug 2009 - Role of the Metadata Server during File I/O

One of Lustre''s key innovations seems to be the separation of metadata
from
file data.  According to Sun, "Lustre file operations bypass the MetaData 
server completely and fully utilize the parallel data paths to all OSSs in 
the cluster."  (See 
http://www.sun.com/software/products/lustre/features.xml) 

However, can this really be the case? 

In POSIX-compliant file I/O, a call to write() that starts at an offset 
which is greater than the file size must write zeroes to disk to make up for 
the missing space.  So if you have a file size of 0 bytes, and then you 
write a single byte at offset 10, bytes 0 through 9 of the file will contain 
zeros. 

But if a file on a Lustre system is striped across multiple OSTs, how does 
Lustre avoid communicating with the Metadata Server at every write 
operation?  Consider the following scenario: 

You have 4 OSTs and you create a new file and stripe it across all 4 OSTs, 
and you set the stripe size to 4 bytes.  (I know that is too small but
I''m
just keeping this simple.) 

Now, suppose you call write() and write 1 byte to the file at offset 5.  
Lustre must now write 4 zero bytes on the first OST, and 1 non-zero byte on 
the second OST.  But in order to know that it is necessary to write zeroes 
to the first OST, the client would need access to global information about 
the total size of the file.  Therefore, wouldn''t it need to check with
the
Metadata Server to determine the total file size before every call to 
write()? 

Any information anyone can provide me on the implementation 
details/strategies used here would be greatly appreciated. 

 -Charles Salvia
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-devel/attachments/20090810/dd95b56f/attachment.html

The file size is computed from the sizes of all its stripes to avoid

unnecessary communication with the MDS while a file is being

extended.

 

Obviously the next question is ?does that mean you have to consult all

the OSTs whenever you stat a file?? and currently the answer is yes.

However we are developing a new feature to allow Lustre to cache the

sizes of inactive files on the MDS to give us the best of both worlds.

Cheers, 
                   Eric 

 

From: lustre-devel-bounces at lists.lustre.org [mailto:lustre-devel-bounces at
lists.lustre.org] On Behalf Of office at hailoo.com
Sent: 11 August 2009 12:32 AM
To: lustre-devel at lists.lustre.org
Subject: [Lustre-devel] Role of the Metadata Server during File I/O

 

One of Lustre''s key innovations seems to be the separation of metadata
from file data.  According to Sun, "Lustre file operations bypass the
MetaData server completely and fully utilize the parallel data paths to all OSSs
in the cluster."  (See
http://www.sun.com/software/products/lustre/features.xml)

However, can this really be the case? 

In POSIX-compliant file I/O, a call to write() that starts at an offset which is
greater than the file size must write zeroes to disk to make up for the missing
space.  So if you have a file size of 0 bytes, and then you write a single byte
at offset 10, bytes 0 through 9 of the file will contain zeros.

But if a file on a Lustre system is striped across multiple OSTs, how does
Lustre avoid communicating with the Metadata Server at every write operation? 
Consider the following scenario:

You have 4 OSTs and you create a new file and stripe it across all 4 OSTs, and
you set the stripe size to 4 bytes.  (I know that is too small but I''m
just keeping this simple.)

Now, suppose you call write() and write 1 byte to the file at offset 5.  Lustre
must now write 4 zero bytes on the first OST, and 1 non-zero byte on the second
OST.  But in order to know that it is necessary to write zeroes to the first
OST, the client would need access to global information about the total size of
the file.  Therefore, wouldn''t it need to check with the Metadata
Server to determine the total file size before every call to write()?

Any information anyone can provide me on the implementation details/strategies
used here would be greatly appreciated.

-Charles Salvia 

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-devel/attachments/20090811/38e6b897/attachment.html

Thanks for the information.  Doesn''t this entail that for every call to
write() involving a striped file, Lustre must necessarily consult all OSSs, 
in order to determine 1) the file size and 2) if the current write operation 
will extend the file? 

If so, then I think it would be advisable for Lustre clients working with 
striped files to avoid small unbuffered file writes, since every call to 
write() requires a lot of network overhead. 

Eric Barton writes: 
> The file size is computed from the sizes of all its stripes to avoid 
> 
> unnecessary communication with the MDS while a file is being 
> 
> extended. 
> 
>   
> 
> Obviously the next question is ?does that mean you have to consult all 
> 
> the OSTs whenever you stat a file?? and currently the answer is yes. 
> 
> However we are developing a new feature to allow Lustre to cache the 
> 
> sizes of inactive files on the MDS to give us the best of both worlds. 
> 
> Cheers, 
>                    Eric  
> 
>   
> 
> From: lustre-devel-bounces at lists.lustre.org [mailto:lustre-devel-bounces
at lists.lustre.org] On Behalf Of office at hailoo.com
> Sent: 11 August 2009 12:32 AM
> To: lustre-devel at lists.lustre.org
> Subject: [Lustre-devel] Role of the Metadata Server during File I/O 
> 
>   
> 
> One of Lustre''s key innovations seems to be the separation of
metadata from file data.  According to Sun, "Lustre file operations bypass
the MetaData server completely and fully utilize the parallel data paths to all
OSSs in the cluster."  (See
http://www.sun.com/software/products/lustre/features.xml)
> 
> However, can this really be the case?  
> 
> In POSIX-compliant file I/O, a call to write() that starts at an offset
which is greater than the file size must write zeroes to disk to make up for the
missing space.  So if you have a file size of 0 bytes, and then you write a
single byte at offset 10, bytes 0 through 9 of the file will contain zeros.
> 
> But if a file on a Lustre system is striped across multiple OSTs, how does
Lustre avoid communicating with the Metadata Server at every write operation? 
Consider the following scenario:
> 
> You have 4 OSTs and you create a new file and stripe it across all 4 OSTs,
and you set the stripe size to 4 bytes.  (I know that is too small but
I''m just keeping this simple.)
> 
> Now, suppose you call write() and write 1 byte to the file at offset 5. 
Lustre must now write 4 zero bytes on the first OST, and 1 non-zero byte on the
second OST.  But in order to know that it is necessary to write zeroes to the
first OST, the client would need access to global information about the total
size of the file.  Therefore, wouldn''t it need to check with the
Metadata Server to determine the total file size before every call to write()?
> 
> Any information anyone can provide me on the implementation
details/strategies used here would be greatly appreciated.
> 
> -Charles Salvia  
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-devel/attachments/20090811/9c3100c1/attachment.html

Hello!

On Aug 11, 2009, at 1:24 PM, office at hailoo.com wrote:
> Thanks for the information.  Doesn''t this entail that for every
call
> to write() involving a striped file, Lustre must necessarily consult  
> all OSSs, in order to determine 1) the file size and 2) if the  
> current write operation will extend the file?
Why does write need to know the entire file size? We only care if we  
are extending currently accessed stripe. We know this the moment we  
obtained the lock on
the stripe region we are interested in.
> If so, then I think it would be advisable for Lustre clients working  
> with striped files to avoid small unbuffered file writes, since  
> every call to write() requires a lot of network overhead.
>
It is indeed not advisable to use small unbuffered writes (though we  
relieved the situation somewhat), but not because of the network  
overhead, rather the lustre
code is not exactly light-weight doing thelock processing and lookup  
and stuff.
I remember when LLNL performed experiments, and 4 byte unbuffered  
writes were 100 times slower than 1M writes.
But since then we improved the situation and I am no longer sure how  
these compare.

Bye,
     Oleg

Oleg Drokin writes: 
> Hello! 
> 
> On Aug 11, 2009, at 1:24 PM, office at hailoo.com wrote: 
> 
>> Thanks for the information.  Doesn''t this entail that for
every call  to
>> write() involving a striped file, Lustre must necessarily consult  all 
>> OSSs, in order to determine 1) the file size and 2) if the  current
write
>> operation will extend the file?
> 
> Why does write need to know the entire file size? We only care if we  are 
> extending currently accessed stripe. We know this the moment we  obtained 
> the lock on
> the stripe region we are interested in.
Consider the following situation: 

You have 4 OSTs and you create a file striped across all 4 OSTs, and you set 
the stripe size to 4 bytes.  (Obviously that is too small, but I just want 
to keep this simple.) 

The file is created and it starts out as a 0 byte file.  Now, suppose you 
write one byte to offset 5.  So now Lustre has to write one byte to the 
second OST.  But, in POSIX compliant file I/O, if you write to an offset 
that is greater than the file size, the file system must write zeros to the 
disk to fill the gap between the old end of the file and the offset.  So, in 
the case of Lustre, the system must not only write a single byte to the 
second OST, it also must write 4 zero-bytes to the first OST.  But in order 
to even know that is has to do this, wouldn''t Lustre need to know the
entire
file size?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-devel/attachments/20090813/d2a7cfd9/attachment.html

Hello!

On Aug 13, 2009, at 2:43 PM, office at hailoo.com wrote:
>>> Thanks for the information.  Doesn''t this entail that for
every
>>> call  to write() involving a striped file, Lustre must necessarily
>>> consult  all OSSs, in order to determine 1) the file size and 2)  
>>> if the  current write operation will extend the file?
>> Why does write need to know the entire file size? We only care if  
>> we  are extending currently accessed stripe. We know this the  
>> moment we  obtained the lock on
>> the stripe region we are interested in.
>>
> Consider the following situation:
> You have 4 OSTs and you create a file striped across all 4 OSTs, and  
> you set the stripe size to 4 bytes.  (Obviously that is too small,  
> but I just want to keep this simple.)
> The file is created and it starts out as a 0 byte file.  Now,  
> suppose you write one byte to offset 5.  So now Lustre has to write  
> one byte to the second OST.  But, in POSIX compliant file I/O, if  
> you write to an offset that is greater than the file size, the file  
> system must write zeros to the disk to fill the gap between the old  
> end of the file
>
This is a misconception. Nowhere does it says we must write zeroes to  
disk. We are fine as long as subsequent reads would get
zeroes.
> and the offset.  So, in the case of Lustre, the system must not only  
> write a single byte to the second OST, it also must write 4 zero- 
> bytes to the first OST.  But in order to even know that is has to do  
> this, wouldn''t Lustre need to know the entire file size?
>
No.
What happens is we write the data and necessary zeroes to 2nd OST (to  
fill the page where this one byte fits) and size on 1st ost remains 0,
lustre is smart enough to fill the next read with zeroes when it  
encounters access to that part of the file.
The file size on the other hand would be properly composed because we  
look at file size at every OST.

Bye,
     Oleg