Lustre discuss - Jul 2006 - Lustre delay in the begining of file IO access

I am facing an issue with lustre 1.4.6.2. When i access a big file I/ 
O access then it starts very slow and then the access speeds up.

dd if=bigfile of=/dev/null bs=1024k count=100

takes 15 seconds but if the count is big then the data transfer  
speeds up after sometime to > 200 MB/s

What might be the issue. The storage backend is DDN S2A9500 with 4  
OSS and 1 MDS.

Thanks

anand

Hi Anand,

There could be multiple reasons.  For example, you may be warming up a
RAID controller cache, or your file may be fragmented.

Have a look at the file:

/proc/fs/lustre/osc/*/rpc_stats
/proc/fs/lustre/obdfilter/*/brw_stats

for each OSC/obdfilter you are using on the clients and servers. These
files should show you what the transfer sizes were during the job.  

Typically Lustre gets best results when a very large part of the
transfers, from server disks to client, is >=1MB.  

If Lustre does show this kind of transfer sizes, then you should perhaps
first measure if your DDN is doing such transfers also - they have their
own statistics available.

- Peter -





 > -----Original Message-----
 > From: lustre-discuss-bounces@clusterfs.com 
 > [mailto:lustre-discuss-bounces@clusterfs.com] On Behalf Of 
 > Anand Bisen
 > Sent: Monday, July 10, 2006 10:23 PM
 > To: lustre-discuss@clusterfs.com
 > Subject: [Lustre-discuss] Lustre delay in the begining of 
 > file IO access
 > 
 > I am facing an issue with lustre 1.4.6.2. When i access a 
 > big file I/ O access then it starts very slow and then the 
 > access speeds up.
 > 
 > dd if=bigfile of=/dev/null bs=1024k count=100
 > 
 > takes 15 seconds but if the count is big then the data 
 > transfer speeds up after sometime to > 200 MB/s
 > 
 > What might be the issue. The storage backend is DDN S2A9500 
 > with 4 OSS and 1 MDS.
 > 
 > Thanks
 > 
 > anand
 > 
 > _______________________________________________
 > Lustre-discuss mailing list
 > Lustre-discuss@clusterfs.com
 > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
 > 
 >

On Jul 10, 2006  21:23 -0700, Anand Bisen wrote:
> I am facing an issue with lustre 1.4.6.2. When i access a big file I/ 
> O access then it starts very slow and then the access speeds up.
> 
> dd if=bigfile of=/dev/null bs=1024k count=100
> 
> takes 15 seconds but if the count is big then the data transfer  
> speeds up after sometime to > 200 MB/s
Is this the first acceess for any file, or just the first file accessed
after the filesystem is mounted?

There was a fix made to the readahead code for the upcoming 1.4.7 release
that improved the initial file read performance.  Could you please retest
with the latest 1.4.7 beta to see if this problem has been fixed?

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.