For whatever reason, searching my lustre mount (ls -R or find), compiling code and other operations involving lots of small files are painfully slow. There is no load on the filesystem other than my various tests. I''ve disabled lnet debugging. Just to give you an idea of how slow it is-- a ./configure of this particular code on a local filesystem takes less than a minute. On lustre it''s been running for five minutes and is hardly half way through. An untar on the local filesystem takes .9 seconds while that same untar takes 12 seconds to our lustre mount. Any ideas for improving this? Aaron Knister Associate Systems Analyst Center for Ocean-Land-Atmosphere Studies (301) 595-7000 aaron at iges.org
On Fri, 2008-01-04 at 09:44 -0500, Aaron Knister wrote:> For whatever reason, searching my lustre mount (ls -R or find), > compiling code and other operations involving lots of small files are > painfully slow. There is no load on the filesystem other than my > various tests. I''ve disabled lnet debugging. Just to give you an idea > of how slow it is-- a ./configure of this particular code on a local > filesystem takes less than a minute. On lustre it''s been running for > five minutes and is hardly half way through. An untar on the local > filesystem takes .9 seconds while that same untar takes 12 seconds to > our lustre mount. Any ideas for improving this?Perhaps bug 14010 is relevant. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080104/e0a95908/attachment-0002.bin
That looks about right. Is there an ETA on it? Sadly we may have to switch file systems if we can''t get the performance of small i/o to at least NFS speeds. On Jan 4, 2008, at 9:55 AM, Brian J. Murrell wrote:> On Fri, 2008-01-04 at 09:44 -0500, Aaron Knister wrote: >> For whatever reason, searching my lustre mount (ls -R or find), >> compiling code and other operations involving lots of small files are >> painfully slow. There is no load on the filesystem other than my >> various tests. I''ve disabled lnet debugging. Just to give you an idea >> of how slow it is-- a ./configure of this particular code on a local >> filesystem takes less than a minute. On lustre it''s been running for >> five minutes and is hardly half way through. An untar on the local >> filesystem takes .9 seconds while that same untar takes 12 seconds to >> our lustre mount. Any ideas for improving this? > > Perhaps bug 14010 is relevant. > > b. > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discussAaron Knister Associate Systems Analyst Center for Ocean-Land-Atmosphere Studies (301) 595-7000 aaron at iges.org
On Fri, 2008-01-04 at 10:13 -0500, Aaron Knister wrote:> That looks about right. Is there an ETA on it?I don''t know of any. You could CC yourself on the bug and ask the status. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080104/1743106e/attachment-0002.bin
On Fri, Jan 04, 2008 at 09:44:54AM -0500, Aaron Knister wrote:>For whatever reason, searching my lustre mount (ls -R or find), >compiling code and other operations involving lots of small files are >painfully slow. There is no load on the filesystem other than my >various tests. I''ve disabled lnet debugging. Just to give you an idea >of how slow it is-- a ./configure of this particular code on a local >filesystem takes less than a minute. On lustre it''s been running for >five minutes and is hardly half way through. An untar on the local >filesystem takes .9 seconds while that same untar takes 12 seconds to >our lustre mount. Any ideas for improving this?do you have striping turned off? that makes a massive difference for metadata operations... lfs setstripe -d /some/lustre/dir/ cheers, robin
Striping is turned off. Are there any other optimizations you know of to increase the speed of metadata operations? On Jan 5, 2008, at 3:50 AM, Robin Humble wrote:> On Fri, Jan 04, 2008 at 09:44:54AM -0500, Aaron Knister wrote: >> For whatever reason, searching my lustre mount (ls -R or find), >> compiling code and other operations involving lots of small files are >> painfully slow. There is no load on the filesystem other than my >> various tests. I''ve disabled lnet debugging. Just to give you an idea >> of how slow it is-- a ./configure of this particular code on a local >> filesystem takes less than a minute. On lustre it''s been running for >> five minutes and is hardly half way through. An untar on the local >> filesystem takes .9 seconds while that same untar takes 12 seconds to >> our lustre mount. Any ideas for improving this? > > do you have striping turned off? > that makes a massive difference for metadata operations... > lfs setstripe -d /some/lustre/dir/ > > cheers, > robinAaron Knister Associate Systems Analyst Center for Ocean-Land-Atmosphere Studies (301) 595-7000 aaron at iges.org
Aaron Knister wrote:> Striping is turned off. Are there any other optimizations you know of > to increase the speed of metadata operations?Not metadata specific, but increasing /proc/fs/lustre/ldlm/namespaces/*/lru_size on the affected clients will help in some cases. This parameter controls locks cached on the client. Increasing it should help your compiles. This parameter can be set per-client, you do not have to alter server-side. cliffw> > On Jan 5, 2008, at 3:50 AM, Robin Humble wrote: > >> On Fri, Jan 04, 2008 at 09:44:54AM -0500, Aaron Knister wrote: >>> For whatever reason, searching my lustre mount (ls -R or find), >>> compiling code and other operations involving lots of small files are >>> painfully slow. There is no load on the filesystem other than my >>> various tests. I''ve disabled lnet debugging. Just to give you an idea >>> of how slow it is-- a ./configure of this particular code on a local >>> filesystem takes less than a minute. On lustre it''s been running for >>> five minutes and is hardly half way through. An untar on the local >>> filesystem takes .9 seconds while that same untar takes 12 seconds to >>> our lustre mount. Any ideas for improving this? >> do you have striping turned off? >> that makes a massive difference for metadata operations... >> lfs setstripe -d /some/lustre/dir/ >> >> cheers, >> robin > > Aaron Knister > Associate Systems Analyst > Center for Ocean-Land-Atmosphere Studies > > (301) 595-7000 > aaron at iges.org > > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
On Sat, Jan 05, 2008 at 11:08:10AM -0500, Aaron Knister wrote:> Striping is turned off. Are there any other optimizations you know of to > increase the speed of metadata operations?having blindingly fast disks as the metadata backing store improves performance greatly. Lustre''s MDS also benefits from as many fast cores as you can throw at it. as a temporary measure you could put the metadata disk in a ramdisk to see if that makes a difference to your tests. eg. mkdir -p /mnt/ramdisk mount -t ramfs none -o rw,size=3072M,mode=755 /mnt/ramdisk dd if=/dev/zero of=/mnt/ramdisk/mds bs=1M count=3000 losetup /dev/loop0 /mnt/ramdisk/mds mkfs.lustre --fsname=testfs --mdt --mgs --mkfsoptions=''"-i 1024"'' --reformat /dev/loop0 hmmm... you say your other option is NFS - are you comparing to NFS in sync mode or async? here''s a toy Linux 2.6.23 kernel benchmark - tar xfj, make -j 6, rm -rf. times in seconds (best of at least 2 runs): Lustre NFS sync NFS async local disk loopback on Lustre tar 377 1157 22 9 9 build 719 916 378 290 291 rm 24 449 15 1 1 Note that the NFS server and the Lustre MDS and the local disk node are the same machine in this test, but the NFS is over GigE to a SAS raid1 pair whilst Lustre is over IB to many disks, so it''s not an apples-to-apples comparison by any means. It does however indicate the differences between NFS settings, and also that Lustre can be faster than NFS when they are both writing to disks and not just caching in the server''s ram - what NFS async does. caching unwritten data in the server''s ram leads to data loss if the server crashes. no global filesystem is going to be as fast as local disk on metadata operations, but the last column of the table shows a neat trick where Lustre can give you a local filesystem that''s even better than a local disk - it has all the metadata speed of a local disk, but also the bandwidth of striped Lustre :-) I created a big striped file on Lustre, and mounted it on a node as a loopback ext2 filesystem. Lustre sees no metadata traffic - just one file open on one node. we''re going to use this technique for ''local'' scratch disks on diskless nodes. cheers, robin
Hello!
Since you did not subscribe to the bug 14010, you might be
interested to know
that I have a better patch there now that you might actually try
and see how
it affects your workloads. Of course this is not the final version
yet, but
I am pretty much concentrating on that area in the short term, so
as luck has
it, we might have something good soon.
Bye,
Oleg
On Jan 4, 2008, at 10:13 AM, Aaron Knister wrote:
> That looks about right. Is there an ETA on it? Sadly we may have to
> switch file systems if we can''t get the performance of small i/o
to at
> least NFS speeds.
>
> On Jan 4, 2008, at 9:55 AM, Brian J. Murrell wrote:
>
>> On Fri, 2008-01-04 at 09:44 -0500, Aaron Knister wrote:
>>> For whatever reason, searching my lustre mount (ls -R or find),
>>> compiling code and other operations involving lots of small files
>>> are
>>> painfully slow. There is no load on the filesystem other than my
>>> various tests. I''ve disabled lnet debugging. Just to give
you an
>>> idea
>>> of how slow it is-- a ./configure of this particular code on a
local
>>> filesystem takes less than a minute. On lustre it''s been
running for
>>> five minutes and is hardly half way through. An untar on the local
>>> filesystem takes .9 seconds while that same untar takes 12 seconds
>>> to
>>> our lustre mount. Any ideas for improving this?
>>
>> Perhaps bug 14010 is relevant.
>>
>> b.
>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at clusterfs.com
>> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
>
> Aaron Knister
> Associate Systems Analyst
> Center for Ocean-Land-Atmosphere Studies
>
> (301) 595-7000
> aaron at iges.org
>
>
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at clusterfs.com
> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss