On Fri, 2006-03-24 at 21:16 +0200, Oleg Drokin wrote:> Hello! > > On Fri, Mar 24, 2006 at 01:59:37PM -0500, RS RS wrote: > > Has anyone there used iometer to measure luster performance? > > I just tried this, and it failed. Running dmesg shows: > > LustreError: 4005:0:(client.c:577:ptlrpc_check_status()) @@@ type == > > PTL_RPC_MSG_ERR, err == -28 req@f67dcc00 x1067/t0 > > o4->ost2-roger_UUID@clustre5.terascala.com_UUID:28 lens 328/288 ref 2 fl > > Rpc:R/0/0 rc 0/-28 > > I opened bug 10340 on this, but if anyone can advise me, I''d appreciate it. > > If you plan to fill entire lustre with single file, you need to have that file > stripped over all OSTs (and also all of those OSTs should be of the same size > currently). > This message you are getting telling you that one of the OSTs in use is out > of space.I wonder if more-informative error messages are on the Lustre roadmap. -jwb
Jeffrey W. Baker wrote:> On Fri, 2006-03-24 at 21:16 +0200, Oleg Drokin wrote: > >>Hello! >> >>On Fri, Mar 24, 2006 at 01:59:37PM -0500, RS RS wrote: >> >>>Has anyone there used iometer to measure luster performance? >>>I just tried this, and it failed. Running dmesg shows: >>>LustreError: 4005:0:(client.c:577:ptlrpc_check_status()) @@@ type == >>>PTL_RPC_MSG_ERR, err == -28 req@f67dcc00 x1067/t0 >>>o4->ost2-roger_UUID@clustre5.terascala.com_UUID:28 lens 328/288 ref 2 fl >>>Rpc:R/0/0 rc 0/-28 >>>I opened bug 10340 on this, but if anyone can advise me, I''d appreciate it. >> >>If you plan to fill entire lustre with single file, you need to have that file >>stripped over all OSTs (and also all of those OSTs should be of the same size >>currently). >>This message you are getting telling you that one of the OSTs in use is out >>of space. > > > I wonder if more-informative error messages are on the Lustre roadmap.Yes, they are. I would have to say at this point we are ''painfully aware'' of the issue :( cliffw> > -jwb > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
----- Original Message ----- From: "Cliff White" <cliffw@clusterfs.com> To: "RS RS" <rs2006ts@hotmail.com> Cc: <lustre-discuss@clusterfs.com> Sent: Friday, March 24, 2006 1:34 PM Subject: Re: [Lustre-discuss] Anyone using iometer? --- snip ---> > #!/bin/bash > > ME=`hostname` > LDIR="/mnt/lustre" > > TESTDIR="$LDIR/$ME" > TESTFLAGS="-e -a -M -R -g 10M -q 512k -i0 -i1 -f $TESTDIR/dummyfile" > IOZONE="./iozone" > > mkdir -p $TESTDIR > $IOZONE $TESTFLAGS > $TESTDIR/results 2>&1 > -------------------------- > > cliffwCliffw, The max file size (-g 10M) may be a bit small and produce some results being cached (readers) One might want to use a max file size that is greater than the amount of RAM in the Lustre client. The writer results should be fine, as the -e will force the writes to storage. Enjoy Don Capps
> >If you plan to fill entire lustre with single file, you need to have that >file >stripped over all OSTs (and also all of those OSTs should be of the same >size >currently). >This message you are getting telling you that one of the OSTs in use is out >of space. > >Bye, > OlegOleg, Your advice on striping got me a lot further. I set striping on the parent directory, and I was able to get much further. However, the test is still failing. Here is why: The ouput from df is: Filesystem 1K-blocks Used Available Use% Mounted on multilustre 3120840 2950532 12832 100% /mnt/lustre Observe that the disk has 12,832 1k blocks available, yet reports itself as 100% full! So, something is not right! Maybe this is also related to striping. Maybe, due to striping, it is not always possible to completely fill a Lustre file system. What do you think? -Roger _________________________________________________________________ Is your PC infected? Get a FREE online computer virus scan from McAfee® Security. http://clinic.mcafee.com/clinic/ibuy/campaign.asp?cid=3963
On Mar 24, 2006 14:25 -0500, RS RS wrote:> I tried striping the file using: > lfs setstripe /mnt/lustre/iobw.tst > > IOMETER is doing something annoying. It is removing that file, then > creating a file from scratch. I have just verified that I can create a > file that fills the filesystem using just shell commands. IOMETER tries > to do very low-level IO, and that may be why it is failing.Instead, set the default striping on a directory and then all NEW files created in that directory will use the new striping: mkdir /mnt/lustre/stripe-all lfs setstripe /mnt/lustre/stripe-all 0 -1 -1 Then, NEW files created in "/mnt/lustre/stripe-all" will stripe over all available OSTs. It uses the default stripe size from the filesystem Similarly, you can change the default striping pattern arbitrarily for each directory (here 2M x 1 OST): mkdir /mnt/lustre/stripe-one lfs setstripe /mnt/lustre/stripe-one 2097152 -1 1 etc. See "lfs setstripe -h" for details. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
----- Original Message ----- From: "RS RS" <rs2006ts@hotmail.com> To: <lustre-discuss@clusterfs.com> Sent: Friday, March 24, 2006 12:59 PM Subject: [Lustre-discuss] Anyone using iometer?> Has anyone there used iometer to measure luster performance? > > I just tried this, and it failed. Running dmesg shows: > > LustreError: 4005:0:(client.c:577:ptlrpc_check_status()) @@@ type == > PTL_RPC_MSG_ERR, err == -28 req@f67dcc00 x1067/t0 > o4->ost2-roger_UUID@clustre5.terascala.com_UUID:28 lens 328/288 ref 2 fl > Rpc:R/0/0 rc 0/-28 > > I opened bug 10340 on this, but if anyone can advise me, I''d appreciate > it. > > Thanks. > > -Roger >Roger, I tend to use Iozone :-) I''m not that fond of Iometer, having read the code and seen it''s behavior... There are many benchmarks that one could try. One possible order might be: Iozone & fileop IOR Bonnie Postmark Netbench dbench dd cat VxBench Iometer Enjoy, Don Capps capps@iozone.org P.S. I may have a certain bias towards Iozone, so you might want to Google for "filesystem benchmark" and use what you find there :-)
Hello! On Fri, Mar 24, 2006 at 01:59:37PM -0500, RS RS wrote:> Has anyone there used iometer to measure luster performance? > I just tried this, and it failed. Running dmesg shows: > LustreError: 4005:0:(client.c:577:ptlrpc_check_status()) @@@ type == > PTL_RPC_MSG_ERR, err == -28 req@f67dcc00 x1067/t0 > o4->ost2-roger_UUID@clustre5.terascala.com_UUID:28 lens 328/288 ref 2 fl > Rpc:R/0/0 rc 0/-28 > I opened bug 10340 on this, but if anyone can advise me, I''d appreciate it.If you plan to fill entire lustre with single file, you need to have that file stripped over all OSTs (and also all of those OSTs should be of the same size currently). This message you are getting telling you that one of the OSTs in use is out of space. Bye, Oleg
> >On Fri, Mar 24, 2006 at 01:59:37PM -0500, RS RS wrote: > > Has anyone there used iometer to measure luster performance? > > I just tried this, and it failed. Running dmesg shows: > > LustreError: 4005:0:(client.c:577:ptlrpc_check_status()) @@@ type => > PTL_RPC_MSG_ERR, err == -28 req@f67dcc00 x1067/t0 > > o4->ost2-roger_UUID@clustre5.terascala.com_UUID:28 lens 328/288 ref 2 fl > > Rpc:R/0/0 rc 0/-28 > > I opened bug 10340 on this, but if anyone can advise me, I''d appreciate >it. > >If you plan to fill entire lustre with single file, you need to have that >file >stripped over all OSTs (and also all of those OSTs should be of the same >size >currently). >This message you are getting telling you that one of the OSTs in use is out >of space. > >Bye, > OlegOleg, I tried striping the file using: lfs setstripe /mnt/lustre/iobw.tst IOMETER is doing something annoying. It is removing that file, then creating a file from scratch. I have just verified that I can create a file that fills the filesystem using just shell commands. IOMETER tries to do very low-level IO, and that may be why it is failing. _________________________________________________________________ FREE pop-up blocking with the new MSN Toolbar – get it now! http://toolbar.msn.click-url.com/go/onm00200415ave/direct/01/
Hello! On Fri, Mar 24, 2006 at 11:22:14AM -0800, Jeffrey W. Baker wrote:> I wonder if more-informative error messages are on the Lustre roadmap.Actually the application should have received proper -ENOSPC, and if it did not reported that back to user, that''s not our fault. The message from lustre should not be output at all, I would say. But it helps to debug cases like this ;) Bye, Oleg
Hello! On Fri, Mar 24, 2006 at 02:25:47PM -0500, RS RS wrote:> I tried striping the file using: > lfs setstripe /mnt/lustre/iobw.tst > IOMETER is doing something annoying. It is removing that file, then > creating a file from scratch. I have just verified that I can create a > file that fills the filesystem using just shell commands. IOMETER tries > to do very low-level IO, and that may be why it is failing.For testing purposes you might set default striping to stripe all files over all OSTs. Then it would work, I guess. Bye, Oleg
RS RS wrote:> Has anyone there used iometer to measure luster performance? > > I just tried this, and it failed. Running dmesg shows: > > LustreError: 4005:0:(client.c:577:ptlrpc_check_status()) @@@ type == > PTL_RPC_MSG_ERR, err == -28 req@f67dcc00 x1067/t0 > o4->ost2-roger_UUID@clustre5.terascala.com_UUID:28 lens 328/288 ref 2 fl > Rpc:R/0/0 rc 0/-28 > > I opened bug 10340 on this, but if anyone can advise me, I''d appreciate it.I would also advise using iozone as opposed to iometer. (Don is biased, but he''s also correct ) Here''s a short script to do a short iozone pass. Note you will need the proper path to the iozone binary.>--------------------------- #!/bin/bash ME=`hostname` LDIR="/mnt/lustre" TESTDIR="$LDIR/$ME" TESTFLAGS="-e -a -M -R -g 10M -q 512k -i0 -i1 -f $TESTDIR/dummyfile" IOZONE="./iozone" mkdir -p $TESTDIR $IOZONE $TESTFLAGS > $TESTDIR/results 2>&1 -------------------------- cliffw> Thanks. > > -Roger > > _________________________________________________________________ > Express yourself instantly with MSN Messenger! Download today - it''s > FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/ > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
On Mar 24, 2006 15:19 -0500, RS RS wrote:> Your advice on striping got me a lot further. I set striping on the parent > directory, and I was able to get much further. However, the test is still > failing. Here is why: > > The ouput from df is: > Filesystem 1K-blocks Used Available Use% Mounted on > multilustre 3120840 2950532 12832 100% /mnt/lustre > > > Observe that the disk has 12,832 1k blocks available, yet reports itself as > 100% full! So, something is not right!The "Use%" number is rounded to the nearest percent.> Maybe this is also related to striping. Maybe, due to striping, it is not > always possible to completely fill a Lustre file system. What do you think?It is true, however, that in some cases it is not possible to fill the filesystem completely full for a single file. The reason is that clients get reservations on the filesystem, to ensure that they do not run out of space when writing their cache to the OSTs. There are also cases when writing from a single client there are multiple write RPCs in flight, and the client runs out of write grant before the filesystem is completely full, but a later write would allow more space to be used. In the vast majority of cases, Lustre filesystems are many TB in size, and missing a few MB at the end is not a serious issue. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
Has anyone there used iometer to measure luster performance? I just tried this, and it failed. Running dmesg shows: LustreError: 4005:0:(client.c:577:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, err == -28 req@f67dcc00 x1067/t0 o4->ost2-roger_UUID@clustre5.terascala.com_UUID:28 lens 328/288 ref 2 fl Rpc:R/0/0 rc 0/-28 I opened bug 10340 on this, but if anyone can advise me, I''d appreciate it. Thanks. -Roger _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today - it''s FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/