Lustre discuss - Nov 2006 - Lustre performace compared to NFS/local Ext3

Skipped content of type multipart/alternative-------------- next part
--------------
A non-text attachment was scrubbed...
Name: ext3-lustre-nfs-iozone-rpt5.xls
Type: application/vnd.ms-excel
Size: 56832 bytes
Desc: ext3-lustre-nfs-iozone-rpt5.xls
Url :
http://mail.clusterfs.com/pipermail/lustre-discuss/attachments/20061121/a822ecf0/ext3-lustre-nfs-iozone-rpt5-0001.xls

Lustre team,

(resending in text format...)

I''m a newbie trying to evaluate Lustre among few other shared storage
products.
 
I''ve installed and configured Luster v1.4.7.3. It is up and running
fine. Now I want to get some performance numbers so that I can compare
w/ other cluster/shared file systems that I plan to evaluate soon. To
start with, I tried Lustre along side with (1) local Ext3 fs and (2) an
NFS to an adjacent Linux server. I''m using IOZONE automatic mode and
numbers are interesting! 
 
(1) NFS out-performs way over Ext3 & Lustre, particularly in Writes. It
almost sounds like an anomaly to me ;-) Do you know anything that might
cause this behavior?
(2) Not surprising, Lustre is lower compared to Ext3, but it seems Reads
in Lustre are consistently very low 
 
Attaching the excel sheet w/ charts to this mail. Does this seem typical
of the results seen in the past in this list? 
 
I also believe every file system need to be tuned to get the best of it.
I would like to know/learn more on how to tune Lustre for various
parameters. Any pointers on this will be helpful.
 
thanks,
Sridhar
-------------------------------
sridhar <at> cisco <dot> com
-------------------------------
 
 
IOZONE commands
---------------------
Local EXT3 Filesystem test...
    /usr/local/bin/iozone -a -y 64 -q 64 -g 1G -f /tmp/iozone/tstfile -R
-b localext3-iozone-rpt4.xls
Lustre Filesystem test...
    /usr/local/bin/iozone -a -y 64 -q 64 -g 1G -f
/mnt/lustre/iozone/tstfile -R -b lustre-iozone-rpt4.xls
NFS Filesystem test...
    /usr/local/bin/iozone -a -y 64 -q 64 -g 1G -f
/localnfs/cfs1/iozone/tstfile -R -b nfs-iozone-rpt4.xls
 
Lustre Configuration
---------------------
 
    System: Linux cfs5 2.6.9-42.0.2.EL_lustre.1.4.7.3smp, 512MB ram
    Node1: 2 OSTs each pointing to 8GB partition of an IDE hard disk
                 Runs MDT
    Node2: 2 OSTs each pointing to 8GB partition of an IDE hard disk
                 Runs the Lustre client
                 (yes, we plan to run both client & OSTs in same server)
 
config.xml
----------
 
rm -f multinode.xml
 
# Create nodes
lmc -m multinode.xml --add node --node cfs1
lmc -m multinode.xml --add node --node cfs5
lmc -m multinode.xml --add node --node client
 
# Create networks
lmc -m multinode.xml --add net --node cfs1 --nid cfs1 --nettype tcp
lmc -m multinode.xml --add net --node cfs5 --nid cfs5 --nettype tcp
lmc -m multinode.xml --add net --node client --nid ''*''
--nettype tcp
 
# Create Meta-Data Server ''MDS''
lmc -m multinode.xml --add mds --node cfs1 --mds lustre-mds --fstype
ext3 --dev /dev/hda10
 
# Create Object-Storage-Targets ''OST'' and
''LOV''
lmc -m multinode.xml --add lov --lov lov1 --mds lustre-mds --stripe_sz
1048576 --stripe_cnt 1 --stripe_pattern 0
 
# Create Object-Storage-Targets ''OST''
# ... in node cfs1
lmc -m multinode.xml --add ost --node cfs1 --ost ost1 --lov lov1
--fstype ext3 --dev /dev/hda8
lmc -m multinode.xml --add ost --node cfs1 --ost ost2 --lov lov1
--fstype ext3 --dev /dev/hda9
 
# ... in node cfs5
lmc -m multinode.xml --add ost --node cfs5 --ost ost3 --lov lov1
--fstype ext3 --dev /dev/hda8
lmc -m multinode.xml --add ost --node cfs5 --ost ost4 --lov lov1
--fstype ext3 --dev /dev/hda9
 
# Create CFS client mount-point
lmc -m multinode.xml --add mtpt --node client --path /mnt/lustre-multi
--mds lustre-mds --lov lov1
 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ext3-lustre-nfs-iozone-rpt5.xls
Type: application/vnd.ms-excel
Size: 56832 bytes
Desc: ext3-lustre-nfs-iozone-rpt5.xls
Url :
http://mail.clusterfs.com/pipermail/lustre-discuss/attachments/20061121/96d28fa6/ext3-lustre-nfs-iozone-rpt5-0001.xls

>>>>> Sridharan Ramaswamy \(srramasw\) (SR\) writes:
 SR\> (1) NFS out-performs way over Ext3 & Lustre, particularly in
Writes. It
 SR\> almost sounds like an anomaly to me ;-) Do you know anything that might
 SR\> cause this behavior?

what mode do you use on NFS server: sync or async?

thanks, Alex

Hi

You could use the IO kit and evaluate your backend and network 
throughput separately.  That should show you the tuning issues.  Also, 
make sure to turn debugging off.

- Peter -

Sridharan Ramaswamy (srramasw) wrote:
> Lustre team,
>
> (resending in text format...)
>
> I''m a newbie trying to evaluate Lustre among few other shared
storage
> products.
>  
> I''ve installed and configured Luster v1.4.7.3. It is up and
running
> fine. Now I want to get some performance numbers so that I can compare
> w/ other cluster/shared file systems that I plan to evaluate soon. To
> start with, I tried Lustre along side with (1) local Ext3 fs and (2) an
> NFS to an adjacent Linux server. I''m using IOZONE automatic mode
and
> numbers are interesting! 
>  
> (1) NFS out-performs way over Ext3 & Lustre, particularly in Writes. It
> almost sounds like an anomaly to me ;-) Do you know anything that might
> cause this behavior?
> (2) Not surprising, Lustre is lower compared to Ext3, but it seems Reads
> in Lustre are consistently very low 
>  
> Attaching the excel sheet w/ charts to this mail. Does this seem typical
> of the results seen in the past in this list? 
>  
> I also believe every file system need to be tuned to get the best of it.
> I would like to know/learn more on how to tune Lustre for various
> parameters. Any pointers on this will be helpful.
>  
> thanks,
> Sridhar
> -------------------------------
> sridhar <at> cisco <dot> com
> -------------------------------
>  
>  
> IOZONE commands
> ---------------------
> Local EXT3 Filesystem test...
>     /usr/local/bin/iozone -a -y 64 -q 64 -g 1G -f /tmp/iozone/tstfile -R
> -b localext3-iozone-rpt4.xls
> Lustre Filesystem test...
>     /usr/local/bin/iozone -a -y 64 -q 64 -g 1G -f
> /mnt/lustre/iozone/tstfile -R -b lustre-iozone-rpt4.xls
> NFS Filesystem test...
>     /usr/local/bin/iozone -a -y 64 -q 64 -g 1G -f
> /localnfs/cfs1/iozone/tstfile -R -b nfs-iozone-rpt4.xls
>  
> Lustre Configuration
> ---------------------
>  
>     System: Linux cfs5 2.6.9-42.0.2.EL_lustre.1.4.7.3smp, 512MB ram
>     Node1: 2 OSTs each pointing to 8GB partition of an IDE hard disk
>                  Runs MDT
>     Node2: 2 OSTs each pointing to 8GB partition of an IDE hard disk
>                  Runs the Lustre client
>                  (yes, we plan to run both client & OSTs in same
server)
>  
> config.xml
> ----------
>  
> rm -f multinode.xml
>  
> # Create nodes
> lmc -m multinode.xml --add node --node cfs1
> lmc -m multinode.xml --add node --node cfs5
> lmc -m multinode.xml --add node --node client
>  
> # Create networks
> lmc -m multinode.xml --add net --node cfs1 --nid cfs1 --nettype tcp
> lmc -m multinode.xml --add net --node cfs5 --nid cfs5 --nettype tcp
> lmc -m multinode.xml --add net --node client --nid ''*''
--nettype tcp
>  
> # Create Meta-Data Server ''MDS''
> lmc -m multinode.xml --add mds --node cfs1 --mds lustre-mds --fstype
> ext3 --dev /dev/hda10
>  
> # Create Object-Storage-Targets ''OST'' and
''LOV''
> lmc -m multinode.xml --add lov --lov lov1 --mds lustre-mds --stripe_sz
> 1048576 --stripe_cnt 1 --stripe_pattern 0
>  
> # Create Object-Storage-Targets ''OST''
> # ... in node cfs1
> lmc -m multinode.xml --add ost --node cfs1 --ost ost1 --lov lov1
> --fstype ext3 --dev /dev/hda8
> lmc -m multinode.xml --add ost --node cfs1 --ost ost2 --lov lov1
> --fstype ext3 --dev /dev/hda9
>  
> # ... in node cfs5
> lmc -m multinode.xml --add ost --node cfs5 --ost ost3 --lov lov1
> --fstype ext3 --dev /dev/hda8
> lmc -m multinode.xml --add ost --node cfs5 --ost ost4 --lov lov1
> --fstype ext3 --dev /dev/hda9
>  
> # Create CFS client mount-point
> lmc -m multinode.xml --add mtpt --node client --path /mnt/lustre-multi
> --mds lustre-mds --lov lov1
>  
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss@clusterfs.com
> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
>   
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://mail.clusterfs.com/pipermail/lustre-discuss/attachments/20061121/3f1aeb16/attachment.html

On Nov 21, 2006  12:23 -0800, Sridharan Ramaswamy (srramasw)
wrote:
> (1) NFS out-performs way over Ext3 & Lustre, particularly in Writes. It
> almost sounds like an anomaly to me ;-) Do you know anything that might
> cause this behavior?
> (2) Not surprising, Lustre is lower compared to Ext3, but it seems Reads
> in Lustre are consistently very low 
>  
> IOZONE commands
> ---------------------
> Local EXT3 Filesystem test...
>     /usr/local/bin/iozone -a -y 64 -q 64 -g 1G -f /tmp/iozone/tstfile -R
> -b localext3-iozone-rpt4.xls
> Lustre Filesystem test...
>     /usr/local/bin/iozone -a -y 64 -q 64 -g 1G -f
> /mnt/lustre/iozone/tstfile -R -b lustre-iozone-rpt4.xls
> NFS Filesystem test...
>     /usr/local/bin/iozone -a -y 64 -q 64 -g 1G -f
> /localnfs/cfs1/iozone/tstfile -R -b nfs-iozone-rpt4.xls
I think your tests & graphs are in a meanlingless part of the sample space.
Limiting file size to 1GB means that almost all of the sample points are
inside the client''s RAM.  Try with -q values from 512MB to 8GB.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

> -----Original Message-----
> From: Andreas Dilger [mailto:adilger@clusterfs.com] 
> Sent: Tuesday, November 21, 2006 1:30 PM
> To: Sridharan Ramaswamy (srramasw)
> Cc: lustre-discuss@clusterfs.com
> Subject: Re: [Lustre-discuss] Lustre performace compared to 
> NFS/local Ext3
> 
<snip>
> >     /usr/local/bin/iozone -a -y 64 -q 64 -g 1G -f
> > /localnfs/cfs1/iozone/tstfile -R -b nfs-iozone-rpt4.xls
> 
> I think your tests & graphs are in a meanlingless part of the 
> sample space.
> Limiting file size to 1GB means that almost all of the sample 
> points are
> inside the client''s RAM.

The reason I picked max file as 1G is,

(1) My Linux server has 512M RAM. I read in IOZone docs that it is
reliable to use twice that amount to avoid just exercising processor
memory
(2) More importantly, this is the file-sz range I believe we''ll have in
our deployment. For this range, do you think Lustre may not be well
suited?
>  Try with -q values from 512MB to 8GB.
Did you mean "-n 512M -g 1G" for specifying file size range? 
"-q" flag
in iozone is record size, which I''m keeping it constant at 64KB. 
I''ll
try using this file size for better understanding Lustre. 

Also I''m thinking of re-running the tests without using the network
(i.e. just w/ one node w/ two OSTs, 1 MDT and 1 client). Just to
eliminate any n/w related issues.


Thanks for your help,
Sridhar
> 
> Cheers, Andreas
> --
> Andreas Dilger
> Principal Software Engineer
> Cluster File Systems, Inc.
>