Lustre discuss - Oct 2009 - [Lustre-Performance] Lustre performance on SLES 11 x86-64

Hi All,


I have SLES 11 x86-64  and lustre v 1.8.1.1 installed in it. This use as MDS
and OSS.

/dev/sda3 ----2 GB------ MDT
/dev/sdb2 ----5 GB------ OST1
/dev/sdb3 ----5 GB------ OST2
/dev/sdb5 ----5 GB------ OST3
/dev/sdb6 ----5 GB------ OST4

Then I have 2 clients running on SLES 11 x86-64 and SLES 10 SP2 x86-64 both
has the lustre v 1.8.1.1 installed.
The SLES 10 client gives the same speed for every bs value in dd command (dd
if=/dev/zero of=/lustre/sles10.out bs=1k gives 43 MB/s)
This is same for the every bs value on SLES 10 SP2.

But for the SLES 11 x86-64 client gives only 4 MB/s speed for the small bs
values. (bs=1k )
It gives 45 MB/s for the bs=32k values and up.

I have attached some outputs from the llstat command on both the clients.

Please give me a solution to tune SLES 11 x86-64 server to get 40 MB/s speed
under bs=1k situvation.
tnx,
Udara
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20091030/706e3fbc/attachment-0001.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: requests
Type: application/octet-stream
Size: 3409 bytes
Desc: not available
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20091030/706e3fbc/attachment-0001.obj

Check the network connections.


What kind of network are you on? 10/100 or 100/1000? Do you see any
dropped packets?



On Fri, Oct 30, 2009 at 7:49 AM, udara weerapperuma <udaraslt at
gmail.com> wrote:
>
> Hi All,
>
>
> I have SLES 11 x86-64? and lustre v 1.8.1.1 installed in it. This use as
MDS
> and OSS.
>
> /dev/sda3 ----2 GB------ MDT
> /dev/sdb2 ----5 GB------ OST1
> /dev/sdb3 ----5 GB------ OST2
> /dev/sdb5 ----5 GB------ OST3
> /dev/sdb6 ----5 GB------ OST4
>
> Then I have 2 clients running on SLES 11 x86-64 and SLES 10 SP2 x86-64 both
> has the lustre v 1.8.1.1 installed.
> The SLES 10 client gives the same speed for every bs value in dd command
(dd
> if=/dev/zero of=/lustre/sles10.out bs=1k gives 43 MB/s)
> This is same for the every bs value on SLES 10 SP2.
>
> But for the SLES 11 x86-64 client gives only 4 MB/s speed for the small bs
> values. (bs=1k )
> It gives 45 MB/s for the bs=32k values and up.
>
> I have attached some outputs from the llstat command on both the clients.
>
> Please give me a solution to tune SLES 11 x86-64 server to get 40 MB/s
speed
> under bs=1k situvation.
> tnx,
> Udara
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>

Hi,

Please see bellow outputs

e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex, Flow
Control: RX

and there is no packet loss on the network.

Kernel Interface table
Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR   TX-OK TX-ERR TX-DRP TX-OVR
Flg
eth0   1500   0 9148114      0      0      016991786      0      0      0
BMRU
lo    16436   0   69389      0      0      0   69389      0      0      0
LRU

regards,
Udara.

On Fri, Oct 30, 2009 at 5:41 PM, Mag Gam <magawake at gmail.com> wrote:
> Check the network connections.
>
>
> What kind of network are you on? 10/100 or 100/1000? Do you see any
> dropped packets?
>
>
>
> On Fri, Oct 30, 2009 at 7:49 AM, udara weerapperuma <udaraslt at
gmail.com>
> wrote:
> >
> > Hi All,
> >
> >
> > I have SLES 11 x86-64  and lustre v 1.8.1.1 installed in it. This use
as
> MDS
> > and OSS.
> >
> > /dev/sda3 ----2 GB------ MDT
> > /dev/sdb2 ----5 GB------ OST1
> > /dev/sdb3 ----5 GB------ OST2
> > /dev/sdb5 ----5 GB------ OST3
> > /dev/sdb6 ----5 GB------ OST4
> >
> > Then I have 2 clients running on SLES 11 x86-64 and SLES 10 SP2 x86-64
> both
> > has the lustre v 1.8.1.1 installed.
> > The SLES 10 client gives the same speed for every bs value in dd
command
> (dd
> > if=/dev/zero of=/lustre/sles10.out bs=1k gives 43 MB/s)
> > This is same for the every bs value on SLES 10 SP2.
> >
> > But for the SLES 11 x86-64 client gives only 4 MB/s speed for the
small
> bs
> > values. (bs=1k )
> > It gives 45 MB/s for the bs=32k values and up.
> >
> > I have attached some outputs from the llstat command on both the
clients.
> >
> > Please give me a solution to tune SLES 11 x86-64 server to get 40 MB/s
> speed
> > under bs=1k situvation.
> > tnx,
> > Udara
> >
> >
> > _______________________________________________
> > Lustre-discuss mailing list
> > Lustre-discuss at lists.lustre.org
> > http://lists.lustre.org/mailman/listinfo/lustre-discuss
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20091030/2f5074c6/attachment.html

On 2009-10-30, at 05:49, udara weerapperuma wrote:
> I have SLES 11 x86-64  and lustre v 1.8.1.1 installed in it. This  
> use as MDS and OSS.
>
> /dev/sda3 ----2 GB------ MDT
> /dev/sdb2 ----5 GB------ OST1
> /dev/sdb3 ----5 GB------ OST2
> /dev/sdb5 ----5 GB------ OST3
> /dev/sdb6 ----5 GB------ OST4
>
> Then I have 2 clients running on SLES 11 x86-64 and SLES 10 SP2  
> x86-64 both has the lustre v 1.8.1.1 installed.
> The SLES 10 client gives the same speed for every bs value in dd  
> command (dd if=/dev/zero of=/lustre/sles10.out bs=1k gives 43 MB/s)
> This is same for the every bs value on SLES 10 SP2.
>
> But for the SLES 11 x86-64 client gives only 4 MB/s speed for the  
> small bs values. (bs=1k) It gives 45 MB/s for the bs=32k values and  
> up.

Using multiple partitions on the same disk is a sure way to get very bad
performance.  Secondly, bs=1k is also a good way to add a lot of  
overhead
to the IO.  At a minimum use bs=4k (to match the client page size).

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.