Lustre discuss - May 2006 - Lustre 1.0 performance with Gigabit Ethernet

have you got portals debug turned on?

do:

cat /proc/sys/portals/debug

if it is not 0, then:

echo ''0'' > /proc/sys/portals/debug

then rerun the test.

Fran

On Fri, 2003-12-12 at 12:38, Christoph Biardzki wrote:
> Hello,
> 
> 
> I''m currently playing with Lustre 1.0 - it works really great
compared
> to earlier releases :)
> 
> On two Dual-Xeon-2.8 GHz servers linked with Gigabit Ethernet I 
> configured one of them as MDS/OST and the other one as a client. The 
> "storage" is a 10k RPM SCSI disk with approx. 50 MB/s sequential
write
> bandwidth. Using Lustre I achieved around ~20 MB/s (even locally - when 
> the server mounted the filesystem). Using striping with two OST''s
on two
> machines the value stays the same (around 18-20 MB/s) - both disks work 
> with ~10 MB/s.
> 
> I used iozone for the tests (with and without O_DIRECT, 64k-1MB write 
> size and files from 500M to 20G). Hardly any difference except O_DIRECT 
> being slower.
> 
> The NIC is an onboard Intel e1000 (I used the drivers from the lustre 
> kernel and also a current version from intel - no difference).
> 
> When using two simultaneous clients total throughput goes up.
> 
> 
> My conclusion is that somehow the client works slowly in my 
> configuration (even on localhost). Are there any settings I could try or 
>   is this normal speed for Gigabit Ethernet connections?
> 
> 
> 
> Thanks alot!
> 
> 
> - Christoph
> 
> 
> 
> 
-- 
Francis Tyers
email: francis.tyers@hp.com
phone: +353 (0) 91 75 41 34

On Fri, 2003-12-12 at 07:38, Christoph Biardzki wrote:
> 
> My conclusion is that somehow the client works slowly in my 
> configuration (even on localhost). Are there any settings I could try or 
>   is this normal speed for Gigabit Ethernet connections?
Have you turned of debugging? I believe the following will do it:
echo 0 > /proc/sys/portals/debug

Nic
-- 
Nicholas Henke
Penguin Herder & Linux Cluster System Programmer
Liniac Project - Univ. of Pennsylvania

> On two Dual-Xeon-2.8 GHz servers linked with Gigabit Ethernet I
> configured one of them as MDS/OST and the other one as a client. The
> "storage" is a 10k RPM SCSI disk with approx. 50 MB/s sequential
write
> bandwidth. Using Lustre I achieved around ~20 MB/s (even locally - when
> the server mounted the filesystem). Using striping with two OST''s
on two
> machines the value stays the same (around 18-20 MB/s) - both disks work
> with ~10 MB/s.
By default Lustre 1.0 generates a detailed log of its operation to aid
in debugging.  This can easily overwhelm the CPU(s).  You can see if
this is the case for you by examining the sysctl that controls the
debugging:

# sysctl portals.debug
portals.debug = -1

-1 indicates full debugging; its a bitmask.  You can disable it by running

# sysctl -w portals.debug=0
portals.debug = 0

on the nodes concerned.   Watch the top or vmstat utilities before and
after the change for a drop in system CPU use while Lustre is working.
> configuration (even on localhost). Are there any settings I could try or
>  is this normal speed for Gigabit Ethernet connections?
I would expect you to be able to saturate your single drive over that
gigabit ethernet pipe on such capable hosts.

- z

Christoph,

If you don''t mind I''d like to add a little on this item since
we found
some performance issues with the Intel Pro 1000 Ethernet drivers.

Firstly it might be a good idea to see if your Gbit network is capable 
of performing at close to the 100MB/s range between two hosts. I suggest 
you use ttcp to do this. It will quickly show whether your Ethernet 
cards are set-up for maximum performance. We have found that if you are 
using Intel Pro1000 cards and the e1000 driver prior to the 5.1.13 
release  you cannot achieve more than about 50-55 MB/s over the Gbit 
link (however given your Lustre performance is lower than that then it 
would appear you have other issues involved here). However it would be 
worth getting the network interface running at the highest speed first. 
To do this with the Intel Pro 1000''s (this includes the onboard
NIC''s on
the most recent Intel motherboards) you will need the e1000 driver 
version 5.1.13 or above and you must set InterruptThrottleRate=0 in 
/etc/modules.conf. This is particularly useful for improving message 
passing application throughput though I don''t know how much mileage it 
will give for file system based activity. However we saw ttcp improve 
from the already stated 50-55MB/s to between 92-100MB/s depending on 
your switch specification.

This will at least help get your network interconnect set-up to maximum 
speed. If your Lustre performance does not improve then the problem lies 
somewhere in the file system code or communication layer.

I you have not already known about this and it can help a little.

Regards,

John.

John Wagner
Fujitsu Systems Europe
8 rue Maryse Hilsz
31500 Toulouse
France

Christoph Biardzki wrote:
>
>
> Hello,
>
>
> I''m currently playing with Lustre 1.0 - it works really great
compared
> to earlier releases :)
>
> On two Dual-Xeon-2.8 GHz servers linked with Gigabit Ethernet I 
> configured one of them as MDS/OST and the other one as a client. The 
> "storage" is a 10k RPM SCSI disk with approx. 50 MB/s sequential
write
> bandwidth. Using Lustre I achieved around ~20 MB/s (even locally - 
> when the server mounted the filesystem). Using striping with two
OST''s
> on two machines the value stays the same (around 18-20 MB/s) - both 
> disks work with ~10 MB/s.
>
> I used iozone for the tests (with and without O_DIRECT, 64k-1MB write 
> size and files from 500M to 20G). Hardly any difference except 
> O_DIRECT being slower.
>
> The NIC is an onboard Intel e1000 (I used the drivers from the lustre 
> kernel and also a current version from intel - no difference).
>
> When using two simultaneous clients total throughput goes up.
>
>
> My conclusion is that somehow the client works slowly in my 
> configuration (even on localhost). Are there any settings I could try 
> or  is this normal speed for Gigabit Ethernet connections?
>
>
>
> Thanks alot!
>
>
> - Christoph
>
>
>
>
>

Hello,


I''m currently playing with Lustre 1.0 - it works really great compared 
to earlier releases :)

On two Dual-Xeon-2.8 GHz servers linked with Gigabit Ethernet I 
configured one of them as MDS/OST and the other one as a client. The 
"storage" is a 10k RPM SCSI disk with approx. 50 MB/s sequential write
bandwidth. Using Lustre I achieved around ~20 MB/s (even locally - when 
the server mounted the filesystem). Using striping with two OST''s on
two
machines the value stays the same (around 18-20 MB/s) - both disks work 
with ~10 MB/s.

I used iozone for the tests (with and without O_DIRECT, 64k-1MB write 
size and files from 500M to 20G). Hardly any difference except O_DIRECT 
being slower.

The NIC is an onboard Intel e1000 (I used the drivers from the lustre 
kernel and also a current version from intel - no difference).

When using two simultaneous clients total throughput goes up.


My conclusion is that somehow the client works slowly in my 
configuration (even on localhost). Are there any settings I could try or 
  is this normal speed for Gigabit Ethernet connections?



Thanks alot!


- Christoph







-- 
Leibniz Rechenzentrum München (LRZ)
http://www.lrz.de
High Performance Systems Division
Barer Str. 21 - 80333 Munich - Germany
Tel. ++49-(0)89 / 289-28853, Room 1527