thr3ads.net - Lustre discuss - [Lustre-discuss] LNET Performance Issue [Feb 2012]

If this information is useful, please help other people find it:
Share via:

Barberi, Carl E

2012-Feb-15 19:30 UTC

[Lustre-discuss] LNET Performance Issue

We are having issues with LNET performance over Infiniband.  We have a
configuration with a single MDT and six (6) OSTs.  The Lustre client I am using
to test is configured to use 6 stripes (lfs setstripe -c  6 /mnt/lustre).  When
I perform a test using the following command:

                dd if=/dev/zero of=/mnt/lustre/test.dat bs=1M count=2000

I typically get a write rate of about 815 MB/s, and we never exceed 848 MB/s. 
When I run obdfilter-survey, we easily get about 3-4GB/s write speed, but when I
run a series of lnet-selftests, the read and write rates range from 850MB/s -
875MB/s max.  I have performed the following optimizations to increase the data
rate:

On the Client:
lctl set_param osc.*.checksums=0
lctl set_param osc.*.max_dirty_mb=256

On the OSTs
lctl set_param obdfilter.*.writethrough_cache_enable=0
lctl set_param obdfilter.*.read_cache_enable=0

echo 4096 > /sys/block/<devices>/queue/nr_requests

I have also loaded the ib_sdp module, which also brought an increase in speed. 
However, we need to be able to record at no less than 1GB/s, which we cannot
achieve right now.  Any thoughts on how I can optimize LNET, which clearly seems
to be the bottleneck?

Thank you for any help you can provide,
Carl Barberi
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20120215/5a9b1935/attachment.html

Kevin Van Maren

2012-Feb-15 20:25 UTC

head link

[Lustre-discuss] LNET Performance Issue

Perhaps someone else here has a thought, but it does not make sense to me that
loading SDP (which accelerates TCP traffic by by-passing the TCP stack) makes
lnet faster if you are using <ip>@o2ib, and _not_ <ip>@tcp0 for your
NIDs.

Any chance you''ve configured both TCP and O2IB nids on the machine, and
it is somehow picking the TCP nids to use?

Can you confirm the "lctl list_nids" output, and your lustre/lnet
sections of your modprobe.conf?

Kevin

On Feb 15, 2012, at 12:30 PM, Barberi, Carl E wrote:

We are having issues with LNET performance over Infiniband. We have a
configuration with a single MDT and six (6) OSTs. The Lustre client I am using
to test is configured to use 6 stripes (lfs setstripe -c 6 /mnt/lustre). When
I perform a test using the following command:

dd if=/dev/zero of=/mnt/lustre/test.dat bs=1M count=2000

I typically get a write rate of about 815 MB/s, and we never exceed 848 MB/s.
When I run obdfilter-survey, we easily get about 3-4GB/s write speed, but when I
run a series of lnet-selftests, the read and write rates range from 850MB/s ?
875MB/s max. I have performed the following optimizations to increase the data
rate:

On the Client:
lctl set_param osc.*.checksums=0
lctl set_param osc.*.max_dirty_mb=256

On the OSTs
lctl set_param obdfilter.*.writethrough_cache_enable=0
lctl set_param obdfilter.*.read_cache_enable=0

echo 4096 > /sys/block/<devices>/queue/nr_requests

I have also loaded the ib_sdp module, which also brought an increase in speed.
However, we need to be able to record at no less than 1GB/s, which we cannot
achieve right now. Any thoughts on how I can optimize LNET, which clearly seems
to be the bottleneck?

Thank you for any help you can provide,
Carl Barberi
<ATT00001..txt>

This e-mail message, its contents and any attachments to it are confidential to
the intended recipient, and may contain information that is privileged and/or
exempt from disclosure under applicable law. If you are not the intended
recipient, please immediately notify the sender and destroy the original e-mail
message and any attachments (and any copies that may have been made) from your
system or otherwise. Any unauthorized use, copying, disclosure or distribution
of this information is strictly prohibited. Email addresses that end with a
?-c? identify the sender as a Fusion-io contractor.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20120215/54e17edf/attachment-0001.html

Barberi, Carl E

2012-Feb-15 20:55 UTC

head link

[Lustre-discuss] LNET Performance Issue

Sure.

lctl list_nids output for test client:

172.25.103.23 at o2ib<mailto:172.25.103.23 at o2ib>
172.25.104.23 at o2ib1<mailto:172.25.104.23 at o2ib1>

lctl list_nids output for MDS:

172.25.103.1 at o2ib<mailto:172.25.103.1 at o2ib>
172.25.103.1 at o2ib1<mailto:172.25.103.1 at o2ib1>

lctl list_nids output for each OSS:
172.25.103.x at o2ib<mailto:172.25.103.x at o2ib>
172.25.104.x at o2ib1<mailto:172.25.104.x at o2ib1>

modprobe.d/lustre.conf output:
MDS: options lnet networks=o2ib0(ib0),o2ib1(ib0)
OSS: options lnet networks=o2ib0(ib0),o2ib1(ib2)
Client: options lnet networks=o2ib0(ib0),o2ib1(ib1)

Carl


From: Kevin Van Maren [mailto:KVanMaren at fusionio.com]
Sent: Wednesday, February 15, 2012 3:25 PM
To: Barberi, Carl E
Cc: lustre-discuss at lists.Lustre.org
Subject: EXTERNAL: Re: [Lustre-discuss] LNET Performance Issue

Perhaps someone else here has a thought, but it does not make sense to me that
loading SDP (which accelerates TCP traffic by by-passing the TCP stack) makes
lnet faster if you are using <ip>@o2ib, and _not_ <ip>@tcp0 for your
NIDs.

Any chance you''ve configured both TCP and O2IB nids on the machine, and
it is somehow picking the TCP nids to use?

Can you confirm the "lctl list_nids" output, and your lustre/lnet
sections of your modprobe.conf?

Kevin


On Feb 15, 2012, at 12:30 PM, Barberi, Carl E wrote:


We are having issues with LNET performance over Infiniband.  We have a
configuration with a single MDT and six (6) OSTs.  The Lustre client I am using
to test is configured to use 6 stripes (lfs setstripe -c  6 /mnt/lustre).  When
I perform a test using the following command:

                dd if=/dev/zero of=/mnt/lustre/test.dat bs=1M count=2000

I typically get a write rate of about 815 MB/s, and we never exceed 848 MB/s. 
When I run obdfilter-survey, we easily get about 3-4GB/s write speed, but when I
run a series of lnet-selftests, the read and write rates range from 850MB/s -
875MB/s max.  I have performed the following optimizations to increase the data
rate:

On the Client:
lctl set_param osc.*.checksums=0
lctl set_param osc.*.max_dirty_mb=256

On the OSTs
lctl set_param obdfilter.*.writethrough_cache_enable=0
lctl set_param obdfilter.*.read_cache_enable=0

echo 4096 > /sys/block/<devices>/queue/nr_requests

I have also loaded the ib_sdp module, which also brought an increase in speed. 
However, we need to be able to record at no less than 1GB/s, which we cannot
achieve right now.  Any thoughts on how I can optimize LNET, which clearly seems
to be the bottleneck?

Thank you for any help you can provide,
Carl Barberi
<ATT00001..txt>



Confidentiality Notice: This e-mail message, its contents and any attachments to
it are confidential to the intended recipient, and may contain information that
is privileged and/or exempt from disclosure under applicable law. If you are not
the intended recipient, please immediately notify the sender and destroy the
original e-mail message and any attachments (and any copies that may have been
made) from your system or otherwise. Any unauthorized use, copying, disclosure
or distribution of this information is strictly prohibited. Email addresses that
end with a "-c" identify the sender as a Fusion-io contractor.
  
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20120215/31a03470/attachment.html

Tommi T

2012-Feb-16 06:35 UTC

head link

[Lustre-discuss] LNET Performance Issue

> lctl list_nids output for MDS: 
>  ? 
> 172.25.103.1 at o2ib 
> 172.25.103.1 at o2ib1 
Is this a typo, o2ib1 should be 172.25.104.1 ?

BR,

Tommi

Liang Zhen

2012-Feb-16 06:44 UTC

head link

[Lustre-discuss] LNET Performance Issue

Hi, I assume you are using "size=1M" for brw test right? performance
could increase if you set "concurrency" while adding brw test, i.e:
--concurrency=16

Liang

On Feb 16, 2012, at 3:30 AM, Barberi, Carl E wrote:
> We are having issues with LNET performance over Infiniband.  We have a
configuration with a single MDT and six (6) OSTs.  The Lustre client I am using
to test is configured to use 6 stripes (lfs setstripe -c  6 /mnt/lustre).  When
I perform a test using the following command:
>  
>                 dd if=/dev/zero of=/mnt/lustre/test.dat bs=1M count=2000
>  
> I typically get a write rate of about 815 MB/s, and we never exceed 848
MB/s.  When I run obdfilter-survey, we easily get about 3-4GB/s write speed, but
when I run a series of lnet-selftests, the read and write rates range from
850MB/s ? 875MB/s max.  I have performed the following optimizations to increase
the data rate:
>  
> On the Client:
> lctl set_param osc.*.checksums=0
> lctl set_param osc.*.max_dirty_mb=256
>  
> On the OSTs
> lctl set_param obdfilter.*.writethrough_cache_enable=0
> lctl set_param obdfilter.*.read_cache_enable=0
>  
> echo 4096 > /sys/block/<devices>/queue/nr_requests
>  
> I have also loaded the ib_sdp module, which also brought an increase in
speed.  However, we need to be able to record at no less than 1GB/s, which we
cannot achieve right now.  Any thoughts on how I can optimize LNET, which
clearly seems to be the bottleneck?
>  
> Thank you for any help you can provide,
> Carl Barberi
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20120216/0eee2a36/attachment-0001.html

Barberi, Carl E

2012-Feb-20 21:59 UTC

head link

[Lustre-discuss] EXTERNAL: Re: LNET Performance Issue

Thank you.   This did help.  With the concurrency set to 16, I was able to get a
max write speed of 1138 MB/s.  Any ideas on how we can make that faster, though?
Ideally, we''d like to get to 1.5 GB/s.

Carl

From: Liang Zhen [mailto:liang at whamcloud.com]
Sent: Thursday, February 16, 2012 1:45 AM
To: Barberi, Carl E
Cc: ''lustre-discuss at lists.Lustre.org''
Subject: EXTERNAL: Re: [Lustre-discuss] LNET Performance Issue

Hi, I assume you are using "size=1M" for brw test right? performance
could increase if you set "concurrency" while adding brw test, i.e:
--concurrency=16

Liang

On Feb 16, 2012, at 3:30 AM, Barberi, Carl E wrote:

We are having issues with LNET performance over Infiniband.  We have a
configuration with a single MDT and six (6) OSTs.  The Lustre client I am using
to test is configured to use 6 stripes (lfs setstripe -c  6 /mnt/lustre).  When
I perform a test using the following command:

                dd if=/dev/zero of=/mnt/lustre/test.dat bs=1M count=2000

I typically get a write rate of about 815 MB/s, and we never exceed 848 MB/s. 
When I run obdfilter-survey, we easily get about 3-4GB/s write speed, but when I
run a series of lnet-selftests, the read and write rates range from 850MB/s -
875MB/s max.  I have performed the following optimizations to increase the data
rate:

On the Client:
lctl set_param osc.*.checksums=0
lctl set_param osc.*.max_dirty_mb=256

On the OSTs
lctl set_param obdfilter.*.writethrough_cache_enable=0
lctl set_param obdfilter.*.read_cache_enable=0

echo 4096 > /sys/block/<devices>/queue/nr_requests

I have also loaded the ib_sdp module, which also brought an increase in speed. 
However, we need to be able to record at no less than 1GB/s, which we cannot
achieve right now.  Any thoughts on how I can optimize LNET, which clearly seems
to be the bottleneck?

Thank you for any help you can provide,
Carl Barberi
_______________________________________________
Lustre-discuss mailing list
Lustre-discuss at lists.lustre.org<mailto:Lustre-discuss at
lists.lustre.org>
http://lists.lustre.org/mailman/listinfo/lustre-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20120220/c6e3983f/attachment.html

Jeremy Filizetti

2012-Feb-21 00:43 UTC

head link

[Lustre-discuss] EXTERNAL: Re: LNET Performance Issue

Am I reading your earlier post correctly that you have a single server
acting as the MDS and OSS?  Have you changed your peer_credits and credits
for ko2iblnd kernel module on the server and client?  You also mentioned
changing osc.*.max_dirty_mb, you probably need to adjust
osc.*.max_rpcs_in_flight as well.  Can you post your rpc stats "lctl
get_param osc.*.rpc_stats"?  I would guess they are bunching up around 7-8
if your running with the default max_rpcs_in_flight=8.

Jeremy


On Mon, Feb 20, 2012 at 4:59 PM, Barberi, Carl E <carl.e.barberi at
lmco.com>wrote:
>  Thank you.   This did help.  With the concurrency set to 16, I was able
> to get a max write speed of 1138 MB/s.  Any ideas on how we can make that
> faster, though?  Ideally, we?d like to get to 1.5 GB/s.****
>
> ** **
>
> Carl****
>
> ** **
>
> *From:* Liang Zhen [mailto:liang at whamcloud.com]
> *Sent:* Thursday, February 16, 2012 1:45 AM
> *To:* Barberi, Carl E
> *Cc:* ''lustre-discuss at lists.Lustre.org''
> *Subject:* EXTERNAL: Re: [Lustre-discuss] LNET Performance Issue****
>
> ** **
>
> Hi, I assume you are using "size=1M" for brw test right?
performance could
> increase if you set "concurrency" while adding brw test, i.e:
> --concurrency=16****
>
> ** **
>
> Liang****
>
> ** **
>
> On Feb 16, 2012, at 3:30 AM, Barberi, Carl E wrote:****
>
>
>
> ****
>
> We are having issues with LNET performance over Infiniband.  We have a
> configuration with a single MDT and six (6) OSTs.  The Lustre client I am
> using to test is configured to use 6 stripes (lfs setstripe -c  6
> /mnt/lustre).  When I perform a test using the following command:****
>
>  ****
>
>                 dd if=/dev/zero of=/mnt/lustre/test.dat bs=1M count=2000**
> **
>
>  ****
>
> I typically get a write rate of about 815 MB/s, and we never exceed 848
> MB/s.  When I run obdfilter-survey, we easily get about 3-4GB/s write
> speed, but when I run a series of lnet-selftests, the read and write rates
> range from 850MB/s ? 875MB/s max.  I have performed the following
> optimizations to increase the data rate:****
>
>  ****
>
> On the Client:****
>
> lctl set_param osc.*.checksums=0****
>
> lctl set_param osc.*.max_dirty_mb=256****
>
>  ****
>
> On the OSTs****
>
> lctl set_param obdfilter.*.writethrough_cache_enable=0****
>
> lctl set_param obdfilter.*.read_cache_enable=0****
>
>  ****
>
> echo 4096 > /sys/block/<devices>/queue/nr_requests****
>
>  ****
>
> I have also loaded the ib_sdp module, which also brought an increase in
> speed.  However, we need to be able to record at no less than 1GB/s, which
> we cannot achieve right now.  Any thoughts on how I can optimize LNET,
> which clearly seems to be the bottleneck?****
>
>  ****
>
> Thank you for any help you can provide,****
>
> Carl Barberi****
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss****
>
> ** **
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20120220/955dc20f/attachment.html

Jeremy Filizetti

2012-Feb-21 00:55 UTC

head link

[Lustre-discuss] EXTERNAL: Re: LNET Performance Issue

Apparently I didn''t read your email very carefully since you were
actually
only wondering about LNET performance and not client performance. You can
ignore the max_rpcs_in_flight for lnet.  There used to be a
srpc_peer_credits setting in lnet_selftest, not sure if it exists in your
version.  Should be able to find out with "modinfo lnet_selftest"

On Mon, Feb 20, 2012 at 7:43 PM, Jeremy Filizetti <
jeremy.filizetti at gmail.com> wrote:
> Am I reading your earlier post correctly that you have a single server
> acting as the MDS and OSS?  Have you changed your peer_credits and credits
> for ko2iblnd kernel module on the server and client?  You also mentioned
> changing osc.*.max_dirty_mb, you probably need to adjust
> osc.*.max_rpcs_in_flight as well.  Can you post your rpc stats "lctl
> get_param osc.*.rpc_stats"?  I would guess they are bunching up around
7-8
> if your running with the default max_rpcs_in_flight=8.
>
> Jeremy
>
>
>
> On Mon, Feb 20, 2012 at 4:59 PM, Barberi, Carl E <carl.e.barberi at
lmco.com>wrote:
>
>>  Thank you.   This did help.  With the concurrency set to 16, I was
able
>> to get a max write speed of 1138 MB/s.  Any ideas on how we can make
that
>> faster, though?  Ideally, we?d like to get to 1.5 GB/s.****
>>
>> ** **
>>
>> Carl****
>>
>> ** **
>>
>> *From:* Liang Zhen [mailto:liang at whamcloud.com]
>> *Sent:* Thursday, February 16, 2012 1:45 AM
>> *To:* Barberi, Carl E
>> *Cc:* ''lustre-discuss at lists.Lustre.org''
>> *Subject:* EXTERNAL: Re: [Lustre-discuss] LNET Performance Issue****
>>
>> ** **
>>
>> Hi, I assume you are using "size=1M" for brw test right?
performance
>> could increase if you set "concurrency" while adding brw
test, i.e:
>> --concurrency=16****
>>
>> ** **
>>
>> Liang****
>>
>> ** **
>>
>> On Feb 16, 2012, at 3:30 AM, Barberi, Carl E wrote:****
>>
>>
>>
>> ****
>>
>> We are having issues with LNET performance over Infiniband.  We have a
>> configuration with a single MDT and six (6) OSTs.  The Lustre client I
am
>> using to test is configured to use 6 stripes (lfs setstripe -c  6
>> /mnt/lustre).  When I perform a test using the following command:****
>>
>>  ****
>>
>>                 dd if=/dev/zero of=/mnt/lustre/test.dat bs=1M
count=2000*
>> ***
>>
>>  ****
>>
>> I typically get a write rate of about 815 MB/s, and we never exceed 848
>> MB/s.  When I run obdfilter-survey, we easily get about 3-4GB/s write
>> speed, but when I run a series of lnet-selftests, the read and write
rates
>> range from 850MB/s ? 875MB/s max.  I have performed the following
>> optimizations to increase the data rate:****
>>
>>  ****
>>
>> On the Client:****
>>
>> lctl set_param osc.*.checksums=0****
>>
>> lctl set_param osc.*.max_dirty_mb=256****
>>
>>  ****
>>
>> On the OSTs****
>>
>> lctl set_param obdfilter.*.writethrough_cache_enable=0****
>>
>> lctl set_param obdfilter.*.read_cache_enable=0****
>>
>>  ****
>>
>> echo 4096 > /sys/block/<devices>/queue/nr_requests****
>>
>>  ****
>>
>> I have also loaded the ib_sdp module, which also brought an increase in
>> speed.  However, we need to be able to record at no less than 1GB/s,
which
>> we cannot achieve right now.  Any thoughts on how I can optimize LNET,
>> which clearly seems to be the bottleneck?****
>>
>>  ****
>>
>> Thank you for any help you can provide,****
>>
>> Carl Barberi****
>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss****
>>
>> ** **
>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>
>>
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20120220/a972955d/attachment-0001.html

Kevin Van Maren

2012-Feb-21 01:14 UTC

head link

[Lustre-discuss] EXTERNAL: Re: LNET Performance Issue

While it''s possible the default credits (8 as I recall) is not enough
for peak performance, it seems to me that something else is wrong:
Each 1MB RPC should take ~300uS (based on MPI/IB xfer rates of 3.2+ GB/s), so
that means there is another 400uS overhead per RPC that is not masked with 8
concurrent RPCs, in addition to the overhead masked when he increased
concurrency.  This is crazy, with a 1uS network latency.

Unless the RPCs are are being broken into tiny chunks or something -- does lnet
do single-page xfers and not use a rendezvous protocol for full-sized RPCs?  It
definitely seems that something is broken when o2iblnd gets ~1/3 of the MPI BW,
given that lnd was designed for high-speed xfers.

The max_rpcs_in_flight normally needs tweaking to improve disk concurrency,
where a single client needs to drive a high queue depth. Still finding it hard
to believe 8 1MB concurrent RPCs can''t handle the network.

Kevin


On Feb 20, 2012, at 5:44 PM, "Jeremy Filizetti"
<<mailto:jeremy.filizetti at gmail.com><mailto:jeremy.filizetti at
gmail.com>jeremy.filizetti at gmail.com<mailto:jeremy.filizetti at
gmail.com>> wrote:

Am I reading your earlier post correctly that you have a single server acting as
the MDS and OSS?  Have you changed your peer_credits and credits for ko2iblnd
kernel module on the server and client?  You also mentioned changing
osc.*.max_dirty_mb, you probably need to adjust osc.*.max_rpcs_in_flight as
well.  Can you post your rpc stats "lctl get_param osc.*.rpc_stats"? 
I would guess they are bunching up around 7-8 if your running with the default
max_rpcs_in_flight=8.

Jeremy


On Mon, Feb 20, 2012 at 4:59 PM, Barberi, Carl E <<mailto:carl.e.barberi
at lmco.com><mailto:carl.e.barberi at
lmco.com><mailto:carl.e.barberi at lmco.com>carl.e.barberi at
lmco.com<mailto:carl.e.barberi at lmco.com>> wrote:
Thank you.   This did help.  With the concurrency set to 16, I was able to get a
max write speed of 1138 MB/s.  Any ideas on how we can make that faster, though?
Ideally, we?d like to get to 1.5 GB/s.

Carl

From: Liang Zhen [mailto:<mailto:liang at whamcloud.com><mailto:liang
at whamcloud.com><mailto:liang at whamcloud.com>liang at
whamcloud.com<mailto:liang at whamcloud.com>]
Sent: Thursday, February 16, 2012 1:45 AM
To: Barberi, Carl E
Cc: ''<mailto:lustre-discuss at
lists.Lustre.org><mailto:lustre-discuss at
lists.Lustre.org><mailto:lustre-discuss at
lists.Lustre.org>lustre-discuss at lists.Lustre.org<mailto:lustre-discuss
at lists.Lustre.org>''
Subject: EXTERNAL: Re: [Lustre-discuss] LNET Performance Issue

Hi, I assume you are using "size=1M" for brw test right? performance
could increase if you set "concurrency" while adding brw test, i.e:
--concurrency=16

Liang

On Feb 16, 2012, at 3:30 AM, Barberi, Carl E wrote:


We are having issues with LNET performance over Infiniband.  We have a
configuration with a single MDT and six (6) OSTs.  The Lustre client I am using
to test is configured to use 6 stripes (lfs setstripe -c  6 /mnt/lustre).  When
I perform a test using the following command:

                dd if=/dev/zero of=/mnt/lustre/test.dat bs=1M count=2000

I typically get a write rate of about 815 MB/s, and we never exceed 848 MB/s. 
When I run obdfilter-survey, we easily get about 3-4GB/s write speed, but when I
run a series of lnet-selftests, the read and write rates range from 850MB/s ?
875MB/s max.  I have performed the following optimizations to increase the data
rate:

On the Client:
lctl set_param osc.*.checksums=0
lctl set_param osc.*.max_dirty_mb=256

On the OSTs
lctl set_param obdfilter.*.writethrough_cache_enable=0
lctl set_param obdfilter.*.read_cache_enable=0

echo 4096 > /sys/block/<devices>/queue/nr_requests

I have also loaded the ib_sdp module, which also brought an increase in speed. 
However, we need to be able to record at no less than 1GB/s, which we cannot
achieve right now.  Any thoughts on how I can optimize LNET, which clearly seems
to be the bottleneck?

Thank you for any help you can provide,
Carl Barberi
_______________________________________________
Lustre-discuss mailing list
<mailto:Lustre-discuss at lists.lustre.org><mailto:Lustre-discuss at
lists.lustre.org><mailto:Lustre-discuss at
lists.lustre.org>Lustre-discuss at lists.lustre.org<mailto:Lustre-discuss
at lists.lustre.org>
<http://lists.lustre.org/mailman/listinfo/lustre-discuss><http://lists.lustre.org/mailman/listinfo/lustre-discuss><http://lists.lustre.org/mailman/listinfo/lustre-discuss>http://lists.lustre.org/mailman/listinfo/lustre-discuss


_______________________________________________
Lustre-discuss mailing list
<mailto:Lustre-discuss at lists.lustre.org><mailto:Lustre-discuss at
lists.lustre.org><mailto:Lustre-discuss at
lists.lustre.org>Lustre-discuss at lists.lustre.org<mailto:Lustre-discuss
at lists.lustre.org>
<http://lists.lustre.org/mailman/listinfo/lustre-discuss><http://lists.lustre.org/mailman/listinfo/lustre-discuss><http://lists.lustre.org/mailman/listinfo/lustre-discuss>http://lists.lustre.org/mailman/listinfo/lustre-discuss


_______________________________________________
Lustre-discuss mailing list
<mailto:Lustre-discuss at lists.lustre.org><mailto:Lustre-discuss at
lists.lustre.org>Lustre-discuss at lists.lustre.org<mailto:Lustre-discuss
at lists.lustre.org>
<http://lists.lustre.org/mailman/listinfo/lustre-discuss><http://lists.lustre.org/mailman/listinfo/lustre-discuss>http://lists.lustre.org/mailman/listinfo/lustre-discuss

Confidentiality Notice: This e-mail message, its contents and any attachments to
it are confidential to the intended recipient, and may contain information that
is privileged and/or exempt from disclosure under applicable law. If you are not
the intended recipient, please immediately notify the sender and destroy the
original e-mail message and any attachments (and any copies that may have been
made) from your system or otherwise. Any unauthorized use, copying, disclosure
or distribution of this information is strictly prohibited.  Email addresses
that end with a ?-c? identify the sender as a Fusion-io contractor.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20120220/150b1773/attachment.html

Jeremy Filizetti

2012-Feb-21 01:36 UTC

head link

[Lustre-discuss] EXTERNAL: Re: LNET Performance Issue

It does seem extreme for data center IB latency but it may not be in the
data center.  The LNet write should take 2 RTT latencies, and 3 for reads
so you could double/triple those times plus any overhead.

Carl can you clarify if you are using QDR IB and/or any campus or wide area
IB extenders?

Jeremy

On Mon, Feb 20, 2012 at 8:14 PM, Kevin Van Maren <KVanMaren at
fusionio.com>wrote:
> While it''s possible the default credits (8 as I recall) is not
enough for
> peak performance, it seems to me that something else is wrong:
> Each 1MB RPC should take ~300uS (based on MPI/IB xfer rates of 3.2+ GB/s),
> so that means there is another 400uS overhead per RPC that is not masked
> with 8 concurrent RPCs, in addition to the overhead masked when he
> increased concurrency.  This is crazy, with a 1uS network latency.
>
> Unless the RPCs are are being broken into tiny chunks or something -- does
> lnet do single-page xfers and not use a rendezvous protocol for full-sized
> RPCs?  It definitely seems that something is broken when o2iblnd gets ~1/3
> of the MPI BW, given that lnd was designed for high-speed xfers.
>
> The max_rpcs_in_flight normally needs tweaking to improve disk
> concurrency, where a single client needs to drive a high queue depth. Still
> finding it hard to believe 8 1MB concurrent RPCs can''t handle the
network.
>
> Kevin
>
>
> On Feb 20, 2012, at 5:44 PM, "Jeremy Filizetti"
<<jeremy.filizetti at gmail.com><jeremy.filizetti at gmail.com>
> jeremy.filizetti at gmail.com> wrote:
>
> Am I reading your earlier post correctly that you have a single server
> acting as the MDS and OSS?  Have you changed your peer_credits and credits
> for ko2iblnd kernel module on the server and client?  You also mentioned
> changing osc.*.max_dirty_mb, you probably need to adjust
> osc.*.max_rpcs_in_flight as well.  Can you post your rpc stats "lctl
> get_param osc.*.rpc_stats"?  I would guess they are bunching up around
7-8
> if your running with the default max_rpcs_in_flight=8.
>
> Jeremy
>
>
> On Mon, Feb 20, 2012 at 4:59 PM, Barberi, Carl E <<carl.e.barberi at
lmco.com><carl.e.barberi at lmco.com><carl.e.barberi at lmco.com>
> carl.e.barberi at lmco.com> wrote:
>
>>  Thank you.   This did help.  With the concurrency set to 16, I was
able
>> to get a max write speed of 1138 MB/s.  Any ideas on how we can make
that
>> faster, though?  Ideally, we?d like to get to 1.5 GB/s.****
>>
>> ** **
>>
>> Carl****
>>
>> ** **
>>
>> *From:* Liang Zhen [mailto: <liang at whamcloud.com> <liang at
whamcloud.com><liang at whamcloud.com>
>> liang at whamcloud.com]
>> *Sent:* Thursday, February 16, 2012 1:45 AM
>> *To:* Barberi, Carl E
>> *Cc:* '' <lustre-discuss at
lists.Lustre.org><lustre-discuss at lists.Lustre.org><lustre-discuss
at lists.Lustre.org>
>> lustre-discuss at lists.Lustre.org''
>> *Subject:* EXTERNAL: Re: [Lustre-discuss] LNET Performance Issue****
>>
>> ** **
>>
>> Hi, I assume you are using "size=1M" for brw test right?
performance
>> could increase if you set "concurrency" while adding brw
test, i.e:
>> --concurrency=16****
>>
>> ** **
>>
>> Liang****
>>
>> ** **
>>
>> On Feb 16, 2012, at 3:30 AM, Barberi, Carl E wrote:****
>>
>>
>>
>> ****
>>
>> We are having issues with LNET performance over Infiniband.  We have a
>> configuration with a single MDT and six (6) OSTs.  The Lustre client I
am
>> using to test is configured to use 6 stripes (lfs setstripe -c  6
>> /mnt/lustre).  When I perform a test using the following command:****
>>
>>  ****
>>
>>                 dd if=/dev/zero of=/mnt/lustre/test.dat bs=1M
count=2000*
>> ***
>>
>>  ****
>>
>> I typically get a write rate of about 815 MB/s, and we never exceed 848
>> MB/s.  When I run obdfilter-survey, we easily get about 3-4GB/s write
>> speed, but when I run a series of lnet-selftests, the read and write
rates
>> range from 850MB/s ? 875MB/s max.  I have performed the following
>> optimizations to increase the data rate:****
>>
>>  ****
>>
>> On the Client:****
>>
>> lctl set_param osc.*.checksums=0****
>>
>> lctl set_param osc.*.max_dirty_mb=256****
>>
>>  ****
>>
>> On the OSTs****
>>
>> lctl set_param obdfilter.*.writethrough_cache_enable=0****
>>
>> lctl set_param obdfilter.*.read_cache_enable=0****
>>
>>  ****
>>
>> echo 4096 > /sys/block/<devices>/queue/nr_requests****
>>
>>  ****
>>
>> I have also loaded the ib_sdp module, which also brought an increase in
>> speed.  However, we need to be able to record at no less than 1GB/s,
which
>> we cannot achieve right now.  Any thoughts on how I can optimize LNET,
>> which clearly seems to be the bottleneck?****
>>
>>  ****
>>
>> Thank you for any help you can provide,****
>>
>> Carl Barberi****
>>
>> _______________________________________________
>> Lustre-discuss mailing list
>>  <Lustre-discuss at lists.lustre.org> <Lustre-discuss at
lists.lustre.org><Lustre-discuss at lists.lustre.org>
>> Lustre-discuss at lists.lustre.org
>> 
<http://lists.lustre.org/mailman/listinfo/lustre-discuss><http://lists.lustre.org/mailman/listinfo/lustre-discuss><http://lists.lustre.org/mailman/listinfo/lustre-discuss>
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss****
>>
>> ** **
>>
>> _______________________________________________
>> Lustre-discuss mailing list
>>  <Lustre-discuss at lists.lustre.org> <Lustre-discuss at
lists.lustre.org><Lustre-discuss at lists.lustre.org>
>> Lustre-discuss at lists.lustre.org
>> 
<http://lists.lustre.org/mailman/listinfo/lustre-discuss><http://lists.lustre.org/mailman/listinfo/lustre-discuss><http://lists.lustre.org/mailman/listinfo/lustre-discuss>
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>
>>
> _______________________________________________
> Lustre-discuss mailing list
> <Lustre-discuss at lists.lustre.org> <Lustre-discuss at
lists.lustre.org>
> Lustre-discuss at lists.lustre.org
>
<http://lists.lustre.org/mailman/listinfo/lustre-discuss><http://lists.lustre.org/mailman/listinfo/lustre-discuss>
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
>
> Confidentiality Notice: This e-mail message, its contents and any
> attachments to it are confidential to the intended recipient, and may
> contain information that is privileged and/or exempt from disclosure under
> applicable law. If you are not the intended recipient, please immediately
> notify the sender and destroy the original e-mail message and any
> attachments (and any copies that may have been made) from your system or
> otherwise. Any unauthorized use, copying, disclosure or distribution of
> this information is strictly prohibited. Email addresses that end with a
> ?-c? identify the sender as a Fusion-io contractor.
>   ??
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20120220/d2b48663/attachment-0001.html

Barberi, Carl E

2012-Feb-21 13:26 UTC

head link

[Lustre-discuss] EXTERNAL: Re: LNET Performance Issue

I am using QDR IB and I am not using any IB extenders.  This is a self-contained
network, with no access outside of the lab it is currently in.

Carl

From: Jeremy Filizetti [mailto:jeremy.filizetti at gmail.com]
Sent: Monday, February 20, 2012 8:37 PM
To: Kevin Van Maren
Cc: Barberi, Carl E; lustre-discuss at lists.Lustre.org; isaac_huang at
xyratex.com
Subject: Re: [Lustre-discuss] EXTERNAL: Re: LNET Performance Issue

It does seem extreme for data center IB latency but it may not be in the data
center.  The LNet write should take 2 RTT latencies, and 3 for reads so you
could double/triple those times plus any overhead.

Carl can you clarify if you are using QDR IB and/or any campus or wide area IB
extenders?

Jeremy
On Mon, Feb 20, 2012 at 8:14 PM, Kevin Van Maren <KVanMaren at
fusionio.com<mailto:KVanMaren at fusionio.com>> wrote:
While it''s possible the default credits (8 as I recall) is not enough
for peak performance, it seems to me that something else is wrong:
Each 1MB RPC should take ~300uS (based on MPI/IB xfer rates of 3.2+ GB/s), so
that means there is another 400uS overhead per RPC that is not masked with 8
concurrent RPCs, in addition to the overhead masked when he increased
concurrency.  This is crazy, with a 1uS network latency.

Unless the RPCs are are being broken into tiny chunks or something -- does lnet
do single-page xfers and not use a rendezvous protocol for full-sized RPCs?  It
definitely seems that something is broken when o2iblnd gets ~1/3 of the MPI BW,
given that lnd was designed for high-speed xfers.

The max_rpcs_in_flight normally needs tweaking to improve disk concurrency,
where a single client needs to drive a high queue depth. Still finding it hard
to believe 8 1MB concurrent RPCs can''t handle the network.

Kevin

On Feb 20, 2012, at 5:44 PM, "Jeremy Filizetti" <jeremy.filizetti
at gmail.com<mailto:jeremy.filizetti at gmail.com>> wrote:
Am I reading your earlier post correctly that you have a single server acting as
the MDS and OSS?  Have you changed your peer_credits and credits for ko2iblnd
kernel module on the server and client?  You also mentioned changing
osc.*.max_dirty_mb, you probably need to adjust osc.*.max_rpcs_in_flight as
well.  Can you post your rpc stats "lctl get_param osc.*.rpc_stats"? 
I would guess they are bunching up around 7-8 if your running with the default
max_rpcs_in_flight=8.

Jeremy

On Mon, Feb 20, 2012 at 4:59 PM, Barberi, Carl E <carl.e.barberi at
lmco.com<mailto:carl.e.barberi at lmco.com>> wrote:
Thank you.   This did help.  With the concurrency set to 16, I was able to get a
max write speed of 1138 MB/s.  Any ideas on how we can make that faster, though?
Ideally, we''d like to get to 1.5 GB/s.

Carl

From: Liang Zhen [mailto:liang at whamcloud.com<mailto:liang at
whamcloud.com>]
Sent: Thursday, February 16, 2012 1:45 AM
To: Barberi, Carl E
Cc: ''lustre-discuss at lists.Lustre.org<mailto:lustre-discuss at
lists.Lustre.org>''
Subject: EXTERNAL: Re: [Lustre-discuss] LNET Performance Issue

Hi, I assume you are using "size=1M" for brw test right? performance
could increase if you set "concurrency" while adding brw test, i.e:
--concurrency=16

Liang

On Feb 16, 2012, at 3:30 AM, Barberi, Carl E wrote:

We are having issues with LNET performance over Infiniband.  We have a
configuration with a single MDT and six (6) OSTs.  The Lustre client I am using
to test is configured to use 6 stripes (lfs setstripe -c  6 /mnt/lustre).  When
I perform a test using the following command:

                dd if=/dev/zero of=/mnt/lustre/test.dat bs=1M count=2000

I typically get a write rate of about 815 MB/s, and we never exceed 848 MB/s. 
When I run obdfilter-survey, we easily get about 3-4GB/s write speed, but when I
run a series of lnet-selftests, the read and write rates range from 850MB/s -
875MB/s max.  I have performed the following optimizations to increase the data
rate:

On the Client:
lctl set_param osc.*.checksums=0
lctl set_param osc.*.max_dirty_mb=256

On the OSTs
lctl set_param obdfilter.*.writethrough_cache_enable=0
lctl set_param obdfilter.*.read_cache_enable=0

echo 4096 > /sys/block/<devices>/queue/nr_requests

I have also loaded the ib_sdp module, which also brought an increase in speed. 
However, we need to be able to record at no less than 1GB/s, which we cannot
achieve right now.  Any thoughts on how I can optimize LNET, which clearly seems
to be the bottleneck?

Thank you for any help you can provide,
Carl Barberi
_______________________________________________
Lustre-discuss mailing list
Lustre-discuss at lists.lustre.org<mailto:Lustre-discuss at
lists.lustre.org>
http://lists.lustre.org/mailman/listinfo/lustre-discuss

_______________________________________________
Lustre-discuss mailing list
Lustre-discuss at lists.lustre.org<mailto:Lustre-discuss at
lists.lustre.org>
http://lists.lustre.org/mailman/listinfo/lustre-discuss

_______________________________________________
Lustre-discuss mailing list
Lustre-discuss at lists.lustre.org<mailto:Lustre-discuss at
lists.lustre.org>
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Confidentiality Notice: This e-mail message, its contents and any attachments to
it are confidential to the intended recipient, and may contain information that
is privileged and/or exempt from disclosure under applicable law. If you are not
the intended recipient, please immediately notify the sender and destroy the
original e-mail message and any attachments (and any copies that may have been
made) from your system or otherwise. Any unauthorized use, copying, disclosure
or distribution of this information is strictly prohibited. Email addresses that
end with a "-c" identify the sender as a Fusion-io contractor.

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20120221/7916799e/attachment-0001.html

Isaac Huang

2012-Feb-23 19:37 UTC

head link

[Lustre-discuss] EXTERNAL: Re: LNET Performance Issue

Hi all,

I''d suggest to start from simple point to point tests. There''s
too
many variables involved in a ''dd''. Please:

- Do a native IB write test from A to B, of 1M transfers, which is the
  max payload per Lustre RPC. With native IB bandwidth test tool, I
  remember there used to be an ib_write_bw tool, or something similar.

- Do a LNet selftest brw write test, size=1M, concurrency=8 and 16, from
  A to B. Be sure to check your lnet_selftest kernel module for an old
  option called srpc_peer_credits; if it''s there, set it to 0. After
  the test is done, please run a "lctl --net o2ib conn_list" (replace
  o2ib with whatever @o2ib network in use). This command shows the
  path MTU for the connection, which might be of interest.

And let us know the results (command line options, selftest script,
and their outputs). Thanks.

- Isaac

On Tue, Feb 21, 2012 at 01:26:44PM +0000, Barberi, Carl E
wrote:> I am using QDR IB and I am not using any IB extenders.  This is a
self-contained network, with no access outside of the lab it is currently in.
> 
> Carl
> 
> From: Jeremy Filizetti [mailto:jeremy.filizetti at gmail.com]
> Sent: Monday, February 20, 2012 8:37 PM
> To: Kevin Van Maren
> Cc: Barberi, Carl E; lustre-discuss at lists.Lustre.org; isaac_huang at
xyratex.com
> Subject: Re: [Lustre-discuss] EXTERNAL: Re: LNET Performance Issue
> 
> It does seem extreme for data center IB latency but it may not be in the
data center.  The LNet write should take 2 RTT latencies, and 3 for reads so you
could double/triple those times plus any overhead.
> 
> Carl can you clarify if you are using QDR IB and/or any campus or wide area
IB extenders?
> 
> Jeremy
> On Mon, Feb 20, 2012 at 8:14 PM, Kevin Van Maren <KVanMaren at
fusionio.com<mailto:KVanMaren at fusionio.com>> wrote:
> While it''s possible the default credits (8 as I recall) is not
enough for peak performance, it seems to me that something else is wrong:
> Each 1MB RPC should take ~300uS (based on MPI/IB xfer rates of 3.2+ GB/s),
so that means there is another 400uS overhead per RPC that is not masked with 8
concurrent RPCs, in addition to the overhead masked when he increased
concurrency.  This is crazy, with a 1uS network latency.
> 
> Unless the RPCs are are being broken into tiny chunks or something -- does
lnet do single-page xfers and not use a rendezvous protocol for full-sized RPCs?
It definitely seems that something is broken when o2iblnd gets ~1/3 of the MPI
BW, given that lnd was designed for high-speed xfers.
> 
> The max_rpcs_in_flight normally needs tweaking to improve disk concurrency,
where a single client needs to drive a high queue depth. Still finding it hard
to believe 8 1MB concurrent RPCs can''t handle the network.
> 
> Kevin
> 
> 
> On Feb 20, 2012, at 5:44 PM, "Jeremy Filizetti"
<jeremy.filizetti at gmail.com<mailto:jeremy.filizetti at
gmail.com>> wrote:
> Am I reading your earlier post correctly that you have a single server
acting as the MDS and OSS?  Have you changed your peer_credits and credits for
ko2iblnd kernel module on the server and client?  You also mentioned changing
osc.*.max_dirty_mb, you probably need to adjust osc.*.max_rpcs_in_flight as
well.  Can you post your rpc stats "lctl get_param osc.*.rpc_stats"? 
I would guess they are bunching up around 7-8 if your running with the default
max_rpcs_in_flight=8.
> 
> Jeremy
> 
> On Mon, Feb 20, 2012 at 4:59 PM, Barberi, Carl E <carl.e.barberi at
lmco.com<mailto:carl.e.barberi at lmco.com>> wrote:
> Thank you.   This did help.  With the concurrency set to 16, I was able to
get a max write speed of 1138 MB/s.  Any ideas on how we can make that faster,
though?  Ideally, we''d like to get to 1.5 GB/s.
> 
> Carl
> 
> From: Liang Zhen [mailto:liang at whamcloud.com<mailto:liang at
whamcloud.com>]
> Sent: Thursday, February 16, 2012 1:45 AM
> To: Barberi, Carl E
> Cc: ''lustre-discuss at lists.Lustre.org<mailto:lustre-discuss
at lists.Lustre.org>''
> Subject: EXTERNAL: Re: [Lustre-discuss] LNET Performance Issue
> 
> Hi, I assume you are using "size=1M" for brw test right?
performance could increase if you set "concurrency" while adding brw
test, i.e: --concurrency=16
> 
> Liang
> 
> On Feb 16, 2012, at 3:30 AM, Barberi, Carl E wrote:
> 
> We are having issues with LNET performance over Infiniband.  We have a
configuration with a single MDT and six (6) OSTs.  The Lustre client I am using
to test is configured to use 6 stripes (lfs setstripe -c  6 /mnt/lustre).  When
I perform a test using the following command:
> 
>                 dd if=/dev/zero of=/mnt/lustre/test.dat bs=1M count=2000
> 
> I typically get a write rate of about 815 MB/s, and we never exceed 848
MB/s.  When I run obdfilter-survey, we easily get about 3-4GB/s write speed, but
when I run a series of lnet-selftests, the read and write rates range from
850MB/s - 875MB/s max.  I have performed the following optimizations to increase
the data rate:
> 
> On the Client:
> lctl set_param osc.*.checksums=0
> lctl set_param osc.*.max_dirty_mb=256
> 
> On the OSTs
> lctl set_param obdfilter.*.writethrough_cache_enable=0
> lctl set_param obdfilter.*.read_cache_enable=0
> 
> echo 4096 > /sys/block/<devices>/queue/nr_requests
> 
> I have also loaded the ib_sdp module, which also brought an increase in
speed.  However, we need to be able to record at no less than 1GB/s, which we
cannot achieve right now.  Any thoughts on how I can optimize LNET, which
clearly seems to be the bottleneck?
> 
> Thank you for any help you can provide,
> Carl Barberi
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org<mailto:Lustre-discuss at
lists.lustre.org>
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> 
> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org<mailto:Lustre-discuss at
lists.lustre.org>
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org<mailto:Lustre-discuss at
lists.lustre.org>
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> 
> 
> Confidentiality Notice: This e-mail message, its contents and any
attachments to it are confidential to the intended recipient, and may contain
information that is privileged and/or exempt from disclosure under applicable
law. If you are not the intended recipient, please immediately notify the sender
and destroy the original e-mail message and any attachments (and any copies that
may have been made) from your system or otherwise. Any unauthorized use,
copying, disclosure or distribution of this information is strictly prohibited.
Email addresses that end with a "-c" identify the sender as a
Fusion-io contractor.
>   
>

Lustre discuss - Feb 2012 - LNET Performance Issue

[Lustre-discuss] LNET Performance Issue

[Lustre-discuss] LNET Performance Issue

[Lustre-discuss] LNET Performance Issue

[Lustre-discuss] LNET Performance Issue

[Lustre-discuss] LNET Performance Issue

[Lustre-discuss] EXTERNAL: Re: LNET Performance Issue

[Lustre-discuss] EXTERNAL: Re: LNET Performance Issue

[Lustre-discuss] EXTERNAL: Re: LNET Performance Issue

[Lustre-discuss] EXTERNAL: Re: LNET Performance Issue

[Lustre-discuss] EXTERNAL: Re: LNET Performance Issue

[Lustre-discuss] EXTERNAL: Re: LNET Performance Issue

[Lustre-discuss] EXTERNAL: Re: LNET Performance Issue