Lustre discuss - Aug 2009 - Lustre featured on podcast (HT: Andreas Dilger)

Thanks to Andreas for taking an hour out to talk with Jeff Squyres and  
myself (Brock Palen) about the Lustre cluster filesystem on our  
podcast www.rce-cast.com,

You can find the whole show at:
http://www.rce-cast.com/index.php/Podcast/rce-14-lustre-cluster-filesystem.html

Thanks again!
If any of you have requests of topics you would like to hear please  
let me know!

Brock Palen
www.umich.edu/~brockp
Center for Advanced Computing
brockp at umich.edu
(734)936-1985

Very nice.

15:54, what is "Nagle" ?

He didn''t say anything about SNS, but changeLogs seems very promising!



On Mon, Aug 3, 2009 at 8:55 AM, Brock Palen<brockp at umich.edu>
wrote:
> Thanks to Andreas for taking an hour out to talk with Jeff Squyres and
> myself (Brock Palen) about the Lustre cluster filesystem on our
> podcast www.rce-cast.com,
>
> You can find the whole show at:
>
http://www.rce-cast.com/index.php/Podcast/rce-14-lustre-cluster-filesystem.html
>
> Thanks again!
> If any of you have requests of topics you would like to hear please
> let me know!
>
> Brock Palen
> www.umich.edu/~brockp
> Center for Advanced Computing
> brockp at umich.edu
> (734)936-1985
>
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>

http://en.wikipedia.org/wiki/Nagle%27s_algorithm

Looks like you intentionally hold up data to try to make fatter  
payloads in packets so they are not 99% header/crc data.  Sounds like  
a way to make latency bad.

Brock Palen
www.umich.edu/~brockp
Center for Advanced Computing
brockp at umich.edu
(734)936-1985



On Aug 3, 2009, at 8:20 PM, Mag Gam wrote:
> Very nice.
>
> 15:54, what is "Nagle" ?
>
> He didn''t say anything about SNS, but changeLogs seems very
promising!
>
>
>
> On Mon, Aug 3, 2009 at 8:55 AM, Brock Palen<brockp at umich.edu>
wrote:
>> Thanks to Andreas for taking an hour out to talk with Jeff Squyres  
>> and
>> myself (Brock Palen) about the Lustre cluster filesystem on our
>> podcast www.rce-cast.com,
>>
>> You can find the whole show at:
>>
http://www.rce-cast.com/index.php/Podcast/rce-14-lustre-cluster-filesystem.html
>>
>> Thanks again!
>> If any of you have requests of topics you would like to hear please
>> let me know!
>>
>> Brock Palen
>> www.umich.edu/~brockp
>> Center for Advanced Computing
>> brockp at umich.edu
>> (734)936-1985
>>
>>
>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>
>
>

eitherway, good job on the interview. Andres was  a sport.  try to
interview more Lustre developers :-)



On Mon, Aug 3, 2009 at 8:35 PM, Brock Palen<brockp at umich.edu>
wrote:
> http://en.wikipedia.org/wiki/Nagle%27s_algorithm
>
> Looks like you intentionally hold up data to try to make fatter payloads in
> packets so they are not 99% header/crc data. ?Sounds like a way to make
> latency bad.
>
> Brock Palen
> www.umich.edu/~brockp
> Center for Advanced Computing
> brockp at umich.edu
> (734)936-1985
>
>
>
> On Aug 3, 2009, at 8:20 PM, Mag Gam wrote:
>
>> Very nice.
>>
>> 15:54, what is "Nagle" ?
>>
>> He didn''t say anything about SNS, but changeLogs seems very
promising!
>>
>>
>>
>> On Mon, Aug 3, 2009 at 8:55 AM, Brock Palen<brockp at umich.edu>
wrote:
>>>
>>> Thanks to Andreas for taking an hour out to talk with Jeff Squyres
and
>>> myself (Brock Palen) about the Lustre cluster filesystem on our
>>> podcast www.rce-cast.com,
>>>
>>> You can find the whole show at:
>>>
>>>
http://www.rce-cast.com/index.php/Podcast/rce-14-lustre-cluster-filesystem.html
>>>
>>> Thanks again!
>>> If any of you have requests of topics you would like to hear please
>>> let me know!
>>>
>>> Brock Palen
>>> www.umich.edu/~brockp
>>> Center for Advanced Computing
>>> brockp at umich.edu
>>> (734)936-1985
>>>
>>>
>>>
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>>
>>
>>
>
>

Yes, originally designed so multiple send() calls with small data have  
a chance to be combined by TCP before being sent over the network --  
improve behavior of applications doing small writes.

Setting tcp_nodelay disables Nagle, as the additional latency can hurt  
some interactive session performance.

Kevin


On Aug 4, 2009, at 10:35 AM, Brock Palen <brockp at umich.edu> wrote:
> http://en.wikipedia.org/wiki/Nagle%27s_algorithm
>
> Looks like you intentionally hold up data to try to make fatter
> payloads in packets so they are not 99% header/crc data.  Sounds like
> a way to make latency bad.
>
> Brock Palen
> www.umich.edu/~brockp
> Center for Advanced Computing
> brockp at umich.edu
> (734)936-1985
>
>
>
> On Aug 3, 2009, at 8:20 PM, Mag Gam wrote:
>
>> Very nice.
>>
>> 15:54, what is "Nagle" ?
>>
>> He didn''t say anything about SNS, but changeLogs seems very  
>> promising!
>>
>>
>>
>> On Mon, Aug 3, 2009 at 8:55 AM, Brock Palen<brockp at umich.edu>
wrote:
>>> Thanks to Andreas for taking an hour out to talk with Jeff Squyres
>>> and
>>> myself (Brock Palen) about the Lustre cluster filesystem on our
>>> podcast www.rce-cast.com,
>>>
>>> You can find the whole show at:
>>>
http://www.rce-cast.com/index.php/Podcast/rce-14-lustre-cluster-filesystem.html
>>>
>>> Thanks again!
>>> If any of you have requests of topics you would like to hear please
>>> let me know!
>>>
>>> Brock Palen
>>> www.umich.edu/~brockp
>>> Center for Advanced Computing
>>> brockp at umich.edu
>>> (734)936-1985
>>>
>>>
>>>
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>>
>>
>>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss

its really bizzare because you can read all the manuals in the world
and read all the responses on the mailing list, but 30 mins of a good
developers time is priceless. I never even knew lnet didn''t use tcp!
Thats great stuff


On Mon, Aug 3, 2009 at 10:32 PM, Kevin Van Maren<Kevin.Vanmaren at
sun.com> wrote:
> Yes, originally designed so multiple send() calls with small data have a
> chance to be combined by TCP before being sent over the network -- improve
> behavior of applications doing small writes.
>
> Setting tcp_nodelay disables Nagle, as the additional latency can hurt some
> interactive session performance.
>
> Kevin
>
>
> On Aug 4, 2009, at 10:35 AM, Brock Palen <brockp at umich.edu> wrote:
>
>> http://en.wikipedia.org/wiki/Nagle%27s_algorithm
>>
>> Looks like you intentionally hold up data to try to make fatter
>> payloads in packets so they are not 99% header/crc data. ?Sounds like
>> a way to make latency bad.
>>
>> Brock Palen
>> www.umich.edu/~brockp
>> Center for Advanced Computing
>> brockp at umich.edu
>> (734)936-1985
>>
>>
>>
>> On Aug 3, 2009, at 8:20 PM, Mag Gam wrote:
>>
>>> Very nice.
>>>
>>> 15:54, what is "Nagle" ?
>>>
>>> He didn''t say anything about SNS, but changeLogs seems
very promising!
>>>
>>>
>>>
>>> On Mon, Aug 3, 2009 at 8:55 AM, Brock Palen<brockp at
umich.edu> wrote:
>>>>
>>>> Thanks to Andreas for taking an hour out to talk with Jeff
Squyres
>>>> and
>>>> myself (Brock Palen) about the Lustre cluster filesystem on our
>>>> podcast www.rce-cast.com,
>>>>
>>>> You can find the whole show at:
>>>>
>>>>
http://www.rce-cast.com/index.php/Podcast/rce-14-lustre-cluster-filesystem.html
>>>>
>>>> Thanks again!
>>>> If any of you have requests of topics you would like to hear
please
>>>> let me know!
>>>>
>>>> Brock Palen
>>>> www.umich.edu/~brockp
>>>> Center for Advanced Computing
>>>> brockp at umich.edu
>>>> (734)936-1985
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Lustre-discuss mailing list
>>>> Lustre-discuss at lists.lustre.org
>>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>>>
>>>
>>>
>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>

Yes, it hold up any packet that is less than the size of the header, and
forwards multiple.




----- Original Message -----
From: Brock Palen [brockp at umich.edu]
Sent: 08/03/2009 08:35 PM AST
To: Mag Gam <magawake at gmail.com>
Cc: lustre-discuss discuss <lustre-discuss at lists.lustre.org>
Subject: Re: [Lustre-discuss] Lustre featured on podcast (HT: Andreas Dilger)



http://en.wikipedia.org/wiki/Nagle%27s_algorithm

Looks like you intentionally hold up data to try to make fatter  
payloads in packets so they are not 99% header/crc data.  Sounds like  
a way to make latency bad.

Brock Palen
www.umich.edu/~brockp
Center for Advanced Computing
brockp at umich.edu
(734)936-1985



On Aug 3, 2009, at 8:20 PM, Mag Gam wrote:
> Very nice.
>
> 15:54, what is "Nagle" ?
>
> He didn''t say anything about SNS, but changeLogs seems very
promising!
>
>
>
> On Mon, Aug 3, 2009 at 8:55 AM, Brock Palen<brockp at umich.edu>
wrote:
>> Thanks to Andreas for taking an hour out to talk with Jeff Squyres  
>> and
>> myself (Brock Palen) about the Lustre cluster filesystem on our
>> podcast www.rce-cast.com,
>>
>> You can find the whole show at:
>>
http://www.rce-cast.com/index.php/Podcast/rce-14-lustre-cluster-filesystem.html
>>
>> Thanks again!
>> If any of you have requests of topics you would like to hear please
>> let me know!
>>
>> Brock Palen
>> www.umich.edu/~brockp
>> Center for Advanced Computing
>> brockp at umich.edu
>> (734)936-1985
>>
>>
>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>
>
>
_______________________________________________
Lustre-discuss mailing list
Lustre-discuss at lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

so, why

 options lnet networks=tcp

What exactly does that do?


On Mon, Aug 3, 2009 at 10:57 PM, Peter J Milanese<PMilanese at nypl.org>
wrote:
> Yes, it hold up any packet that is less than the size of the header, and
forwards multiple.
>
>
>
>
> ----- Original Message -----
> From: Brock Palen [brockp at umich.edu]
> Sent: 08/03/2009 08:35 PM AST
> To: Mag Gam <magawake at gmail.com>
> Cc: lustre-discuss discuss <lustre-discuss at lists.lustre.org>
> Subject: Re: [Lustre-discuss] Lustre featured on podcast (HT: Andreas
Dilger)
>
>
>
> http://en.wikipedia.org/wiki/Nagle%27s_algorithm
>
> Looks like you intentionally hold up data to try to make fatter
> payloads in packets so they are not 99% header/crc data. ?Sounds like
> a way to make latency bad.
>
> Brock Palen
> www.umich.edu/~brockp
> Center for Advanced Computing
> brockp at umich.edu
> (734)936-1985
>
>
>
> On Aug 3, 2009, at 8:20 PM, Mag Gam wrote:
>
>> Very nice.
>>
>> 15:54, what is "Nagle" ?
>>
>> He didn''t say anything about SNS, but changeLogs seems very
promising!
>>
>>
>>
>> On Mon, Aug 3, 2009 at 8:55 AM, Brock Palen<brockp at umich.edu>
wrote:
>>> Thanks to Andreas for taking an hour out to talk with Jeff Squyres
>>> and
>>> myself (Brock Palen) about the Lustre cluster filesystem on our
>>> podcast www.rce-cast.com,
>>>
>>> You can find the whole show at:
>>>
http://www.rce-cast.com/index.php/Podcast/rce-14-lustre-cluster-filesystem.html
>>>
>>> Thanks again!
>>> If any of you have requests of topics you would like to hear please
>>> let me know!
>>>
>>> Brock Palen
>>> www.umich.edu/~brockp
>>> Center for Advanced Computing
>>> brockp at umich.edu
>>> (734)936-1985
>>>
>>>
>>>
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>>
>>
>>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>

On Aug 03, 2009  22:33 -0400, Mag Gam wrote:
> its really bizzare because you can read all the manuals in the world
> and read all the responses on the mailing list, but 30 mins of a good
> developers time is priceless. I never even knew lnet didn''t use
tcp!
To be more clear - TCP isn''t used on IB, Elan, Myrinet, Cray networks.
On the socklnd (TCP sockets over IP, called tcplnd on some platforms)
it IS used.   I''m just listening to the podcast to see what it is I
said...
> On Mon, Aug 3, 2009 at 10:32 PM, Kevin Van Maren<Kevin.Vanmaren at
sun.com> wrote:
> > Yes, originally designed so multiple send() calls with small data have
a
> > chance to be combined by TCP before being sent over the network --
improve
> > behavior of applications doing small writes.
> >
> > Setting tcp_nodelay disables Nagle, as the additional latency can hurt
some
> > interactive session performance.
> >
> > Kevin
> >
> >
> > On Aug 4, 2009, at 10:35 AM, Brock Palen <brockp at umich.edu>
wrote:
> >
> >> http://en.wikipedia.org/wiki/Nagle%27s_algorithm
> >>
> >> Looks like you intentionally hold up data to try to make fatter
> >> payloads in packets so they are not 99% header/crc data. ?Sounds
like
> >> a way to make latency bad.
> >>
> >> Brock Palen
> >> www.umich.edu/~brockp
> >> Center for Advanced Computing
> >> brockp at umich.edu
> >> (734)936-1985
> >>
> >>
> >>
> >> On Aug 3, 2009, at 8:20 PM, Mag Gam wrote:
> >>
> >>> Very nice.
> >>>
> >>> 15:54, what is "Nagle" ?
> >>>
> >>> He didn''t say anything about SNS, but changeLogs
seems very promising!
> >>>
> >>>
> >>>
> >>> On Mon, Aug 3, 2009 at 8:55 AM, Brock Palen<brockp at
umich.edu> wrote:
> >>>>
> >>>> Thanks to Andreas for taking an hour out to talk with Jeff
Squyres
> >>>> and
> >>>> myself (Brock Palen) about the Lustre cluster filesystem
on our
> >>>> podcast www.rce-cast.com,
> >>>>
> >>>> You can find the whole show at:
> >>>>
> >>>>
http://www.rce-cast.com/index.php/Podcast/rce-14-lustre-cluster-filesystem.html
> >>>>
> >>>> Thanks again!
> >>>> If any of you have requests of topics you would like to
hear please
> >>>> let me know!
> >>>>
> >>>> Brock Palen
> >>>> www.umich.edu/~brockp
> >>>> Center for Advanced Computing
> >>>> brockp at umich.edu
> >>>> (734)936-1985
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Lustre-discuss mailing list
> >>>> Lustre-discuss at lists.lustre.org
> >>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> >>>>
> >>>
> >>>
> >>
> >> _______________________________________________
> >> Lustre-discuss mailing list
> >> Lustre-discuss at lists.lustre.org
> >> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> >
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

On Aug 03, 2009  08:55 -0400, Brock Palen wrote:
> Thanks to Andreas for taking an hour out to talk with Jeff Squyres and  
> myself (Brock Palen) about the Lustre cluster filesystem on our  
> podcast www.rce-cast.com,
> 
> You can find the whole show at:
>
http://www.rce-cast.com/index.php/Podcast/rce-14-lustre-cluster-filesystem.html
Sigh, I realize I totally omitted a TON of stuff about how Lustre is
tuned for high bandwidth/performance, and focussed exclusively on one
tiny part of the network stack, due to the previous topic.  Many things,
like zero data copies, large request sizes from the client down to the
disk, efficient network protocol, etc. are all probably more important
than the low-latency message passing implementation.
> Thanks again!
> If any of you have requests of topics you would like to hear please  
> let me know!
> 
> Brock Palen
> www.umich.edu/~brockp
> Center for Advanced Computing
> brockp at umich.edu
> (734)936-1985
> 
> 
> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

On Tue, 2009-08-04 at 11:17 -0600, Andreas Dilger wrote:
> 
> To be more clear - TCP isn''t used on IB, Elan, Myrinet, Cray
networks.
> On the socklnd (TCP sockets over IP, called tcplnd on some platforms)
tcplnd or socklnd?  Generally it''s the latter, but I wonder if you are
referring to some platform I am not familiar with.

b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090804/6ccf8c4c/attachment.bin

On Aug 04, 2009  14:35 -0400, Brian J. Murrell wrote:
> On Tue, 2009-08-04 at 11:17 -0600, Andreas Dilger wrote:
> > To be more clear - TCP isn''t used on IB, Elan, Myrinet, Cray
networks.
> > On the socklnd (TCP sockets over IP, called tcplnd on some platforms)
> 
> tcplnd or socklnd?  Generally it''s the latter, but I wonder if you
are
> referring to some platform I am not familiar with.
My bad.  The user-space TCP interface used to be called tcplnd, but I
don''t see that anymore.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Either way Andres. We appreciate the talk. My masters OS design class
talked about this for 45 mins today :-) You are a celebrity.


On Tue, Aug 4, 2009 at 2:35 PM, Brian J. Murrell<Brian.Murrell at sun.com>
wrote:
> On Tue, 2009-08-04 at 11:17 -0600, Andreas Dilger wrote:
>>
>> To be more clear - TCP isn''t used on IB, Elan, Myrinet, Cray
networks.
>> On the socklnd (TCP sockets over IP, called tcplnd on some platforms)
>
> tcplnd or socklnd? ?Generally it''s the latter, but I wonder if you
are
> referring to some platform I am not familiar with.
>
> b.
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>