Dai, Manhong
2008-Aug-26 01:37 UTC
[Gluster-users] help, glusterfs test caused very high tcp segment retransmission rate (Collin Douglas)
Hi Collin, Thanks a lot for your help. The interconnect is Gigabit Ethernet. I am guessing that it is caused by the router. Computing nodes that connect to this router always retransmit 5% of segments under heavy load. My test on other router has lower ratio, around 0.3%. I am about to replace the router and perform another test. Anyway this problem is not related to glusterfs at all. Although I do have another question that is about glusterfs, which I will send in a separated email. Best, Manhong Date: Mon, 25 Aug 2008 08:41:25 -0500 From: Collin Douglas <cbd at adfitech.com> Subject: Re: [Gluster-users] help, glusterfs test caused very high tcp segment retransmission rate Cc: gluster-users at gluster.org Message-ID: <48B2B685.2050603 at adfitech.com> Content-Type: text/plain; charset="ISO-8859-1" -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 This sounds a lot like a physical layer issue -- most notably a speed/duplex issue. Any time I've seen a large number of retransmits occur during heavy load, this is what I suspect. That said, I don't know what type of interconnect you're using. Tell us more about your configuration. - -Collin Dai, Manhong wrote:> > > Hi, > > > I found the aggregated IO speed is only about 100MB/s on 4 Giga-bit > Brick. This test is done over 12 computing nodes with command dd > if=/dev/zero of=bar bs=1048576 count=20480. Because our brick has very > fast local IO speed, the problem could be network. > > > Then I found computing nodes got too many retransmited segments > during test according to netstat -st. The retransmission ratio is > about 5%, but brick node has a normally low transmission rate. > > > Could some shed some light? > > Best, > Manhong-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20080825/2099f803/attachment.html>
Collin Douglas
2008-Aug-30 15:59 UTC
[Gluster-users] help, glusterfs test caused very high tcp segment retransmission rate (Collin Douglas)
I'll be interested to hear how this works for you. THanks for the update. -Collin Dai, Manhong wrote:> > Hi Collin, > > Thanks a lot for your help. The interconnect is Gigabit Ethernet. > > I am guessing that it is caused by the router. Computing nodes that > connect to this router always retransmit 5% of segments under heavy > load. My test on other router has lower ratio, around 0.3%. I am about > to replace the router and perform another test. > > Anyway this problem is not related to glusterfs at all. Although I > do have another question that is about glusterfs, which I will send in > a separated email. > > Best, > Manhong > > Date: Mon, 25 Aug 2008 08:41:25 -0500 > From: Collin Douglas <cbd at adfitech.com> > Subject: Re: [Gluster-users] help, glusterfs test caused very high tcp > segment retransmission rate > Cc: gluster-users at gluster.org > Message-ID: <48B2B685.2050603 at adfitech.com> > Content-Type: text/plain; charset="ISO-8859-1" > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > This sounds a lot like a physical layer issue -- most notably a > speed/duplex issue. Any time I've seen a large number of retransmits > occur during heavy load, this is what I suspect. > > That said, I don't know what type of interconnect you're using. Tell us > more about your configuration. > > - -Collin > > Dai, Manhong wrote: > > > > > > Hi, > > > > > > I found the aggregated IO speed is only about 100MB/s on 4 Giga-bit > > Brick. This test is done over 12 computing nodes with command dd > > if=/dev/zero of=bar bs=1048576 count=20480. Because our brick has very > > fast local IO speed, the problem could be network. > > > > > > Then I found computing nodes got too many retransmited segments > > during test according to netstat -st. The retransmission ratio is > > about 5%, but brick node has a normally low transmission rate. > > > > > > Could some shed some light? > > > > Best, > > Manhong > > ------------------------------------------------------------------------ > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20080830/121a4170/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 476 bytes Desc: not available URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20080830/121a4170/attachment.sig>