while checking out the quality of a switch, I came about a very disturbing dicovery: FreeBSD <-> Linux througput is MUCH better than FreeBSD <-> FreeBSD Setup: 2 blades in the same bladeserver, A running FreeBSD 5.4, B running Linux C is running FreeBSD 5.4 all are connected at 1gb. A -+ (FreeBSD) | B -+ (Linux) | [switch] | +---- [router] --- C (FreeBSD) A & B are on the same Vlan. iperf results: Interval Transfer Bandwidth A <=> B 0.0-10.0 sec 1.09 GBytes 939 Mbits/sec A <=> C 0.0-10.0 sec 515 MBytes 432 Mbits/sec B <=> C 0.0-10.0 sec 1.07 GBytes 918 Mbits/sec I've run the tests several times, and the numbers are very similar, so BIG Question: is there anything that can be tunned on the FreeBSD to better the throughput? danny
we need more data points - did you test tcp or udp ? who is sourcing data ? are the bandwidth symmetric (i.e. A-> same as B -> A ? cheers luigi On Tue, Jul 12, 2005 at 09:21:13AM +0300, Danny Braniss wrote:> while checking out the quality of a switch, I came about a very disturbing > dicovery: FreeBSD <-> Linux througput is MUCH better than FreeBSD <-> FreeBSD > > Setup: > 2 blades in the same bladeserver, A running FreeBSD 5.4, B running Linux > C is running FreeBSD 5.4 > all are connected at 1gb. > > A -+ (FreeBSD) > | > B -+ (Linux) > | > [switch] > | > +---- [router] --- C (FreeBSD) > A & B are on the same Vlan. > > iperf results: > Interval Transfer Bandwidth > > A <=> B 0.0-10.0 sec 1.09 GBytes 939 Mbits/sec > > A <=> C 0.0-10.0 sec 515 MBytes 432 Mbits/sec > > B <=> C 0.0-10.0 sec 1.07 GBytes 918 Mbits/sec > > I've run the tests several times, and the numbers are very similar, > so BIG Question: is there anything that can be tunned on the FreeBSD to > better the throughput? > > danny > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
> > > Are the window sizes on Linux bigger or smaller? > > > TCP window size: 16.0 KByte (default) > > smaller :-(, but increasing it does not make any change > > Hmmm... Various things that you could try (I'd try them > one by on, rather than all together): > > 1) sysctl net.inet.tcp.inflight_enable=0 > > 2) sysctl net.inet.tcp.sendspace=131072 > sysctl net.inet.tcp.recvspace=131072 > > 3) sysctl net.inet.tcp.sack.enable=0> > I'd be interested to know if any of them make much difference.combining sysctl net.inet.tcp.sendspace=131072 and sysctl net.inet.tcp.inflight.enable=0 did the trick! now can someone remind me what inflight does? and could someone explain why increasing sendspace alone did not do the trick? (i had it at 64k, which got things better, but not sufficient). thanks!> > David.
> combining > sysctl net.inet.tcp.sendspace=131072 > and > sysctl net.inet.tcp.inflight.enable=0 > > did the trick!Congratulations! But I wonder why the throughput of FreeBSD=>Linux was almost equal to that of Linux=>FreeBSD. If the settings above improves the throughput of FreeBSD=>FreeBSD, the throughput of FreeBSD=>Linux would also be improved with them. Is it improved?> now can someone remind me what inflight does?In my understanding, it tries to estimate bandwidth-delay product and tries to avoid injecting too much data segments into networks. So, if it underestimates bandwidth-delay product, throughtput may be reduced. Regards, Noritoshi Demizu
> > combining > > sysctl net.inet.tcp.sendspace=131072 > > and > > sysctl net.inet.tcp.inflight.enable=0 > > > > did the trick! > > Congratulations! But I wonder why the throughput of FreeBSD=>Linux > was almost equal to that of Linux=>FreeBSD. If the settings above > improves the throughput of FreeBSD=>FreeBSD, the throughput of > FreeBSD=>Linux would also be improved with them. Is it improved? >i did the sysctl on the client/sender :-)> > now can someone remind me what inflight does? > > In my understanding, it tries to estimate bandwidth-delay product and > tries to avoid injecting too much data segments into networks. > So, if it underestimates bandwidth-delay product, throughtput may be > reduced. > > Regards, > Noritoshi Demizu
> In the last episode (Jul 12), Danny Braniss said: > > [...] > > > You might want to apply the patch at the bottom of > > > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/75122 ; without it, new > > > connections get a random initial bandwidth. > > > > how far 'bottom' should i go? > > Search for "Final patch follows".ok, did the patches (by hand, since the patch is a bit outdated), but it didn't help. the speed up is only realized when increasing the recvspace AND disabling inflight. danny