freebsd stable - Feb 2010 - best practice to watch TCP parms of established sockets

Hello,

while doing some ZFS tests with RELENG_8 I recognized a mysterious 
performace drop after an hour uptime.
Now my first idea is to compare MSS and windows sizes before and after 
the performance drop.
How do I best capture them? tdpcump? It's GbE linkspeed...
Or is netstat capable to show these values? Or is there any way to read 
out the values stored in tcp.hostcache?

Thanks,

-Harry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: OpenPGP digital signature
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20100217/0b15104d/signature.pgp

Hi--

On Feb 17, 2010, at 8:03 AM, Harald Schmalzbauer wrote:
> while doing some ZFS tests with RELENG_8 I recognized a mysterious
performace drop after an hour uptime.
> Now my first idea is to compare MSS and windows sizes before and after the
performance drop.
> How do I best capture them? tdpcump? It's GbE linkspeed...
It seems more likely that ZFS is running into slowdowns from resource
contention, memory fragmentation, etc than your network would suddenly drop out,
but tcpdump -w outfile.pcap is a good method of looking....

Regards,
-- 
-Chuck

Stephen Hurd schrieb am 18.02.2010 17:01 (localtime):
> Harald Schmalzbauer wrote:
>> Some experimental results:
>> When rsyncing with windows, and FreeBSD is receiver, I see the same 
>> ACK ever two segemnts, but speed is at 72MB/s.
>> When FreeBSD is sender and Windows is receiver, it looks more I 
>> expected. There are about 20 data segments before a ACK is returned. 
>> And there are  TCP Window Update Segments, reflecting smaller receiver 
>> buffers on the windows side. But this happens at a throughput of 
>> 82MB/s!!! So the windows machine is behaving like I understand the TCP 
>> flow control.
>> Any explanation why the FreeBSD machine seems to ignore window size?
> 
> IIRC, the delayed ACK RFC requires an ACK at least every second segment.
Good hint, but disabling leads to ACK after every single data segment ?!?

Thanks,

-Harry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: OpenPGP digital signature
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20100218/6560805c/signature.pgp

Nikos Ntarmos schrieb am 24.02.2010 16:37 (localtime):
> On Fri, Feb 19, 2010 at 11:55:39AM +0200, Nikos Ntarmos wrote:
>> On Thu, Feb 18, 2010 at 10:41:28PM +0100, Harald Schmalzbauer wrote:
>>> Kevin Oberman schrieb am 18.02.2010 20:23 (localtime):
>>> ...
>>>> window allows for many packets to be in flight and with a 3
Gbps flow,
>>>> that is a LOT of data. While an ACK is sent every two packets
of
>>>> received data, the transmitting side does not wait for the
ACKs. It
>>> ...
>>>> That is a VERY simple and incomplete explanation of what is
happening
>>>> with the window, but most of that is irrelevant in local
transfers with
>>> Thanks a lot, then I understood it at least half correct ;) My
>>> missunderstanding was that I thought the receiver would reduce
>>> ACKs... Now I know more :)
>>> But unfortunately that makes it more mysterious where the
throughput
>>> problem lies...
>>>
>>> Thanks to everyone so far,
>> Hi there.
>>
>> This is a long shot but have you tried disabling checksum and
>> segmentation offloading? I've found that they cause trouble with
some
>> NICs. FYI on FreeBSD the first is done through 'ifconfig
-rxcsum' while
>> the latter through 'sysctl net.inet.tcp.tso=0'. If you try this
out,
>> remember to disable these features on both communicating boxes for the
>> period of the test, just to be sure that it's not the other box
causing
>> these issues. You mentioned samba so if one of them is a win32 box, you
>> can access these settings through the hardware options of your NIC.
> 
> Hi again.
> 
> Just a friendly nudge. :) Did you find the root of these issues yet?
I don't have solid conclusions. But I ruled out some things.
First, the em driver has no problems in my setup. Neither disabling 
rx/txcsum nor disabling TSO made any differenz in trhoughput, though no 
direct recognizable load behaviour on my 2x3GHz machine. Mybe checsum 
offload is beneficial on slower machines.
One major problem was that one of the two FreeBSD Boxes was on a VMware 
which slowed things down a bit, although the FreeBSD System showed 
plenty free resources.

Disabling rfc1323 defnetily increases the throughput on gigabit 
ethernet. I can rsync between two (native) FreeBSD machines with 72-92 
MB/s averaging at 80+MB/s.
What I couldn't investigeate yet is why I always get 10% more throughput 
when one side is windwos, no matter which direction, no matter what 
application (cifs, rsync, ftp).
Another big point on my todo list is to find out why tcp.inflight brakes 
my webserver downloads really often to less than a quarter of the 
available bandwith (client bw, server bw is 100mb). I saw many 10mb/s E3 
pipes with 15ms delay, but limited transfer rates to 200kb/s. Disabling 
tcp.inflight opens that brake.

Since I could only capture "bad IP length" packets with checsum 
offloading enabled on the em, I guess it does also IP checksum 
offloading, not only TCP. I'll have to set up a port mirroring test bed 
to get the line packets. I hope I'll find the rfc1323 slowdown and the 
difference between FreeBSD clients and Windows clients. But at the 
moment I don't have spare equipment.

Greets,

-Harry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: OpenPGP digital signature
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20100225/0b4f2ff4/signature.pgp