Giovanni P. Tirloni
2003-Jun-01 05:16 UTC
Very weird network behaviour with 4.7-RELEASE-p10 (large)
Hi,
I have been doing some tests using hping2 and TCP SYN
pings targeting local and remote hosts from two FreeBSD
4.7-RELEASE-p10 and one Linux 2.4.18 host.
The three machines have the same hardware configuration
and have been running for 6 months now. The average load
isnt too high (usually 0.01 to 0.15) on the FreeBSD machines.
Here is the output from hping2 (excuse me the line wrap),
root:~# hping -S -p 80 -c 6 www.freebsd.org
HPING www.freebsd.org (fxp0 216.136.204.117): S set, 40 headers + 0 data bytes
len=46 ip=216.136.204.117 ttl=50 DF id=20997 sport=80 flags=SA seq=0 win=57344
rtt=212.6 ms
len=46 ip=216.136.204.117 ttl=50 DF id=21140 sport=80 flags=SA seq=1 win=57344
rtt=212.3 ms
len=46 ip=216.136.204.117 ttl=50 DF id=21301 sport=80 flags=SA seq=2 win=57344
rtt=212.3 ms
DUP! len=46 ip=216.136.204.117 ttl=50 DF id=21494 sport=80 flags=SA seq=0
win=57344 rtt=3207.3 ms
len=46 ip=216.136.204.117 ttl=50 DF id=21498 sport=80 flags=SA seq=3 win=57344
rtt=212.3 ms
DUP! len=46 ip=216.136.204.117 ttl=50 DF id=21650 sport=80 flags=SA seq=1
win=57344 rtt=3209.4 ms
--- www.freebsd.org hping statistic ---
5 packets tramitted, 6 packets received, -20% packet loss
round-trip min/avg/max = 212.3/1211.0/3209.4 ms
And tcpdump confirms this and shows more packets after hping2
was stopped,
root@srv0-cta:~# tcpdump -n host 216.136.204.117 and port 80
tcpdump: listening on fxp0
03:55:07.989538 200.203.183.32.1288 > 216.136.204.117.80: S
852569665:852569665(0) win 512
03:55:08.202061 216.136.204.117.80 > 200.203.183.32.1288: S
3771704683:3771704683(0) ack 85256966603:55:11.229621 216.136.204.117.80 >
200.203.183.32.1291: S 2387314094:2387314094(0) ack 2741695 win 57344 <mss
1460> (DF)
03:55:12.027408 200.203.183.32.1292 > 216.136.204.117.80: S
1532350261:1532350261(0) win 512
03:55:12.206721 216.136.204.117.80 > 200.203.183.32.1289: S
3305368856:3305368856(0) ack 1218353109 win 57344 <mss 1460> (DF)
03:55:12.239377 216.136.204.117.80 > 200.203.183.32.1292: S
3378516967:3378516967(0) ack 1532350262 win 57344 <mss 1460> (DF)
03:55:13.216677 216.136.204.117.80 > 200.203.183.32.1290: S
2652913389:2652913389(0) ack 422129860 win 57344 <mss 1460> (DF)
03:55:14.226573 216.136.204.117.80 > 200.203.183.32.1291: S
2387314094:2387314094(0) ack 2741695 win 57344 <mss 1460> (DF)
03:55:15.236648 216.136.204.117.80 > 200.203.183.32.1292: S
3378516967:3378516967(0) ack 1532350262 win 57344 <mss 1460> (DF)
03:55:17.196452 216.136.204.117.80 > 200.203.183.32.1288: S
3771704683:3771704683(0) ack 852569666 win 57344 <mss 1460> (DF)
03:55:18.206412 216.136.204.117.80 > 200.203.183.32.1289: S
3305368856:3305368856(0) ack 1218353109 win 57344 <mss 1460> (DF)
03:55:19.216406 216.136.204.117.80 > 200.203.183.32.1290: S
2652913389:2652913389(0) ack 422129860 win 57344 <mss 1460> (DF)
03:55:20.226331 216.136.204.117.80 > 200.203.183.32.1291: S
2387314094:2387314094(0) ack 2741695 win 57344 <mss 1460> (DF)
03:55:21.236273 216.136.204.117.80 > 200.203.183.32.1292: S
3378516967:3378516967(0) ack 1532350262 win 57344 <mss 1460> (DF)
03:55:29.195990 216.136.204.117.80 > 200.203.183.32.1288: S
3771704683:3771704683(0) ack 852569666 win 57344 <mss 1460> (DF)
03:55:30.205987 216.136.204.117.80 > 200.203.183.32.1289: S
3305368856:3305368856(0) ack 1218353109 win 57344 <mss 1460> (DF)
03:55:31.216041 216.136.204.117.80 > 200.203.183.32.1290: S
2652913389:2652913389(0) ack 422129860 win 57344 <mss 1460> (DF)
03:55:32.225922 216.136.204.117.80 > 200.203.183.32.1291: S
2387314094:2387314094(0) ack 2741695 win 57344 <mss 1460> (DF)
03:55:33.235869 216.136.204.117.80 > 200.203.183.32.1292: S
3378516967:3378516967(0) ack 1532350262 win 57344 <mss 1460> (DF)
Another different output from hping (strange rtt's),
root@srv0-cta:~# hping -S -p 80 -c 8 www.freebsd.org
HPING www.freebsd.org (fxp0 216.136.204.117): S set, 40 headers + 0 data bytes
len=46 ip=216.136.204.117 ttl=50 DF id=38289 sport=80 flags=SA seq=0 win=57344
rtt=212.5 ms
len=46 ip=216.136.204.117 ttl=50 DF id=38291 sport=80 flags=SA seq=0 win=57344
rtt=0.0 ms
len=46 ip=216.136.204.117 ttl=50 DF id=38371 sport=80 flags=SA seq=1 win=57344
rtt=211.9 ms
len=46 ip=216.136.204.117 ttl=50 DF id=38372 sport=80 flags=SA seq=0 win=57344
rtt=0.0 ms
len=46 ip=216.136.204.117 ttl=50 DF id=38404 sport=80 flags=SA seq=2 win=57344
rtt=211.9 ms
len=46 ip=216.136.204.117 ttl=50 DF id=38405 sport=80 flags=SA seq=0 win=57344
rtt=0.0 ms
DUP! len=46 ip=216.136.204.117 ttl=50 DF id=38687 sport=80 flags=SA seq=0
win=57344 rtt=3207.3 ms
len=46 ip=216.136.204.117 ttl=50 DF id=38702 sport=80 flags=SA seq=3 win=57344
rtt=212.0 ms
--- www.freebsd.org hping statistic ---
4 packets tramitted, 8 packets received, -100% packet loss
round-trip min/avg/max = 211.9/811.1/3207.3 ms
And the tcpdump output,
04:00:57.419466 216.136.204.117.80 > 200.203.183.32.2381: S
1076154568:1076154568(0) ack 1439276851 win 57344 <mss 1460> (DF)
04:00:58.201940 200.203.183.32.1995 > 216.136.204.117.80: S
264430648:264430648(0) win 512
04:00:58.414397 216.136.204.117.80 > 200.203.183.32.1995: S
2379854270:2379854270(0) ack 264430649 win 57344 <mss 1460> (DF)
04:00:58.429518 216.136.204.117.80 > 200.203.183.32.2382: S
3830791941:3830791941(0) ack 451178042 win 57344 <mss 1460> (DF)
04:00:59.206642 200.203.183.32.1996 > 216.136.204.117.80: S
1522574199:1522574199(0) win 512
04:00:59.418526 216.136.204.117.80 > 200.203.183.32.1996: S
3804689919:3804689919(0) ack 1522574200 win 57344 <mss 1460> (DF)
04:00:59.439330 216.136.204.117.80 > 200.203.183.32.2383: S
1160193874:1160193874(0) ack 1028452909 win 57344 <mss 1460> (DF)
04:01:00.216649 200.203.183.32.1997 > 216.136.204.117.80: S
442375938:442375938(0) win 512
04:01:00.428557 216.136.204.117.80 > 200.203.183.32.1997: S
3901380776:3901380776(0) ack 442375939 win 57344 <mss 1460> (DF)
04:01:00.449324 216.136.204.117.80 > 200.203.183.32.2384: S
3552821723:3552821723(0) ack 1069527292 win 57344 <mss 1460> (DF)
04:01:01.226660 200.203.183.32.1998 > 216.136.204.117.80: S
240094281:240094281(0) win 512
04:01:01.409234 216.136.204.117.80 > 200.203.183.32.1995: S
2379854270:2379854270(0) ack 264430649 win 57344 <mss 1460> (DF)
04:01:01.438601 216.136.204.117.80 > 200.203.183.32.1998: S
4153685029:4153685029(0) ack 240094282 win 57344 <mss 1460> (DF)
04:01:01.529193 216.136.204.117.80 > 200.203.183.32.2065: S
4017739092:4017739092(0) ack 1616120023 win 57344 <mss 1460> (DF)
04:01:02.409213 216.136.204.117.80 > 200.203.183.32.1996: S
3804689919:3804689919(0) ack 1522574200 win 57344 <mss 1460> (DF)
04:01:02.529165 216.136.204.117.80 > 200.203.183.32.2066: S
754183561:754183561(0) ack 1231654702 win 57344 <mss 1460> (DF)
04:01:03.419286 216.136.204.117.80 > 200.203.183.32.1997: S
3901380776:3901380776(0) ack 442375939 win 57344 <mss 1460> (DF)
04:01:03.539156 216.136.204.117.80 > 200.203.183.32.2067: S
1059652438:1059652438(0) ack 2093248177 win 57344 <mss 1460> (DF)
04:01:04.429131 216.136.204.117.80 > 200.203.183.32.1998: S
4153685029:4153685029(0) ack 240094282 win 57344 <mss 1460> (DF)
04:01:04.549163 216.136.204.117.80 > 200.203.183.32.2068: S
3136323399:3136323399(0) ack 723495504 win 57344 <mss 1460> (DF)
04:01:07.409090 216.136.204.117.80 > 200.203.183.32.1995: S
2379854270:2379854270(0) ack 264430649 win 57344 <mss 1460> (DF)
04:01:08.408993 216.136.204.117.80 > 200.203.183.32.1996: S
3804689919:3804689919(0) ack 1522574200 win 57344 <mss 1460> (DF)
04:01:09.418980 216.136.204.117.80 > 200.203.183.32.1997: S
3901380776:3901380776(0) ack 442375939 win 57344 <mss 1460> (DF)
04:01:10.429020 216.136.204.117.80 > 200.203.183.32.1998: S
4153685029:4153685029(0) ack 240094282 win 57344 <mss 1460> (DF)
04:01:19.408629 216.136.204.117.80 > 200.203.183.32.1995: S
2379854270:2379854270(0) ack 264430649 win 57344 <mss 1460> (DF)
04:01:20.408544 216.136.204.117.80 > 200.203.183.32.1996: S
3804689919:3804689919(0) ack 1522574200 win 57344 <mss 1460> (DF)
04:01:21.418566 216.136.204.117.80 > 200.203.183.32.1997: S
3901380776:3901380776(0) ack 442375939 win 57344 <mss 1460> (DF)
04:01:22.428427 216.136.204.117.80 > 200.203.183.32.1998: S
4153685029:4153685029(0) ack 240094282 win 57344 <mss 1460> (DF)
Some users have complaint about server timeouts since Monday (but
I havent experienced the problems they seem to be having). At first
I thought it was the switch but I don't see errors from the Linux
machine.
That happens when pinging (with hping -S -p 80),
bsd1:fxp0 -> bsd2
bsd2:fxp0 -> bsd1
bsd1:lo0 -> bsd1:lo0
bsd2:lo0 -> bsd2:lo0
bsd1:fxp0 -> remote
bsd2:fxp0 -> remote
bsd1:fxp0 -> linux
Any help is welcome and sorry if this email doesn't have all the
information needed to solve this problem. I'll be happy to provide
more details and do more tests if needed.
Here is the output from ifconfig fxp0,
fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
inet 200.203.183.32 netmask 0xffffffc0 broadcast 200.203.183.63
inet 200.203.183.33 netmask 0xffffffff broadcast 200.203.183.33
inet 200.203.183.37 netmask 0xffffffff broadcast 200.203.183.37
ether 00:07:e9:ad:2a:ab
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
Someone mentioned cvsup'ing to 4.8-STABLE but I'm afraid of doing
so because those two boxes are very critical.
--
Giovanni P. Tirloni
http://www.tirloni.org
Crist J. Clark
2003-Jun-04 07:10 UTC
Very weird network behaviour with 4.7-RELEASE-p10 (large)
On Sun, Jun 01, 2003 at 04:12:55AM -0300, Giovanni P. Tirloni wrote:> Hi, > > I have been doing some tests using hping2 and TCP SYN > pings targeting local and remote hosts from two FreeBSD > 4.7-RELEASE-p10 and one Linux 2.4.18 host. > > The three machines have the same hardware configuration > and have been running for 6 months now. The average load > isnt too high (usually 0.01 to 0.15) on the FreeBSD machines. > > Here is the output from hping2 (excuse me the line wrap), > > root:~# hping -S -p 80 -c 6 www.freebsd.org > HPING www.freebsd.org (fxp0 216.136.204.117): S set, 40 headers + 0 data bytes > len=46 ip=216.136.204.117 ttl=50 DF id=20997 sport=80 flags=SA seq=0 win=57344 rtt=212.6 ms > len=46 ip=216.136.204.117 ttl=50 DF id=21140 sport=80 flags=SA seq=1 win=57344 rtt=212.3 ms > len=46 ip=216.136.204.117 ttl=50 DF id=21301 sport=80 flags=SA seq=2 win=57344 rtt=212.3 ms > DUP! len=46 ip=216.136.204.117 ttl=50 DF id=21494 sport=80 flags=SA seq=0 win=57344 rtt=3207.3 ms > len=46 ip=216.136.204.117 ttl=50 DF id=21498 sport=80 flags=SA seq=3 win=57344 rtt=212.3 ms > DUP! len=46 ip=216.136.204.117 ttl=50 DF id=21650 sport=80 flags=SA seq=1 win=57344 rtt=3209.4 ms > > --- www.freebsd.org hping statistic --- > 5 packets tramitted, 6 packets received, -20% packet loss > round-trip min/avg/max = 212.3/1211.0/3209.4 ms[snip]> Any help is welcome and sorry if this email doesn't have all the > information needed to solve this problem. I'll be happy to provide > more details and do more tests if needed.Errr... What problem? The hping2 and tcpdump output all look normal. You send a SYN to a remote host, it keeps sending SYN-ACKs back trying to establish the connection. You get long "RTTs" due to the time elapsed between the initial SYN and the continuing responses. That RTT has nothing to do with how fast a packet crosses the network. -- Crist J. Clark | cjclark@alum.mit.edu | cjclark@jhu.edu http://people.freebsd.org/~cjc/ | cjc@freebsd.org