Hi all,
I've got a persistent problem with my LAN. I'm running a FreeBSD
8.0
box as a home server performing the following functions for wired and
wireless networks: router; firewall; DHCP server; and file server. For what
it's worth, I've got ZFS up and running as the main filesystem. The
recurring issue is that file transfers from the FreeBSD box to computers on
the wired network (gigabit) start out fast and then become agonizingly
slow. I'm sharing home directories over Samba, and those transfers work
briefly and then tail off to a few kilobytes per second. The failure is
somewhat predicatable in that it tends to happen once a few hundred
megabytes have been transferred. I've swapped out hardware, I've
Googled
extensively, and all of the (possibly benign) error messages that I've found
have been eliminated. I'm happy to post logs, configs, etc., and I'd
appreciate any help with a diagnosis. For the moment:
> uname -a
FreeBSD EightOh.dew.home 8.0-RELEASE-p2 FreeBSD 8.0-RELEASE-p2 #0: Tue Jan
5 21:11:58 UTC 2010
root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
amd64
> ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:0e:0c:b7:71:44
inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
ral0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 2290
ether 00:1f:1f:3f:76:f3
media: IEEE 802.11 Wireless Ethernet autoselect mode 11g <hostap>
status: running
nfe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 00:01:29:d4:2d:6b
inet XXX.XXX.XXX.XXX netmask 0xfffffc00 broadcast 255.255.255.255
media: Ethernet autoselect (100baseTX <full-duplex,flag0,flag1>)
status: active
plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1500
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff000000
wlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
ether 00:1f:1f:3f:76:f3
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
media: IEEE 802.11 Wireless Ethernet autoselect mode 11g <hostap>
status: running
ssid FreeBSD_AP channel 7 (2442 Mhz 11g) bssid 00:1f:1f:3f:76:f3
country US authmode WPA privacy MIXED deftxkey 2 TKIP 2:128-bit
txpower 0 scanvalid 60 protmode CTS dtimperiod 1 -dfs
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33152
Thanks in advance!
Dave
On Sun, 4 Jul 2010, David Warren wrote:> Hi all, > > I've got a persistent problem with my LAN. I'm running a FreeBSD 8.0 > box as a home serverIs it feasible for you to update to 8.1-RC2 and see if that helps? Doug -- Improve the effectiveness of your Internet presence with a domain name makeover! http://SupersetSolutions.com/ Computers are useless. They can only give you answers. -- Pablo Picasso
On Jul 4, 2010, at 5:52 PM, David Warren wrote:> Hi all, > > I've got a persistent problem with my LAN. I'm running a FreeBSD 8.0 > box as a home server performing the following functions for wired and > wireless networks: router; firewall; DHCP server; and file server. For what > it's worth, I've got ZFS up and running as the main filesystem. The > recurring issue is that file transfers from the FreeBSD box to computers on > the wired network (gigabit) start out fast and then become agonizingly > slow. I'm sharing home directories over Samba, and those transfers work > briefly and then tail off to a few kilobytes per second. The failure is > somewhat predicatable in that it tends to happen once a few hundred > megabytes have been transferred. I've swapped out hardware, I've Googled > extensively, and all of the (possibly benign) error messages that I've found > have been eliminated. I'm happy to post logs, configs, etc., and I'd > appreciate any help with a diagnosis. For the moment:Disable rxcsum and txcsum on the cards; see if that does the trick. Before that, I'd see whether or not Jack Vogel (the current em(4) maintainer) would some details about your traffic, like a tcpdump session log. HTH, -Garrett
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/07/2010 01:52:11, David Warren wrote:> I've got a persistent problem with my LAN. I'm running a FreeBSD 8.0 > box as a home server performing the following functions for wired and > wireless networks: router; firewall; DHCP server; and file server. For what > it's worth, I've got ZFS up and running as the main filesystem. The > recurring issue is that file transfers from the FreeBSD box to computers on > the wired network (gigabit) start out fast and then become agonizingly > slow. I'm sharing home directories over Samba, and those transfers work > briefly and then tail off to a few kilobytes per second. The failure is > somewhat predicatable in that it tends to happen once a few hundred > megabytes have been transferred. I've swapped out hardware, I've Googled > extensively, and all of the (possibly benign) error messages that I've found > have been eliminated. I'm happy to post logs, configs, etc., and I'd > appreciate any help with a diagnosis. For the moment:Does this affect both wired and wireless LANs? Does this affect specific protocols more severely than other protocols? Does this affect all traffic happening at that point, all traffic on a particular interface or just specific connections? The effect you describe sounds bizarrely like entropy-pool exhaustion, which would kill any traffic over your wireless network, or anything using crypto protocols on either wired or wireless nets. I say "sounds like" because the yarrow entropy pool maintenance code in FreeBSD is extremely good, and I've seen FreeBSD boxes serving HTTPS at Mb/s without anything like what you are experiencing. Also, you're seeing it affect Samba over a wired connection, and that should be minimally affected by that sort of thing. Hmmm... Can you try transferring some large files (DVD .iso images around a few GB in size are handy for this) between systems using: * netcat or HTTP over wired connection * ssh over wired connection * netcat or HTTP over wireless * ssh over wireless Try these in both directions -- this should help show if it's slow disk performance on the receiving side. Also, check the output of 'netstat -i' to see if interface error counters are increasing: Ierrors, Idrop and Oerrors should all ideally be zero. If you see those starting to rise, it means either there's a configuration problem somewhere, like a duplex mismatch [no evidence for that from the ifconfig output you showed though] or perhaps there's still some dodgy hardware somewhere on your network despite all the swapping-out you've been doing. One final thought -- perhaps this isn't to do with the network at all, but it's disk IO performance bottoming out. In which case you should be able to see much the same effect copying files between different zpools, or between your main zpool and say, a USB memory stick. Cheers, Matthew - -- Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate JID: matthew@infracaninophile.co.uk Kent, CT11 9PW -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkwxb5IACgkQ8Mjk52CukIz6xQCffeqtaiZ5zc8JDX5EWzGAgdPo 1BoAn0F7Upq4QoTkg2O8mUPBCnYtom/T =EqiV -----END PGP SIGNATURE-----
On Sun, Jul 04, 2010 at 07:52:11PM -0500, David Warren wrote:> I've got a persistent problem with my LAN. I'm running a FreeBSD 8.0 > box as a home server performing the following functions for wired and > wireless networks: router; firewall; DHCP server; and file server. For what > it's worth, I've got ZFS up and running as the main filesystem. The > recurring issue is that file transfers from the FreeBSD box to computers on > the wired network (gigabit) start out fast and then become agonizingly > slow. I'm sharing home directories over Samba, and those transfers work > briefly and then tail off to a few kilobytes per second. The failure is > somewhat predicatable in that it tends to happen once a few hundred > megabytes have been transferred.Your system has 3 different network interfaces on it (em, ral, nfe), but you don't tell us across which interface the slow transfers happen. You also don't tell us which firewalling stack you're using (ipfw, ipfilter, pf). Let us know. I'm going to make the assumption that based on your "...on the wired network (gigabit)..." statement that the transfers are going across the em0 interface, but again, I'm not sure. Relevant interfaces (wlan0 is tied to ral0):> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM> > ether 00:0e:0c:b7:71:44 > inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255 > media: Ethernet autoselect (1000baseT <full-duplex>) > status: active > ral0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 2290 > ether 00:1f:1f:3f:76:f3 > media: IEEE 802.11 Wireless Ethernet autoselect mode 11g <hostap> > status: running > nfe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > options=8<VLAN_MTU> > ether 00:01:29:d4:2d:6b > inet XXX.XXX.XXX.XXX netmask 0xfffffc00 broadcast 255.255.255.255 > media: Ethernet autoselect (100baseTX <full-duplex,flag0,flag1>) > status: active > [...] > wlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > ether 00:1f:1f:3f:76:f3 > inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255 > media: IEEE 802.11 Wireless Ethernet autoselect mode 11g <hostap> > status: running > ssid FreeBSD_AP channel 7 (2442 Mhz 11g) bssid 00:1f:1f:3f:76:f3 > country US authmode WPA privacy MIXED deftxkey 2 TKIP 2:128-bit > txpower 0 scanvalid 60 protmode CTS dtimperiod 1 -dfsFirst and foremost: is the problem specific to Samba? Can you reproduce the problem when using the FTP protocol? Are there any indications of problems in "dmesg" when the issue is happening? Can you provide output from "vmstat -i" while the problem is happening? Can you provide output from "pciconf -lvc"? Only interested in the sections relevant to the above devices. Can you provide contents of /etc/make.conf, /etc/sysctl.conf, and /boot/loader.conf? Have you looked at "netstat -I <iface> -indb" output during the slow transfers to see if there's any indication of problems, or some sort of "common rate" (transfer, etc.) Does disabling the firewalling stack improve things at all? Can the slowness be reproduced using benchmarks/netperf or only when using something that involves actual disk I/O? (To use netperf you'll need two FreeBSD boxes). If only disk I/O, then ZFS analysis might be needed (there are some performance adjustments that are often required). Focusing more on em0: Have you tried disabling rxcsum and txcsum (using ifconfig) to see if there's any improvement? I don't see TSO used by your interface, so that should rule out any problems with that feature. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
On 7/4/2010 8:52 PM, David Warren wrote:> Hi all, > > I've got a persistent problem with my LAN. I'm running a FreeBSD 8.0 > box as a home server performing the following functions for wired and > wireless networks: router; firewall; DHCP server; and file server. For what > it's worth, I've got ZFS up and running as the main filesystem. The > recurring issue is that file transfers from the FreeBSD box to computers on > the wired network (gigabit) start out fast and then become agonizingly > slow. I'm sharing home directories over Samba, and those transfers work > briefly and then tail off to a few kilobytes per second. The failure is >I remember having a simmilar issue, back in the early 7.x days. Try the following tunnables. net.inet.tcp.hostcache.expire=1 net.inet.tcp.inflight.enable=0 Thanks.