I believe this is the second time I''ve encountered this issue within
the
past 6 mos or more. While performing some read/write tests between a
NFS client and server, I noticed speed was severely hindered on my
100mbits network with tests of writing a ~17MB file taking ~15 minutes,
of which, should only take ~8-10 seconds.
After restarting Shorewall (firewall), speeds immediately returned to normal.
Steps I took to restart the server were "/etc/init.d/shorewall stop
&&
shorewall clear && /etc/init.d/shorewall start". (I was trying to
see if the
configuration files were directly hindering performance. Upon restarting
shorewall, no further speed issues were noted.)
This leads me to believe, Shorewall is borking someplace -- or more correctly
put, one of the kernel modules concerning Netfilter is failing??
(Since this issue is extremely rare, it''s probably next to impossible
to
debug, except on past experience from others.)
Some bleak possibilities might include the use of Suspend to Disk (ie.
TuxOnIce) and kexec (kernel reload).
Here''s the requested debug info for this case scenario:
=net-firewall/shorewall-3.4.8
=net-fs/nfs-utils-1.1.1
=net-libs/libnfsidmap-0.16
=net-libs/librpcsecgss-0.16
=net-libs/libgssglue-0.1
# ip addr
show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc
noqueue
link/loopback 00:00:00:00:00:00 brd
00:00:00:00:00:00
inet 127.0.0.1/8 brd 127.255.255.255 scope host
lo
inet6 ::1/128 scope
host
valid_lft forever preferred_lft
forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
qlen 1000
link/ether 00:90:27:c6:b1:08 brd
ff:ff:ff:ff:ff:ff
inet 192.168.1.3/24 brd 192.168.1.255 scope global
eth0
inet6 fe80::290:27ff:fec6:b108/64 scope
link
valid_lft forever preferred_lft
forever
3: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc
noop
link/ether 92:74:43:62:0a:57 brd
ff:ff:ff:ff:ff:ff
4: tunl0: <NOARP> mtu 1480 qdisc
noop
link/ipip 0.0.0.0 brd
0.0.0.0
5: gre0: <NOARP> mtu 1476 qdisc
noop
link/gre 0.0.0.0 brd 0.0.0.0
# ip route
show
192.168.1.0/24 dev eth0 proto kernel scope link src
192.168.1.3
127.0.0.0/8 dev lo scope
link
default via 192.168.1.1 dev eth0
--
Roger
http://www.eskimo.com/~roger/index.html
Key fingerprint = 8977 A252 2623 F567 70CD 1261 640F C963 1005 1D61
Mon Mar 31 09:53:09 AKDT 2008
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It''s the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
roger wrote:> I believe this is the second time I''ve encountered this issue within the > past 6 mos or more. While performing some read/write tests between a > NFS client and server, I noticed speed was severely hindered on my > 100mbits network with tests of writing a ~17MB file taking ~15 minutes, > of which, should only take ~8-10 seconds.Is this with NFS over UDP or over TCP?> > After restarting Shorewall (firewall), speeds immediately returned to normal. > Steps I took to restart the server were "/etc/init.d/shorewall stop && > shorewall clear && /etc/init.d/shorewall start". (I was trying to see if the > configuration files were directly hindering performance. Upon restarting > shorewall, no further speed issues were noted.) > > This leads me to believe, Shorewall is borking someplace -- or more correctly > put, one of the kernel modules concerning Netfilter is failing??Given that Shorewall isn''t something that runs continuously in your system, it''s hard to imagine that this problem is Shorewall-related.> > (Since this issue is extremely rare, it''s probably next to impossible to > debug, except on past experience from others.) > > Some bleak possibilities might include the use of Suspend to Disk (ie. > TuxOnIce) and kexec (kernel reload). > > > Here''s the requested debug info for this case scenario:Better problem documentation would include: a) Output of "shorewall dump" at the time that the problem was encountered. b) Output of "shorewall dump" after recovery measures were taken and performance restored. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It''s the best place to buy or sell services for just about anything Open Source. http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
On Mon, Mar 31, 2008 at 11:47:54AM -0700, Tom Eastep wrote:>> After restarting Shorewall (firewall), speeds immediately returned to >> normal. Steps I took to restart the server were "/etc/init.d/shorewall >> stop && >> shorewall clear && /etc/init.d/shorewall start". (I was trying to see if the >> configuration files were directly hindering performance. Upon restarting >> shorewall, no further speed issues were noted.) >> >> This leads me to believe, Shorewall is borking someplace -- or more correctly >> put, one of the kernel modules concerning Netfilter is failing?? > > Given that Shorewall isn''t something that runs continuously in your > system, it''s hard to imagine that this problem is Shorewall-related.My bet is on a kernel, hardware, or network fabric issue that was shaken loose by the brief traffic interruption that occurs when shorewall starts. It could also be a traffic shaping corner case (some flawed configurations have cascade failure modes, where everything seizes up for certain traffic patterns but not others, and the effect is self-continuing).>> Here''s the requested debug info for this case scenario: > > Better problem documentation would include: > > a) Output of "shorewall dump" at the time that the problem was encountered. > b) Output of "shorewall dump" after recovery measures were taken and > performance restored.Other significant data points: - what''s on the network when the problem occurs? Steady-but-slow NFS traffic, fast-but-rare bursts, collisions, corrupted packets, something else entirely? Whenever I see something like this, I hit tcpdump -w first and think about it later; a packet dump explains most issues when you can study it at leisure. - relevant /proc/mounts entries. Do these change when the problem occurs? - physical network configuration - vmstat output when the problem occurs But I doubt it''s a shorewall problem. ------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It''s the best place to buy or sell services for just about anything Open Source. http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
On Mon, 2008-03-31 at 11:47 -0700, Tom Eastep wrote:> roger wrote: > > I believe this is the second time I''ve encountered this issue within the > > past 6 mos or more. While performing some read/write tests between a > > NFS client and server, I noticed speed was severely hindered on my > > 100mbits network with tests of writing a ~17MB file taking ~15 minutes, > > of which, should only take ~8-10 seconds. > > Is this with NFS over UDP or over TCP?Sorry, forgot to state I''m using NFSv4! TCP> Given that Shorewall isn''t something that runs continuously in your system, > it''s hard to imagine that this problem is Shorewall-related.What I''m figuring too.> a) Output of "shorewall dump" at the time that the problem was encountered. > b) Output of "shorewall dump" after recovery measures were taken and > performance restored. > > -TomHopefully, in six plus months, I''ll remember to do this prior to restarting the firewall. -- Roger http://www.eskimo.com/~roger/index.html Key fingerprint = 8977 A252 2623 F567 70CD 1261 640F C963 1005 1D61 Mon Mar 31 12:41:45 AKDT 2008 ------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It''s the best place to buy or sell services for just about anything Open Source. http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
On Mon, 2008-03-31 at 20:18 +0100, Andrew Suffield wrote:> My bet is on a kernel, hardware, or network fabric issue that was > shaken loose by the brief traffic interruption that occurs when > shorewall starts. It could also be a traffic shaping corner case (some > flawed configurations have cascade failure modes, where everything > seizes up for certain traffic patterns but not others, and the effect > is self-continuing).> > Other significant data points: > > - what''s on the network when the problem occurs? Steady-but-slow NFS > traffic, fast-but-rare bursts, collisions, corrupted packets, > something else entirely? Whenever I see something like this, I hit > tcpdump -w first and think about it later; a packet dump explains > most issues when you can study it at leisure. > > - relevant /proc/mounts entries. Do these change when the problem occurs? > > - physical network configuration > > - vmstat output when the problem occurs > > But I doubt it''s a shorewall problem.Right, it is possible the laptop''s kernel hibernation (Suspend2/TuxOnIce) process is faulty on resume. A more likely scenario I just thought of, switching between a wireless (wlan0) and wired NIC (eth0), even though I''m keeping the same IP address for both NICs, the kernel netfilter might get confused? -- Roger http://www.eskimo.com/~roger/index.html Key fingerprint = 8977 A252 2623 F567 70CD 1261 640F C963 1005 1D61 Mon Mar 31 12:48:25 AKDT 2008 ------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It''s the best place to buy or sell services for just about anything Open Source. http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
On Mon, Mar 31, 2008 at 12:41:46PM -0800, roger wrote:> > On Mon, 2008-03-31 at 11:47 -0700, Tom Eastep wrote: > > roger wrote: > > > I believe this is the second time I''ve encountered this issue within the > > > past 6 mos or more. While performing some read/write tests between a > > > NFS client and server, I noticed speed was severely hindered on my > > > 100mbits network with tests of writing a ~17MB file taking ~15 minutes, > > > of which, should only take ~8-10 seconds. > > > > Is this with NFS over UDP or over TCP? > > Sorry, forgot to state I''m using NFSv4!Then it''s probably still the buggy pile of weird that it was when I last looked at it. Stick with NFSv3 for production systems. ------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It''s the best place to buy or sell services for just about anything Open Source. http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
On Mon, 2008-03-31 at 22:07 +0100, Andrew Suffield wrote:> On Mon, Mar 31, 2008 at 12:41:46PM -0800, roger wrote:> > Sorry, forgot to state I''m using NFSv4! > > Then it''s probably still the buggy pile of weird that it was when I > last looked at it. Stick with NFSv3 for production systems.I had to upgrade. Having too many problems with NFSv3. :-/ -- Roger http://www.eskimo.com/~roger/index.html Key fingerprint = 8977 A252 2623 F567 70CD 1261 640F C963 1005 1D61 Mon Mar 31 13:22:33 AKDT 2008 ------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It''s the best place to buy or sell services for just about anything Open Source. http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
roger wrote:> > A more likely scenario I just thought of, switching between a wireless > (wlan0) and wired NIC (eth0), even though I''m keeping the same IP > address for both NICs, the kernel netfilter might get confused? >Netfilter doesn''t track by interface. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It''s the best place to buy or sell services for just about anything Open Source. http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
On Mon, 2008-03-31 at 15:53 -0700, Tom Eastep wrote:> Netfilter doesn''t track by interface. > > -TomThanks everyone for your quick responses. I''ll forward the info learned here to my bug report for my distro''s bug site. Thank you for your time! -- Roger http://www.eskimo.com/~roger/index.html Key fingerprint = 8977 A252 2623 F567 70CD 1261 640F C963 1005 1D61 Tue Apr 1 10:48:21 AKDT 2008 ------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It''s the best place to buy or sell services for just about anything Open Source. http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace