G''day, I have a server machine connected to the net via eth0 to a Netgear ADSL router. There are other machines connected to the Netgear router that use the server as an nfs/samba/apache/proxy/dnscache/etc. The server is also a public webserver. The netgear is configured to forward all incoming ADSL traffic to the server. I have shorewall configured on the server to protect it, including things like rate-limiting incoming email and ssh connections. Ideally I''d like to run shorewall on the Netgear... (Oh how I miss having a linux router I could control...) I''ve also found that the server ADSL web traffic can be busy enough that my battlefield 2142 games on another machine complain about network latency. So I thought I''d configure shorewall''s internal TC stuff to limit the outgoing traffic and save a bit for the other machines. This worked great... until I stopped playing Battlefield and tried to use NFS. At first I thought that local NFS traffic was incorrectly being classified and rate limited as ADSL traffic. However, further digging with "shorewall show tc|mangle|connections|classifers", running NFS reads/writes, and using iptraff to measure network throughput showed that; 1) NFS was not being incorrectly classified as ADSL traffic. All NFS traffic was marked and classified to use the 1:14 class with a rate/ceil of 99mbit/100mbit. 2) The NFS slowdowns only affected NFS writes. NFS reads went fast and could saturate the 100M eth0 interface at about 80Mbit/sec. Writes were so slow the client would often appear to have hung, and interface was around 500kbit/sec. 3) The issue was not CPU. The loadavg on the server was around 0.1 and CPU was 98% idle throughout the NFS write tests. 4) The problem was definitely caused by turning on TC. The problem only occured with "TC_ENABLED=Internal" in shorewall. With "TC_ENABLED=No" the NFS write speeds were nice and snappy, hitting the same 80Mbit/sec interface traffic of reads. So I simplified things down to a single default tcclass with rate/ceil of 99mbit/100mbit and no tcrules... and the problem still present. It seems that just turning on TC was enough. Then I noticed the "HTB: quantum of class 10001 is big. Consider r2q change." message in dmesg and /var/log/syslog. Online searches showed that quantum should be less than 60000, so I modified "calculate_quantum()" in /usr/share/shorewall/compiler so that it limited quantum to 60000. This didn''t help, though the warning message continued to show after I did this... "shorewall show tc" did show that class 1:14 had the limited quantum, but I suspect that my change only affected the "leaf" classes, so maybe this is still the problem. In summary, I don''t think this is a problem with shorewall... it looks like a problem between HTB TC and NFS. It''s rather strange that it only affects writes, but I suspect this is because writes use more back and forth traffic to verify write success, and this is different to the more continuous streaming of reads. Any suggestions on where to go from here are welcome. I''m going to dig further to try and get the quantum warning to go away and see if that helps. Otherwise I''m going to come to the conclusion that NFS and TC are just incompatible. -- Donovan Baarda http://minkirri.apana.org.au/~abo/ ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net''s Techsay panel and you''ll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
On Mon, 2007-04-09 at 12:48 +0100, Donovan Baarda wrote: [...]> Then I noticed the "HTB: quantum of class 10001 is big. Consider r2q > change." message in dmesg and /var/log/syslog. Online searches showed > that quantum should be less than 60000, so I modified > "calculate_quantum()" in /usr/share/shorewall/compiler so that it > limited quantum to 60000. This didn''t help, though the warning message > continued to show after I did this... "shorewall show tc" did show that > class 1:14 had the limited quantum, but I suspect that my change only > affected the "leaf" classes, so maybe this is still the problem.So I hacked away at /usr/share/shorewall/compiler, fixing some bad mbit/kbit conversions (network bandwidth uses 1000, not 1024), and adding calculation of r2q based on the interface outband speed. This made the quantum warning go away, but it didn''t help. I also read in the HTB docs that a qdisk with a default of 0 will make unclassified packets bypass the classes and be sent at hardware speed. So I modified /usr/share/shorewall/compiler to support having no default class for an interface so that unclassified packets bypass TC... this didn''t help either.> In summary, I don''t think this is a problem with shorewall... it looks > like a problem between HTB TC and NFS. It''s rather strange that it only > affects writes, but I suspect this is because writes use more back and > forth traffic to verify write success, and this is different to the more > continuous streaming of reads.[...] I''ve now figured it out; a Doh! moment. The difference between nfs reads and writes is one is outgoing traffic, the other is incoming traffic... the problem was the ingress stuff. I haven''t figured out exactly why the ingress was a problem, but I did notice an undocumented feature; set in-bandwidth to 0mbit in tcdevices and shorewall will not set up the ingress stuff. This means I can turn off the ingress stuff and NFS writes are nice and fast again. There could be something about how shorewall sets up the ingress stuff that is bad for 100mbit speeds, or it could just be that the TC ingress handling just can''t work at that speed... I dunno, but for me turning ingress stuff off is good enough. I don''t know if anyone is interested in my mods to /usr/share/shorewall/compiler. The "no default" change allows me to set the eth0 device at 512kbit and only classify the ADSL traffic. All unclassified local traffic bypasses the TC stuff and goes a full 100mbit speed. The fixes to 1000/1024 mbit/kbit conversions should probably go upstream. The r2q calculation that makes the quantum warning go away is probably only a niceness. -- Donovan Baarda <abo@minkirri.apana.org.au> http://minkirri.apana.org.au/~abo/ ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net''s Techsay panel and you''ll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
Donovan Baarda wrote:> > I''ve now figured it out; a Doh! moment. The difference between nfs reads > and writes is one is outgoing traffic, the other is incoming traffic... > the problem was the ingress stuff.I hadn''t responded to your post yet in the hope that you would realize that ;-)> I haven''t figured out exactly why the > ingress was a problem, but I did notice an undocumented feature; set > in-bandwidth to 0mbit in tcdevices and shorewall will not set up the > ingress stuff.It''s documented in the ''shorewall-tcdevices'' man page. I hope to phase out the existing reference documentation at http://www.shorewall.net/Documentation.htm so I''ve not been updating it regularly.> This means I can turn off the ingress stuff and NFS > writes are nice and fast again. > > I don''t know if anyone is interested in my mods > to /usr/share/shorewall/compiler. The "no default" change allows me to > set the eth0 device at 512kbit and only classify the ADSL traffic. All > unclassified local traffic bypasses the TC stuff and goes a full 100mbit > speed.> The fixes to 1000/1024 mbit/kbit conversions should probably go > upstream. The r2q calculation that makes the quantum warning go away is > probably only a niceness. >If you want to throw you changes over the fence, I''ll take a look at them. Thanks, -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net''s Techsay panel and you''ll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
On Mon, 2007-04-09 at 15:04 -0700, Tom Eastep wrote:> Donovan Baarda wrote: > > > > > I''ve now figured it out; a Doh! moment. The difference between nfs reads > > and writes is one is outgoing traffic, the other is incoming traffic... > > the problem was the ingress stuff. > > I hadn''t responded to your post yet in the hope that you would realize that ;-)[...] Any idea why the shorewall ingress setup for 100mbit in-bandwidth would hurt so bad?> > I don''t know if anyone is interested in my mods > > to /usr/share/shorewall/compiler. The "no default" change allows me to > > set the eth0 device at 512kbit and only classify the ADSL traffic. All > > unclassified local traffic bypasses the TC stuff and goes a full 100mbit > > speed. > > > The fixes to 1000/1024 mbit/kbit conversions should probably go > > upstream. The r2q calculation that makes the quantum warning go away is > > probably only a niceness. > > > > If you want to throw you changes over the fence, I''ll take a look at them.Patch attached. The calculation for r2q is pretty basic... the (kbits / 200) is a compromise to try and get decent quantum values for rates between 1mbit and 100mbit. This gives quantum ~25,000 for speed "full", and ~2500 for speeds "full/10". For rates < 1mbit I limit the minimum r2q to 5 and rely on the fact that the minimum quantum is limited to the device mtu. You can change the /200 to /400 to give quantum ~50,000 for "full", and by my calcs you should still stay under the 60,000 max, but I get the impression that high quantum''s are a bad idea. Note at one stage I had also changed the calculate_quantum() function to limit the output to 60000. However, this function is not used for the device root class, so it didn''t fix the "quantum warning". With the r2q calc''s you should never exceed 60,000 anyway... The most useful addition IMHO is the ability to not define a default class, and have all unclassified traffic bypass TC. -- Donovan Baarda <abo@minkirri.apana.org.au> http://minkirri.apana.org.au/~abo/ ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net''s Techsay panel and you''ll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
Donovan Baarda wrote:> On Mon, 2007-04-09 at 15:04 -0700, Tom Eastep wrote:> Patch attached. The calculation for r2q is pretty basic... the (kbits / > 200) is a compromise to try and get decent quantum values for rates > between 1mbit and 100mbit. This gives quantum ~25,000 for speed "full", > and ~2500 for speeds "full/10". For rates < 1mbit I limit the minimum > r2q to 5 and rely on the fact that the minimum quantum is limited to the > device mtu. You can change the /200 to /400 to give quantum ~50,000 for > "full", and by my calcs you should still stay under the 60,000 max, but > I get the impression that high quantum''s are a bad idea. > > Note at one stage I had also changed the calculate_quantum() function to > limit the output to 60000. However, this function is not used for the > device root class, so it didn''t fix the "quantum warning". With the r2q > calc''s you should never exceed 60,000 anyway... > > The most useful addition IMHO is the ability to not define a default > class, and have all unclassified traffic bypass TC.Thanks. Unfortunately, this patch is against Shorewall 3.2. Shorewall 3.4 is the current stable release and would be the earliest release that I would consider adding these changes to. I''ll hack around with it when I have the time. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net''s Techsay panel and you''ll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV