LARTC - Aug 2007 - Question about how TC enforces bandwidth limiting

Hello,

I run one of my PCs as my personal router, with iptables+tc to control 
traffic and be my firewall.

In TC, I use a combination of htb, qdisc and sfq (as well as prio) to 
classify bandwidth. In my current setup, I have 10 classifications of my 
bandwidth. (Even I admit this is probably more than I need, but at this 
point I''m still learning, so I''ll just leave them be.) This
leads me to
my question about how TC enforces maximum bandwidth limits.

I am on DSL internet with rates 1.5Mbps/384kbps. I do not make complete 
use of my pipe just in case of a massive burst. I know I will probably 
not burst such a massive burst, but its better to be safe than sorry.

To help understand this, here are the classes and their associated 
bandwidths as I have them set in TC:

    (MAX_RATE is 360kbps)

    $TC qdisc add dev $IFext root handle 1: htb default 90
    $TC class add dev $IFext parent 1: classid 1:1 htb rate $MAX_RATE 
quantum $QUANTUM
    $TC class add dev $IFext parent 1:1 classid 1:10 htb rate 240kbps 
ceil $MAX_RATE prio 0 quantum $QUANTUM
    $TC class add dev $IFext parent 1:1 classid 1:20 htb rate 192kbps 
ceil $MAX_RATE prio 1 quantum $QUANTUM
    $TC class add dev $IFext parent 1:1 classid 1:30 htb rate 80kbps 
ceil $MAX_RATE prio 2 quantum $QUANTUM
    $TC class add dev $IFext parent 1:1 classid 1:40 htb rate 64kbps 
ceil $MAX_RATE prio 3 quantum $QUANTUM
    $TC class add dev $IFext parent 1:1 classid 1:50 htb rate 32kbps 
ceil $MAX_RATE prio 4 quantum $QUANTUM
    $TC class add dev $IFext parent 1:1 classid 1:60 htb rate 20kbps 
ceil $MAX_RATE prio 5 quantum $QUANTUM
    $TC class add dev $IFext parent 1:1 classid 1:70 htb rate 16kbps 
ceil $MAX_RATE prio 6 quantum $QUANTUM
    $TC class add dev $IFext parent 1:1 classid 1:80 htb rate 8kbps ceil 
$MAX_RATE prio 7 quantum $QUANTUM
    $TC class add dev $IFext parent 1:1 classid 1:90 htb rate 2kbps ceil 
$MAX_RATE prio 8 quantum $QUANTUM
    $TC class add dev $IFext parent 1:1 classid 1:100 htb rate 2kbps 
ceil 20kbps prio 9 quantum $QUANTUM
    $TC qdisc add dev $IFext parent 1:10 handle 10: sfq perturb 10
    $TC qdisc add dev $IFext parent 1:20 handle 20: sfq perturb 10
    $TC qdisc add dev $IFext parent 1:30 handle 30: sfq perturb 10
    $TC qdisc add dev $IFext parent 1:40 handle 40: sfq perturb 10
    $TC qdisc add dev $IFext parent 1:50 handle 50: sfq perturb 10
    $TC qdisc add dev $IFext parent 1:60 handle 60: sfq perturb 10
    $TC qdisc add dev $IFext parent 1:70 handle 70: sfq perturb 10
    $TC qdisc add dev $IFext parent 1:80 handle 80: sfq perturb 10
    $TC qdisc add dev $IFext parent 1:90 handle 90: sfq perturb 10
    $TC qdisc add dev $IFext parent 1:100 handle 100: sfq perturb 10

Class 90 is the default. Class 100 is a special class, and what my 
question specifically relates to. Class 100 is for bit torrent. I do not 
like the other people in my house using very much bandwidth for 
torrenting as it has a tendency to slow things down to greatly. The 
problem I have is this: when I disable a given torrent clients upload 
limits, the bandwidth climbs to above the 20kbps limit I have set for 
it. When I classify the traffic in iptables, i put it into class 100, so 
it shouldn''t getting put into the default class.

While I understand how/why TC enforces minimum bandwidth for a given 
class, why is it that for class 100 TC is not enforcing the cap of 
20kbps to traffic that it is classified at? Is there something else I 
need to do to make TC also enforce arbitrary maximum limits for a given 
classification?

Vadtec

Hello,

I run one of my PCs as my personal router, with iptables+tc to control 
traffic and be my firewall.

In TC, I use a combination of htb, qdisc and sfq (as well as prio) to 
classify bandwidth. In my current setup, I have 10 classifications of my 
bandwidth. (Even I admit this is probably more than I need, but at this 
point I''m still learning, so I''ll just leave them be.) This
leads me to
my question about how TC enforces maximum bandwidth limits.

I am on DSL internet with rates 1.5Mbps/384kbps. I do not make complete 
use of my pipe just in case of a massive burst. I know I will probably 
not burst such a massive burst, but its better to be safe than sorry.

To help understand this, here are the classes and their associated 
bandwidths as I have them set in TC:

   (MAX_RATE is 360kbps)

   $TC qdisc add dev $IFext root handle 1: htb default 90
   $TC class add dev $IFext parent 1: classid 1:1 htb rate $MAX_RATE 
quantum $QUANTUM
   $TC class add dev $IFext parent 1:1 classid 1:10 htb rate 240kbps 
ceil $MAX_RATE prio 0 quantum $QUANTUM
   $TC class add dev $IFext parent 1:1 classid 1:20 htb rate 192kbps 
ceil $MAX_RATE prio 1 quantum $QUANTUM
   $TC class add dev $IFext parent 1:1 classid 1:30 htb rate 80kbps ceil 
$MAX_RATE prio 2 quantum $QUANTUM
   $TC class add dev $IFext parent 1:1 classid 1:40 htb rate 64kbps ceil 
$MAX_RATE prio 3 quantum $QUANTUM
   $TC class add dev $IFext parent 1:1 classid 1:50 htb rate 32kbps ceil 
$MAX_RATE prio 4 quantum $QUANTUM
   $TC class add dev $IFext parent 1:1 classid 1:60 htb rate 20kbps ceil 
$MAX_RATE prio 5 quantum $QUANTUM
   $TC class add dev $IFext parent 1:1 classid 1:70 htb rate 16kbps ceil 
$MAX_RATE prio 6 quantum $QUANTUM
   $TC class add dev $IFext parent 1:1 classid 1:80 htb rate 8kbps ceil 
$MAX_RATE prio 7 quantum $QUANTUM
   $TC class add dev $IFext parent 1:1 classid 1:90 htb rate 2kbps ceil 
$MAX_RATE prio 8 quantum $QUANTUM
   $TC class add dev $IFext parent 1:1 classid 1:100 htb rate 2kbps ceil 
20kbps prio 9 quantum $QUANTUM
   $TC qdisc add dev $IFext parent 1:10 handle 10: sfq perturb 10
   $TC qdisc add dev $IFext parent 1:20 handle 20: sfq perturb 10
   $TC qdisc add dev $IFext parent 1:30 handle 30: sfq perturb 10
   $TC qdisc add dev $IFext parent 1:40 handle 40: sfq perturb 10
   $TC qdisc add dev $IFext parent 1:50 handle 50: sfq perturb 10
   $TC qdisc add dev $IFext parent 1:60 handle 60: sfq perturb 10
   $TC qdisc add dev $IFext parent 1:70 handle 70: sfq perturb 10
   $TC qdisc add dev $IFext parent 1:80 handle 80: sfq perturb 10
   $TC qdisc add dev $IFext parent 1:90 handle 90: sfq perturb 10
   $TC qdisc add dev $IFext parent 1:100 handle 100: sfq perturb 10

Class 90 is the default. Class 100 is a special class, and what my 
question specifically relates to. Class 100 is for bit torrent. I do not 
like the other people in my house using very much bandwidth for 
torrenting as it has a tendency to slow things down to greatly. The 
problem I have is this: when I disable a given torrent clients upload 
limits, the bandwidth climbs to above the 20kbps limit I have set for 
it. When I classify the traffic in iptables, i put it into class 100, so 
it shouldn''t getting put into the default class.

While I understand how/why TC enforces minimum bandwidth for a given 
class, why is it that for class 100 TC is not enforcing the cap of 
20kbps to traffic that it is classified at? Is there something else I 
need to do to make TC also enforce arbitrary maximum limits for a given 
classification?

Vadtec

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Vadtec,

I think you may be making two of the most common problems facing 
novices working with traffic control, so I hope you don''t mind my 
picking on you!

Problem #0
- ----------
You are applying your shaping on the outbound traffic (presuming 
that "$IFext" is your external interface.  Unless you also have 
shaping on your inbound traffic ("$IFint"?), then you are only 
applying shaping characteristics to the upload traffic.  This brings 
us back to two fundamental rules of traffic shaping:

  * For optimal results, your shaping device should be the 
    bottleneck, so that it can act as the traffic flow valve.
  * You can only shape what you transmit.  If your edge device is 
    performing shaping, then it should shape upload traffic by a 
    policy applied to the external interface and it should shape 
    download traffic by policy applied to the internal interface.

In short, add a similar set of HTB + SFQ queues to your internal 
interface, along with the appropriate classifiers and try again.

 : While I understand how/why TC enforces minimum bandwidth for a 
 : given class, why is it that for class 100 TC is not enforcing the 
 : cap of 20kbps to traffic that it is classified at? Is there 
 : something else I need to do to make TC also enforce arbitrary 
 : maximum limits for a given classification?
 :
 : I am on DSL internet with rates 1.5Mbps/384kbps. 

That 1.5Mbps (conventional networking terminology and units) is 
written as 1.5Mbit in terms used by tc.

Problem #1
- ----------
I think you may be making an error in your units.  This is one of 
the most frequent problems when people start using "tc".  Since
"tc"
sprang from the primordial soup, the following units are used:

  bps = bytes per second
  bit = bits per second

The unfortunate problem with this marking for units is that we say 
many other places in networking that bits per second is bps.  This 
is not true with tc.  So, if I look at your rate specifications 
below, they look off by a factor of 8.  Please try altering all 
instances of "kbps" to "kbit" and try your script again. See
also
these URLs [0] [1] [2].

 : I do not make complete use of my pipe just in case of a massive 
 : burst. I know I will probably not burst such a massive burst, but 
 : its better to be safe than sorry.

This is wise.

 : Class 90 is the default. Class 100 is a special class, and what 
 : my question specifically relates to. Class 100 is for bit 
 : torrent. I do not like the other people in my house using very 
 : much bandwidth for torrenting as it has a tendency to slow things 
 : down to greatly.

If you place FIFOs in any of your HTB leaf classes, you can vary the 
depth of the FIFO queue to help control latency, in addition to that 
class''s total throughput.  This is a cheap and dirty way to 
accomplish this task.

 : The problem I have is this: when I disable a given torrent 
 : clients upload limits, the bandwidth climbs to above the 20kbps 
 : limit I have set for it. When I classify the traffic in iptables, 
 : i put it into class 100, so it shouldn''t getting put into the 
 : default class.

While you are starting and stopping your torrent client, you should 
also take a look at the class statistics:

  watch -n 1 tc -s class show dev $INTERFACE_NAME

This will allow you to see which class is being used to carry that 
traffic.

Good luck,

- -Martin

 [0] http://www.docum.org/docum.org/faq/cache/74.html
 [1] http://mailman.ds9a.nl/pipermail/lartc/2003q4/010826.html
 [2] http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm

- -- 
Martin A. Brown
http://linux-ip.net/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: pgf-0.72 (http://linux-ip.net/sw/pine-gpg-filter/)

iD8DBQFG3FYGHEoZD1iZ+YcRAr3NAKC2Iq1mtkEwd3edzU8mY6CQx/PuKgCggE0F
hcIyU0L25TYNwMkXGcjusWw=ifyk
-----END PGP SIGNATURE-----

Martin A. Brown wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Vadtec,
>
> I think you may be making two of the most common problems facing 
> novices working with traffic control, so I hope you don''t mind my 
> picking on you!
>   
Not at all. :)
> Problem #0
> - ----------
> You are applying your shaping on the outbound traffic (presuming 
> that "$IFext" is your external interface.  Unless you also have 
> shaping on your inbound traffic ("$IFint"?), then you are only 
> applying shaping characteristics to the upload traffic.  This brings 
> us back to two fundamental rules of traffic shaping:
>   
$IFext is eth0, and is my link to my DSL Modem. $IFint would be eth1, 
which is my interface
to my LAN (though I am not shaping traffic on it in any way). Both of 
these devices plug into
my 24 port switch, which the DSL modem is hooked into, as well as all 
the other computers in
the house.

So just for clarity sake, here are the port assignments on the switch:
Port 1 - DSL Modem uplink
Port 2 - $IFext (eth0) on my router PC
Port 3 - $IFint (eth1) on my router PC
Port 4 - My laptop
Port 5 - My brothers PC
Port 6/25 - Unused currently, but used at random as
needed.
>   * For optimal results, your shaping device should be the 
>     bottleneck, so that it can act as the traffic flow valve.
>   * You can only shape what you transmit.  If your edge device is 
>     performing shaping, then it should shape upload traffic by a 
>     policy applied to the external interface and it should shape 
>     download traffic by policy applied to the internal interface.
>
> In short, add a similar set of HTB + SFQ queues to your internal 
> interface, along with the appropriate classifiers and try again.
>   
So you are saying I have to not only do traffic shaping, but also 
traffic policing on my internal device?
Or do I have to do traffic shaping on both devices and no traffic 
policing? In other words, how much
traffic shaping/policing do I need to put into effect, and on which 
interfaces.
>  : While I understand how/why TC enforces minimum bandwidth for a 
>  : given class, why is it that for class 100 TC is not enforcing the 
>  : cap of 20kbps to traffic that it is classified at? Is there 
>  : something else I need to do to make TC also enforce arbitrary 
>  : maximum limits for a given classification?
>  :
>  : I am on DSL internet with rates 1.5Mbps/384kbps. 
>
> That 1.5Mbps (conventional networking terminology and units) is 
> written as 1.5Mbit in terms used by tc.
>
> Problem #1
> - ----------
> I think you may be making an error in your units.  This is one of 
> the most frequent problems when people start using "tc".  Since
"tc"
> sprang from the primordial soup, the following units are used:
>
>   bps = bytes per second
>   bit = bits per second
>
> The unfortunate problem with this marking for units is that we say 
> many other places in networking that bits per second is bps.  This 
> is not true with tc.  So, if I look at your rate specifications 
> below, they look off by a factor of 8.  Please try altering all 
> instances of "kbps" to "kbit" and try your script
again. See also
> these URLs [0] [1] [2].
>   
So, what you are saying is, its just a matter of different naming. In 
essence,
368kbps (conventional) is the same as 368kbit (tc),
right?
>  : I do not make complete use of my pipe just in case of a massive 
>  : burst. I know I will probably not burst such a massive burst, but 
>  : its better to be safe than sorry.
>
> This is wise.
>   
:)
>  : Class 90 is the default. Class 100 is a special class, and what 
>  : my question specifically relates to. Class 100 is for bit 
>  : torrent. I do not like the other people in my house using very 
>  : much bandwidth for torrenting as it has a tendency to slow things 
>  : down to greatly.
>
> If you place FIFOs in any of your HTB leaf classes, you can vary the 
> depth of the FIFO queue to help control latency, in addition to that 
> class''s total throughput.  This is a cheap and dirty way to 
> accomplish this task.
>   
I have no idea what that means. How do I vary the depth of the FIFO to 
help control latency?
>  : The problem I have is this: when I disable a given torrent 
>  : clients upload limits, the bandwidth climbs to above the 20kbps 
>  : limit I have set for it. When I classify the traffic in iptables, 
>  : i put it into class 100, so it shouldn''t getting put into the 
>  : default class.
>
> While you are starting and stopping your torrent client, you should 
> also take a look at the class statistics:
>
>   watch -n 1 tc -s class show dev $INTERFACE_NAME
>
> This will allow you to see which class is being used to carry that 
> traffic.
>   
I ran both watch -n 1 tc -s class show dev eth0 and watch -n 1 tc -s 
qdisc show eth0.
(When I ran class show, i did not have enough room to see classes 80 and 
90. When I ran qdisc
show, I was able to see all the classes.) During my runs of tc in this 
manner, I saw zero traffic going
to class 100 when running, starting, or stopping bit torrent. Almost all 
the traffic was going to class 10
and 90 (default) with the exception of my ICMP and UDP traffic which was 
going to class 70 and
class 60 which I have set aside for IRC traffic. Class 100 saw 
absolutely zero traffic.

Is this a case where the default class (90) is getting all the traffic 
because it can handle it as my LAN
has very little "other" traffic most of the time to deal with, so
there
is no need to throttle it back?
If so, how can I force a particular class to be used regardless of the 
default,
so that I can control individual apps by them selves?
> Good luck,
>
> - -Martin
>
>  [0] http://www.docum.org/docum.org/faq/cache/74.html
>  [1] http://mailman.ds9a.nl/pipermail/lartc/2003q4/010826.html
>  [2] http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm
>
> - -- 
> Martin A. Brown
> http://linux-ip.net/
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.2 (GNU/Linux)
> Comment: pgf-0.72 (http://linux-ip.net/sw/pine-gpg-filter/)
>
> iD8DBQFG3FYGHEoZD1iZ+YcRAr3NAKC2Iq1mtkEwd3edzU8mY6CQx/PuKgCggE0F
> hcIyU0L25TYNwMkXGcjusWw> =ifyk
> -----END PGP SIGNATURE-----
>
>   
Thank you for your help!

Vadtec

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Greetings again,

 : $IFext is eth0, and is my link to my DSL Modem. $IFint would be 
 : eth1, which is my interface to my LAN (though I am not shaping 
 : traffic on it in any way).

Yes, exactly.  You grasped this below, too.  Recall that you can 
only shape what you transmit?  Think about why.  If you already have 
a packet, you can delay the transmission.  This is the fundamental 
behaviour of all non-work conserving queues.  To introduce a delay 
to a packet involves work.

 : So you are saying I have to not only do traffic shaping, but also 
 : traffic policing on my internal device? Or do I have to do 
 : traffic shaping on both devices and no traffic policing? In other 
 : words, how much traffic shaping/policing do I need to put into 
 : effect, and on which interfaces.

Be careful.  Shaping and policing are two very different things.  I 
probably could have chosen a word other than "policy" to make this 
clear.  I''ll restate this.

If you wish to shape upload traffic, then you must put your traffic 
control structures on the the device closest to the Internet.  If 
you wish to shape download traffic, then you must put your traffic 
control structures on the device closest to the internal network(s). 
( If you would like to see how you can break this basic rule, see 
some advanced traffic control structures called imq and ifb [*]. )

 : So, what you are saying is, its just a matter of different 
 : naming. In essence, 368kbps (conventional) is the same as 368kbit 
 : (tc), right?

Bingo.

 : I have no idea what that means. How do I vary the depth of the 
 : FIFO to help control latency?

OK, in order to vary the exact depth of the fifo to suit your fancy, 
you''d need to use terminal FIFOs instead of using terminal SFQs:

  tc qdisc add dev $INTERFACE parent 10:1 handle 100:0 pfifo limit 10

Assuming your HTB tree has a leaf class of 10:1, into which you have 
classified some of your traffic, you now have a very short queue 
there.  If a burst of traffic involving more than ten packets 
arrives, the traffic will tail drop.  In heavily congested 
situations, this can be useful, although this still allows for a 
single dominating UDP flow, so I''d probably prefer the SFQ solution, 
and maybe I shouldn''t have brought up this little trick.

 : I ran both watch -n 1 tc -s class show dev eth0 and watch -n 1 tc 
 : -s qdisc show eth0. (When I ran class show, i did not have enough 
 : room to see classes 80 and 90. When I ran qdisc show, I was able 
 : to see all the classes.) During my runs of tc in this manner, I 
 : saw zero traffic going to class 100 when running, starting, or 
 : stopping bit torrent.

If you expect that the torrential traffic to appear in class 100, 
and it does not, then you have done something wrong in your 
classifier.  Look there.

 : Almost all the traffic was going to class 10 and 90 (default) 
 : with the exception of my ICMP and UDP traffic which was going to 
 : class 70 and class 60 which I have set aside for IRC traffic. 
 : Class 100 saw absolutely zero traffic.
 : 
 : Is this a case where the default class (90) is getting all the 
 : traffic because it can handle it as my LAN has very little 
 : "other" traffic most of the time to deal with, so there is no 
 : need to throttle it back? If so, how can I force a particular 
 : class to be used regardless of the default, so that I can control 
 : individual apps by them selves?

Try looking at your classifiers ("tc filter" statements) to 
determine where the problem is.  If you can''t figure it out, then 
send the list your classifiers and class structure (tc commands).

I''d skip playing with terminal FIFOs at first (if ever).  Try 
working on the following:

  * switch the units to kbit
  * fix the classifiers and verify that you are seeing all traffic
    in the class you want to see the traffic in
  * turn the torrent client up to "insane" and watch how your IRC or 
    ssh latency is very low

Good luck,

- -Martin

 [*] IMQ, intermediate queuing device
     http://www.linuximq.net/
     IFB, intermediate functional block
     http://linux-net.osdl.org/index.php/IFB

- -- 
Martin A. Brown
http://linux-ip.net/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: pgf-0.72 (http://linux-ip.net/sw/pine-gpg-filter/)

iD4DBQFG3L5sHEoZD1iZ+YcRAso6AJjnbMjcso8t4+GugFC6eHQOyqJQAJ9kpWbZ
sOS369AbZkPQ9rg3rhiawg==gvDh
-----END PGP SIGNATURE-----

Martin A. Brown wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Greetings again,
>
>  : So you are saying I have to not only do traffic shaping, but also 
>  : traffic policing on my internal device? Or do I have to do 
>  : traffic shaping on both devices and no traffic policing? In other 
>  : words, how much traffic shaping/policing do I need to put into 
>  : effect, and on which interfaces.
>
> Be careful.  Shaping and policing are two very different things.  I 
> probably could have chosen a word other than "policy" to make
this
> clear.  I''ll restate this.
>
> If you wish to shape upload traffic, then you must put your traffic 
> control structures on the the device closest to the Internet.  If 
> you wish to shape download traffic, then you must put your traffic 
> control structures on the device closest to the internal network(s). 
> ( If you would like to see how you can break this basic rule, see 
> some advanced traffic control structures called imq and ifb [*]. )
>   
No need for advanced traffic controls for me. All I want to do is 
control how much band width
gets out to the internet. Internally it doesn''t matter as its all CAT5,
so there is plenty of data pipe.
>  : I have no idea what that means. How do I vary the depth of the 
>  : FIFO to help control latency?
>
> OK, in order to vary the exact depth of the fifo to suit your fancy, 
> you''d need to use terminal FIFOs instead of using terminal SFQs:
>
>   tc qdisc add dev $INTERFACE parent 10:1 handle 100:0 pfifo limit 10
>
> Assuming your HTB tree has a leaf class of 10:1, into which you have 
> classified some of your traffic, you now have a very short queue 
> there.  If a burst of traffic involving more than ten packets 
> arrives, the traffic will tail drop.  In heavily congested 
> situations, this can be useful, although this still allows for a 
> single dominating UDP flow, so I''d probably prefer the SFQ
solution,
> and maybe I shouldn''t have brought up this little trick.
>   
I think I''ll stick to htb...
>  : I ran both watch -n 1 tc -s class show dev eth0 and watch -n 1 tc 
>  : -s qdisc show eth0. (When I ran class show, i did not have enough 
>  : room to see classes 80 and 90. When I ran qdisc show, I was able 
>  : to see all the classes.) During my runs of tc in this manner, I 
>  : saw zero traffic going to class 100 when running, starting, or 
>  : stopping bit torrent.
>
> If you expect that the torrential traffic to appear in class 100, 
> and it does not, then you have done something wrong in your 
> classifier.  Look there.
>   
By classifier I think you mean:
iptables -t mangle -A POSTROUTING -o ${IFext} -p tcp --dport 10000:10000 
-j CLASSIFY --set-class 1:100

And having looked at that, I see part of my problem. --dport should be 
--sport as I have no way to control what port
the other client wishes to use, but I do have control over what ports I 
use to send my data stream.
>  : Almost all the traffic was going to class 10 and 90 (default) 
>  : with the exception of my ICMP and UDP traffic which was going to 
>  : class 70 and class 60 which I have set aside for IRC traffic. 
>  : Class 100 saw absolutely zero traffic.
>  : 
>  : Is this a case where the default class (90) is getting all the 
>  : traffic because it can handle it as my LAN has very little 
>  : "other" traffic most of the time to deal with, so there is no 
>  : need to throttle it back? If so, how can I force a particular 
>  : class to be used regardless of the default, so that I can control 
>  : individual apps by them selves?
>
> Try looking at your classifiers ("tc filter" statements) to 
> determine where the problem is.  If you can''t figure it out, then 
> send the list your classifiers and class structure (tc commands).
>   
I ran "tc filter" on the command line, but received no output in
return.
I read the man page and it leads me to believe
that it''s not meant for viewing the filters. I also did some packet 
sniffing on my router PC. I am not 100% sure why yet,
but, when I would filter out port 10000-10004, wireshark would return 
packets that had source ports other than the
range I was looking for. While I cannot be sure as of yet, I think this 
is a result of the NAT process and the router box
is using whatever port it has available while allowing my bit torrent 
client to send on the proper ports. (I am still learning
about NAT, so I''m still confused about some things. To be honest, I 
would like to be able to NAT not only IPs, but
ports as well so that I do not have to reconfigure things like SSH to 
run on non-standard ports. But thats for another
mailing list. :P) One thing I did notice was, while I did see traffic in 
class 100 for the first time, my torrent client still showed
outgoing bandwidth of more than 20kbit. Is this simply a function of the 
router actually limiting the traffic and the torrent
client simply not knowing? Or (and I assume this is incorrect thinking) 
should the torrent client visibly indicate to me that
it can only send at X rate because its limited?

To make this much simpler, I will paste my tc rules and iptables rules 
(which classify my traffic) at the bottom of this e-mail.
I hope you can find something (related specifically to bit torrent) that 
will allow me to limit torrent traffic without the need
to limit each client by hand.
> I''d skip playing with terminal FIFOs at first (if ever).  Try 
> working on the following:
>
>   * switch the units to kbit
>   * fix the classifiers and verify that you are seeing all traffic
>     in the class you want to see the traffic in
>   * turn the torrent client up to "insane" and watch how your IRC
or
>     ssh latency is very low
>   
Already done the switch to kbit. Started seeing traffic in class 100 by 
changing my iptables rules. Set the torrent client to
unlimited outgoing bandwidth and as usual things like SSH slowed down on 
sending (which is a bit perplexing). As soon
as I set the torrent client back down to a 20kbit limit (in the client), 
SSH response came back to normal. As you can see
below, this is probably because I have bulk SSH traffic set in the 
default class (90) which should allow SSH to use 2kbit
to 360kbit of bandwidth. So I am at a total loss as to why (outgoing) 
SSH traffic would become so slow, because it has
access to more bandwidth than torrent (at least in my
thinking).
> Good luck,
>
> - -Martin
>
>  [*] IMQ, intermediate queuing device
>      http://www.linuximq.net/
>      IFB, intermediate functional block
>      http://linux-net.osdl.org/index.php/IFB
>
> - -- 
> Martin A. Brown
> http://linux-ip.net/
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.2 (GNU/Linux)
> Comment: pgf-0.72 (http://linux-ip.net/sw/pine-gpg-filter/)
>
> iD4DBQFG3L5sHEoZD1iZ+YcRAso6AJjnbMjcso8t4+GugFC6eHQOyqJQAJ9kpWbZ
> sOS369AbZkPQ9rg3rhiawg=> =gvDh
> -----END PGP SIGNATURE-----
>
>   
Thanks again for all your help. Maybe you will be able to help me learn 
this and figure out what I''ve
been doing wrong. :)

Vadtec

--------------------------------------------------------------------------------------------------------

$IFext is of course eth0 (link to the modem)

# 360kbit traffic (1:10)
iptables -t mangle -A POSTROUTING -o ${IFext} -p tcp --syn -m length 
--length 40:68 -j CLASSIFY --set-class 1:10
iptables -t mangle -A POSTROUTING -o ${IFext} -p tcp --tcp-flags ALL 
SYN,ACK -m length --length 40:68 -j CLASSIFY --set-class 1:10
iptables -t mangle -A POSTROUTING -o ${IFext} -p tcp --tcp-flags ALL ACK 
-m length --length 40:100 -j CLASSIFY --set-class 1:10
iptables -t mangle -A POSTROUTING -o ${IFext} -p tcp --tcp-flags ALL RST 
-j CLASSIFY --set-class 1:10
iptables -t mangle -A POSTROUTING -o ${IFext} -p tcp --tcp-flags ALL 
ACK,RST -j CLASSIFY --set-class 1:10
iptables -t mangle -A POSTROUTING -o ${IFext} -p tcp --tcp-flags ALL 
ACK,FIN -j CLASSIFY --set-class 1:10

# 256kbit traffic (1:20)
# interactive SSH traffic
iptables -t mangle -A POSTROUTING -o ${IFext} -p tcp --sport 10022 -m 
length --length 40:100 -j CLASSIFY --set-class 1:20
iptables -t mangle -A POSTROUTING -o ${IFext} -p tcp --dport 10022 -m 
length --length 40:100 -j CLASSIFY --set-class 1:20

# 128kbit traffic (1:30)
# interactive mail or web traffic
iptables -t mangle -A POSTROUTING -o ${IFext} -p tcp -m multiport 
--sport http,imap,https,imaps -j CLASSIFY --set-class 1:30
# dns lookups
iptables -t mangle -A POSTROUTING -o ${IFext} -p tcp --dport domain -j 
CLASSIFY --set-class 1:30
# team speak
iptables -t mangle -A POSTROUTING -o ${IFext} -p tcp --dport 8767 -j 
CLASSIFY --set-class 1:30


# 64kbit traffic (1:40)

# 32kbit traffic (1:50)
# SSH
iptables -t mangle -A POSTROUTING -o ${IFext} -p tcp --sport 10022 -m 
length --length 101: -j CLASSIFY --set-class 1:50

# NFS
#iptables -t mangle -A POSTROUTING -o ${IFext} -p udp --sport 2049 -j 
CLASSIFY --set-class 1:50
#iptables -t mangle -A POSTROUTING -o ${IFext} -p tcp --sport 2049 -j 
CLASSIFY --set-class 1:50

# 20kbit traffic (1:60)
# default irc traffic
iptables -t mangle -A POSTROUTING -o ${IFext} -p tcp --dport 6660:6669 
-j CLASSIFY --set-class 1:60
iptables -t mangle -A POSTROUTING -o ${IFext} -p tcp --dport 6999 -j 
CLASSIFY --set-class 1:60
iptables -t mangle -A POSTROUTING -o ${IFext} -p tcp --dport 7000 -j 
CLASSIFY --set-class 1:60

# 16kbit traffic (1:70)
# ICMP, UDP
iptables -t mangle -A POSTROUTING -o ${IFext} -p udp -j CLASSIFY 
--set-class 1:70
iptables -t mangle -A POSTROUTING -o ${IFext} -p icmp -m length --length 
28:1500 -m limit --limit 2/s --limit-burst 5 -j CLASSIFY -$

# 8kbit traffic (1:80)

# 2kbit traffic (1:90)
# bulk traffic
iptables -t mangle -A POSTROUTING -o ${IFext} -p tcp --sport 25 -j 
CLASSIFY --set-class 1:90
iptables -t mangle -A POSTROUTING -o ${IFext} -p tcp --sport 10000:10004 
-j CLASSIFY --set-class 1:100 <=== this is the bit torrent classifier

--------------------------------------------------------------------------------------------------------

$IFext is of course eth0 (link to the modem), $QUANTUM is 1490 (due to, 
if I assume correctly, my MTU
being 1492, in the modem), $MAX_RATE is 360kbit (360kbps conventional talk)

    $TC qdisc add dev $IFext root handle 1: htb default 90
    $TC class add dev $IFext parent 1: classid 1:1 htb rate $MAX_RATE 
quantum $QUANTUM
    $TC class add dev $IFext parent 1:1 classid 1:10 htb rate 240kbit 
ceil $MAX_RATE prio 0 quantum $QUANTUM
    $TC class add dev $IFext parent 1:1 classid 1:20 htb rate 192kbit 
ceil $MAX_RATE prio 1 quantum $QUANTUM
    $TC class add dev $IFext parent 1:1 classid 1:30 htb rate 80kbit 
ceil $MAX_RATE prio 2 quantum $QUANTUM
    $TC class add dev $IFext parent 1:1 classid 1:40 htb rate 64kbit 
ceil $MAX_RATE prio 3 quantum $QUANTUM
    $TC class add dev $IFext parent 1:1 classid 1:50 htb rate 32kbit 
ceil $MAX_RATE prio 4 quantum $QUANTUM
    $TC class add dev $IFext parent 1:1 classid 1:60 htb rate 20kbit 
ceil $MAX_RATE prio 5 quantum $QUANTUM
    $TC class add dev $IFext parent 1:1 classid 1:70 htb rate 16kbit 
ceil $MAX_RATE prio 6 quantum $QUANTUM
    $TC class add dev $IFext parent 1:1 classid 1:80 htb rate 8kbit ceil 
$MAX_RATE prio 7 quantum $QUANTUM
    $TC class add dev $IFext parent 1:1 classid 1:90 htb rate 2kbit ceil 
$MAX_RATE prio 8 quantum $QUANTUM
    $TC class add dev $IFext parent 1:1 classid 1:100 htb rate 2kbit 
ceil 20kbit prio 9 quantum $QUANTUM
    $TC qdisc add dev $IFext parent 1:10 handle 10: sfq perturb 10
    $TC qdisc add dev $IFext parent 1:20 handle 20: sfq perturb 10
    $TC qdisc add dev $IFext parent 1:30 handle 30: sfq perturb 10
    $TC qdisc add dev $IFext parent 1:40 handle 40: sfq perturb 10
    $TC qdisc add dev $IFext parent 1:50 handle 50: sfq perturb 10
    $TC qdisc add dev $IFext parent 1:60 handle 60: sfq perturb 10
    $TC qdisc add dev $IFext parent 1:70 handle 70: sfq perturb 10
    $TC qdisc add dev $IFext parent 1:80 handle 80: sfq perturb 10
    $TC qdisc add dev $IFext parent 1:90 handle 90: sfq perturb 10
    $TC qdisc add dev $IFext parent 1:100 handle 100: sfq perturb 10

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Good morning,

 : By classifier I think you mean:
 : iptables -t mangle -A POSTROUTING -o ${IFext} -p tcp --dport 10000:10000 -j
 : CLASSIFY --set-class 1:100

Exactly.

 : And having looked at that, I see part of my problem. --dport should be
 : --sport as I have no way to control what port
 : the other client wishes to use, but I do have control over what ports I use
 : to send my data stream.

Yes.  We think about services (at TCP) as being on a standard port, 
but iptables operates at L3 (even though it can examine higher 
layers).  So, if the packet in question is a return packet from an 
HTTP server to a client, then as far as iptables is concerned, the 
source port is tcp/80.

 : I ran "tc filter" on the command line, but received no output in 
 : return. I read the man page and it leads me to believe that it''s 
 : not meant for viewing the filters.

Depends, but yes, the "tc filter" output is not necessarily 
attractive.  You can classify with "tc filter" instead of iptables, 
but if you reach your goal, it doesn''t matter which mechanism you 
use to classify your packets.

 : One thing I did notice was, while I did see traffic in class 100 
 : for the first time, my torrent client still showed outgoing 
 : bandwidth of more than 20kbit.

Well, a typical torrent client will use a number of connections.  
Perhaps some of the connections are classified correctly and some 
are not?

 : Is this simply a function of the router actually limiting the 
 : traffic and the torrent client simply not knowing? Or (and I 
 : assume this is incorrect thinking) should the torrent client 
 : visibly indicate to me that it can only send at X rate because 
 : its limited?

I would expect the torrent client to be reporting the actual 
speed(s), so I would expect it to be reporting 20kbit rates.

 : To make this much simpler, I will paste my tc rules and iptables 
 : rules (which classify my traffic) at the bottom of this e-mail. I 
 : hope you can find something (related specifically to bit torrent) 
 : that will allow me to limit torrent traffic without the need to 
 : limit each client by hand.

You are getting there.

 : So I am at a total loss as to why (outgoing) SSH traffic would 
 : become so slow, because it has access to more bandwidth than 
 : torrent (at least in my thinking).

Are you sure that it''s outgoing SSH traffic that is slow?  Consider 
the following scenario:

  * your outgoing queues are configured completely correctly
  * outgoing ssh IP packet gets into correct queue
  * inbound return IP packet from ssh server is delayed inbound
    because you are not shaping downstream traffic

So, before you conclude that your shaping isn''t working, I think 
you''ll need to apply some sort of mechanisms also on your internal 
interface.

Snipped iptables classification.  Nothing looks fishy to me there 
(though I haven''t used the iptables CLASSIFY target).

 : $IFext is of course eth0 (link to the modem), $QUANTUM is 1490 
 : (due to, if I assume correctly, my MTU being 1492, in the modem), 
 : $MAX_RATE is 360kbit (360kbps conventional talk)

I wouldn''t set quantum unless you need to do so.  HTB will calculate 
this for you.  If you do need to set the quantum, I''d recommend 
setting it just a bit larger than the MTU...so 1500 or 1536 in your 
pppoe situation.

Good luck,

- -Martin

- -- 
Martin A. Brown
http://linux-ip.net/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: pgf-0.72 (http://linux-ip.net/sw/pine-gpg-filter/)

iD8DBQFG3Vd1HEoZD1iZ+YcRAv0pAKDZ0qtpxyOkGJJQ7H5rWtFi3HlM2gCgrugd
WyARMeCjVI9TD/1CcTTDAsw=RKrP
-----END PGP SIGNATURE-----

Martin A. Brown wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Good morning,
>
>  : I ran "tc filter" on the command line, but received no output
in
>  : return. I read the man page and it leads me to believe that
it''s
>  : not meant for viewing the filters.
>
> Depends, but yes, the "tc filter" output is not necessarily 
> attractive.  You can classify with "tc filter" instead of
iptables,
> but if you reach your goal, it doesn''t matter which mechanism you 
> use to classify your packets.
>   
Can you provide a simple example of how to filter with tc rather than 
iptables? Just enough of an idea
for me to grasp it on my own. I think it would be better to use tc to do 
the filtering rather than iptables,
as thats what tc is meant to do.
>  : One thing I did notice was, while I did see traffic in class 100 
>  : for the first time, my torrent client still showed outgoing 
>  : bandwidth of more than 20kbit.
>
> Well, a typical torrent client will use a number of connections.  
> Perhaps some of the connections are classified correctly and some 
> are not?
>   
One thing I had failed to take into account was the possibility that bit 
torrent *may* be using some UDP
ports. In an attempt to test this theory, I added the udp filters to 
iptables and watched the data stream.
No change. -_- Bit torrent is still showing outgoing speeds of above 
20k. So, I then limitted the filter
to the LAN IP for my router box and forced my torrent client to use that 
IP. Still above 20k. Long story
short, I tried about 5 different things to get it to work properly. No 
luck. I think part of the problem is that
my torrent client doesn''t seem to honor the port ranges I put into 
effect. Which would definitely allow traffic
to get past it.
>  : Is this simply a function of the router actually limiting the 
>  : traffic and the torrent client simply not knowing? Or (and I 
>  : assume this is incorrect thinking) should the torrent client 
>  : visibly indicate to me that it can only send at X rate because 
>  : its limited?
>
> I would expect the torrent client to be reporting the actual 
> speed(s), so I would expect it to be reporting 20kbit rates.
>   
As would I.
>  : To make this much simpler, I will paste my tc rules and iptables 
>  : rules (which classify my traffic) at the bottom of this e-mail. I 
>  : hope you can find something (related specifically to bit torrent) 
>  : that will allow me to limit torrent traffic without the need to 
>  : limit each client by hand.
>
> You are getting there.
>
>  : So I am at a total loss as to why (outgoing) SSH traffic would 
>  : become so slow, because it has access to more bandwidth than 
>  : torrent (at least in my thinking).
>
> Are you sure that it''s outgoing SSH traffic that is slow? 
Consider
> the following scenario:
>
>   * your outgoing queues are configured completely correctly
>   * outgoing ssh IP packet gets into correct queue
>   * inbound return IP packet from ssh server is delayed inbound
>     because you are not shaping downstream traffic
>
> So, before you conclude that your shaping isn''t working, I think 
> you''ll need to apply some sort of mechanisms also on your internal
> interface.
>   
Bah, I need to figure out outgoing before I mess with incoming. I''m 
liable to cut my self off
from the internet. :P
> Snipped iptables classification.  Nothing looks fishy to me there 
> (though I haven''t used the iptables CLASSIFY target).
>
>  : $IFext is of course eth0 (link to the modem), $QUANTUM is 1490 
>  : (due to, if I assume correctly, my MTU being 1492, in the modem), 
>  : $MAX_RATE is 360kbit (360kbps conventional talk)
>
> I wouldn''t set quantum unless you need to do so.  HTB will
calculate
> this for you.  If you do need to set the quantum, I''d recommend 
> setting it just a bit larger than the MTU...so 1500 or 1536 in your 
> pppoe situation.
>   
Ok, quantum has always been 1512 for me, so I assume tc is taking care 
of it.
> Good luck,
>
> - -Martin
>
> - -- 
> Martin A. Brown
> http://linux-ip.net/
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.2 (GNU/Linux)
> Comment: pgf-0.72 (http://linux-ip.net/sw/pine-gpg-filter/)
>
> iD8DBQFG3Vd1HEoZD1iZ+YcRAv0pAKDZ0qtpxyOkGJJQ7H5rWtFi3HlM2gCgrugd
> WyARMeCjVI9TD/1CcTTDAsw> =RKrP
> -----END PGP SIGNATURE-----
>
>   
So, I am at a total loss as to why this isn''t working. Well, not a
total
loss, but enough of a total loss
as to disable some of the shaping I had been using  that was giving me 
fits. I think it''s much better for
me to let the default class handle everything and go from there for now.

As I said above, can you provide a simple example of how to filter with 
tc? I think filtering in tc will be
both more appropriate and less hassling.

I really appreciate you taking the time to help me with this. I am by no 
means a noob to networking,
but when it comes to traffic shaping, I might as well be. Though, on a 
positive note, at least I''ve been able
to properly shape some traffic, like IRC. :) (I''m also going to contact
the developers of my torrent client
and ask them about port range limiting. And why it doesn''t seem to be 
working.)

Thanks again,


Vadtec

After another two days of trying to get this to work like I think it 
should, I''ve still hit a brick wall.

As I am at a total loss as to what''s going on, I am providing my shell 
script that I use to setup my
tc rules.

You will notice a big section that is commented out. Those are rules I 
have been using that (for
all intents and purposes) for like I expect.

I adapted my section based on the example I found at
http://lartc.org/howto/lartc.cookbook.ultimate-tc.html section 15.8.3

#!/bin/sh

IFext=''eth0''
IFint=''eth1''
rc_done="  done"
rc_failed="  failed"

MAX_RATE=''360kbit''
MAX_RATEA=''360''
INGRESS_RATE=''1400kbit''

return=$rc_done

TC=''/sbin/tc''

tc_reset ()
{
    # Reset everything to a known state (cleared)
    $TC qdisc del dev $IFext root 2> /dev/null > /dev/null
}

tc_status ()
{
    echo "[qdisc - $IFext]"
    $TC -s qdisc show dev $IFext
    echo "------------------------"
    echo
    echo "[class - $IFext]"
    $TC -s class show dev $IFext
}

tc_showfilter ()
{
    echo "[filter - $IFext]"
    $TC -s filter show dev $IFext
}

case "$1" in

    start)
    echo -n "Starting traffic shaping"
    tc_reset
#    U320="$TC filter add dev $IFext protocol ip parent 1:0 prio 0
u32"

    $TC qdisc del dev eth0 root
    $TC qdisc del dev eth0 ingress

    # uplink
    # dev eth0
    $TC qdisc add dev $IFext root handle 1: htb default 20
    $TC class add dev $IFext parent 1: classid 1:1 htb rate $MAX_RATE 
burst 6k
    $TC class add dev $IFext parent 1:1 classid 1:10 htb rate 240kbit 
ceil $MAX_RATE burst 6k prio 1
    $TC class add dev $IFext parent 1:1 classid 1:20 htb rate 200kbit 
ceil $[9*$MAX_RATEA/10]kbit burst 6k prio 2
    $TC qdisc add dev $IFext parent 1:10 handle 10: sfq perturb 5
    $TC qdisc add dev $IFext parent 1:20 handle 20: sfq perturb 5

    $TC filter add dev $IFext parent 1:0 protocol ip prio 10 u32 match 
ip tos 0x10 0xff flowid 1:10
    $TC filter add dev $IFext parent 1:0 protocol ip u32 match ip 
protocol 1 0xff flowid 1:10
    $TC filter add dev $IFext parent 1: protocol ip prio 10 u32 match ip 
protocol 6 0xff match u8 0x05 0x0f at 0 match u16 0x0000 0xffc0 at 2 
match u8 0x10 0xff at 3 flowid 1:10

    # downlink
    # dev eth0
    tc qdisc add dev $IFext handle ffff: ingress

    $TC filter add dev $IFext parent ffff: protocol ip prio 50 u32 match 
ip src 0.0.0.0/0 police rate $INGRESS_RATE burst 10k drop flowid :1

    #
    # dev eth0 - creating qdiscs & classes
    #
    #$TC qdisc add dev $IFext root handle 1: htb default 90
    #$TC class add dev $IFext parent 1: classid 1:1 htb rate $MAX_RATE
    #$TC class add dev $IFext parent 1:1 classid 1:10 htb rate 240kbit 
ceil $MAX_RATE prio 0
    #$TC class add dev $IFext parent 1:1 classid 1:20 htb rate 192kbit 
ceil $MAX_RATE prio 1
    #$TC class add dev $IFext parent 1:1 classid 1:30 htb rate 80kbit 
ceil $MAX_RATE prio 2
    #$TC class add dev $IFext parent 1:1 classid 1:40 htb rate 64kbit 
ceil $MAX_RATE prio 3
    #$TC class add dev $IFext parent 1:1 classid 1:50 htb rate 32kbit 
ceil $MAX_RATE prio 4
    #$TC class add dev $IFext parent 1:1 classid 1:60 htb rate 20kbit 
ceil $MAX_RATE prio 5
    #$TC class add dev $IFext parent 1:1 classid 1:70 htb rate 16kbit 
ceil $MAX_RATE prio 6
    #$TC class add dev $IFext parent 1:1 classid 1:80 htb rate 8kbit 
ceil $MAX_RATE prio 7
    #$TC class add dev $IFext parent 1:1 classid 1:90 htb rate 2kbit 
ceil $MAX_RATE prio 8
    #$TC class add dev $IFext parent 1:1 classid 1:100 htb rate 2kbit 
ceil 20kbit prio 9
    #$TC qdisc add dev $IFext parent 1:10 handle 10: sfq perturb 10
    #$TC qdisc add dev $IFext parent 1:20 handle 20: sfq perturb 10
    #$TC qdisc add dev $IFext parent 1:30 handle 30: sfq perturb 10
    #$TC qdisc add dev $IFext parent 1:40 handle 40: sfq perturb 10
    #$TC qdisc add dev $IFext parent 1:50 handle 50: sfq perturb 10
    #$TC qdisc add dev $IFext parent 1:60 handle 60: sfq perturb 10
    #$TC qdisc add dev $IFext parent 1:70 handle 70: sfq perturb 10
    #$TC qdisc add dev $IFext parent 1:80 handle 80: sfq perturb 10
    #$TC qdisc add dev $IFext parent 1:90 handle 90: sfq perturb 10
    #$TC qdisc add dev $IFext parent 1:100 handle 100: sfq perturb 10
    tc_status
    ;;

     stop)
     echo -n "Stopping traffic shaper"
     tc_reset || return=$rc_failed
     echo -e "$return"
     ;;
   
    restart|reload)
    $0 stop
    $0 start || return=$rc_failed
    ;;
   
    stats|status)
    tc_status
    ;;

    filter)
    tc_showfilter
    ;;

    *)
    echo "Usage: $0 {start|stop|restart|stats|filter}"
    exit 1

esac
test "$return" = "$rc_done" || exit 1



Regardless of the values I use for any of the rates, I still have the 
same problem. I do not classify ANY traffic with iptables for this set 
of tc filters, so any traffic that is generated is solely shaped by tc 
in this case.

The problem I have is this, whenever I allow any torrent client to use 
above 20k of outgoing bandwidth, *everything* becomes laggy for some 
reason. Most notable is SSH. While I have no accurate way to time the 
lag, as near as I can tell it lags about 2 seconds. When I press a key 
on my keyboard it takes ~2 seconds to show up in the SSH client. Or, if 
I enter lets say "ls" and press enter, it takes roughly 2 seconds for 
the ls output to reach me, and while its being displayed, its choppy 
appearing (as in it comes in chunks rather than a nice stream).

I see absolutely no reason for this to be happening. Why should anything 
lag when I''m using more outgoing bandwidth? Why would more outgoing 
bandwidth cause a slow down on incoming bandwidth. Or, why would more 
outgoing bandwidth slow down the filters/tc? I''ve verified that this 
happens on more than just my modem. I''ve used two different routers and
a cable modem. All suffer from the same symptoms.

For the record, my router PC is built as such:
CentOS 5 64bit
AMD Sempron 2800+ (64bit, 1.6Ghz)
2GB of DDR
2GB of swap

As you can see, this PC is more than capable of acting as my router.

I would greatly appreciate any input as to why this torrent problem 
keeps poping up. As near as I can tell, it doesn''t happen during FTP 
upload or any other sort of intensive upload action.


Thanks,


Vadtec

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Greetings again Vadtec,

 : After another two days of trying to get this to work like I think 
 : it should, I''ve still hit a brick wall.

This can be frustrating, I know.

[ snip ]  (I admit, I didn''t look at the rules very closely, but 
           fundamentally the rules themselves look good.)

You do use an ingress filter with a policer.  Since the policer 
applies equally to all inbound traffic flows, it doesn''t really do 
you any good here.

 : Regardless of the values I use for any of the rates, I still have 
 : the same problem. I do not classify ANY traffic with iptables for 
 : this set of tc filters, so any traffic that is generated is 
 : solely shaped by tc in this case.
 : 
 : The problem I have is this, whenever I allow any torrent client 
 : to use above 20k of outgoing bandwidth, *everything* becomes 
 : laggy for some reason. Most notable is SSH. While I have no 
 : accurate way to time the lag, as near as I can tell it lags about 
 : 2 seconds. When I press a key on my keyboard it takes ~2 seconds 
 : to show up in the SSH client. Or, if I enter lets say "ls" and 
 : press enter, it takes roughly 2 seconds for the ls output to 
 : reach me, and while its being displayed, its choppy appearing (as 
 : in it comes in chunks rather than a nice stream).
 : 
 : I see absolutely no reason for this to be happening. Why should 
 : anything lag when I''m using more outgoing bandwidth? Why would 
 : more outgoing bandwidth cause a slow down on incoming bandwidth. 
 : Or, why would more outgoing bandwidth slow down the filters/tc? 
 : I''ve verified that this happens on more than just my modem.
I''ve
 : used two different routers and a cable modem. All suffer from the 
 : same symptoms.

You are neglecting one important consideration, and that is the 
download traffic.  You may well be shaping the upstream traffic, but 
the queues transmitting downstream are also full.

So, once again--I''d suggest that you consider what it takes for your 
"ls" keystrokes and then the output to make it back to you.

  * you press a key ("l"), your ssh client transmits a packet
  * this packet goes across your congested link (probably 
    significantly delayed, because there''s congestion here)
  * the packet arrives on the ultimate destination
  * the sshd on the remote side receives the packet, passes it to
    the application layer (blah blah, bash, fork, exec, ls)
  * the sshd receives data from the application layer and transmits 
    a packet
  * your traffic control structures take this inbound ssh packet and 
    give it its dedicated slot (however you have built your queues)
  * packet returns quickly to your system

So, your shaping is great, but you aren''t shaping all of the 
paths through which your application flow must pass.

 : For the record, my router PC is built as such:
 : CentOS 5 64bit
 : AMD Sempron 2800+ (64bit, 1.6Ghz)
 : 2GB of DDR
 : 2GB of swap
 : 
 : As you can see, this PC is more than capable of acting as my router.

And then some.  Very reasonable looking box.

Good luck,

- -Martin

- -- 
Martin A. Brown
http://linux-ip.net/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: pgf-0.72 (http://linux-ip.net/sw/pine-gpg-filter/)

iD8DBQFG32pfHEoZD1iZ+YcRAp8DAKDDb7eO6/cNZp+lLg8tyO07QffzuQCfcTA3
DnHkDHC/Ea09qNsuEBhi+vs=VBjg
-----END PGP SIGNATURE-----

Martin A. Brown wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Greetings again Vadtec,
>
>  : After another two days of trying to get this to work like I think 
>  : it should, I''ve still hit a brick wall.
>
> This can be frustrating, I know.
>
> [ snip ]  (I admit, I didn''t look at the rules very closely, but 
>            fundamentally the rules themselves look good.)
>
> You do use an ingress filter with a policer.  Since the policer 
> applies equally to all inbound traffic flows, it doesn''t really do
> you any good here.
>
>  : Regardless of the values I use for any of the rates, I still have 
>  : the same problem. I do not classify ANY traffic with iptables for 
>  : this set of tc filters, so any traffic that is generated is 
>  : solely shaped by tc in this case.
>  : 
>  : The problem I have is this, whenever I allow any torrent client 
>  : to use above 20k of outgoing bandwidth, *everything* becomes 
>  : laggy for some reason. Most notable is SSH. While I have no 
>  : accurate way to time the lag, as near as I can tell it lags about 
>  : 2 seconds. When I press a key on my keyboard it takes ~2 seconds 
>  : to show up in the SSH client. Or, if I enter lets say "ls" and
>  : press enter, it takes roughly 2 seconds for the ls output to 
>  : reach me, and while its being displayed, its choppy appearing (as 
>  : in it comes in chunks rather than a nice stream).
>  : 
>  : I see absolutely no reason for this to be happening. Why should 
>  : anything lag when I''m using more outgoing bandwidth? Why would 
>  : more outgoing bandwidth cause a slow down on incoming bandwidth. 
>  : Or, why would more outgoing bandwidth slow down the filters/tc? 
>  : I''ve verified that this happens on more than just my modem.
I''ve
>  : used two different routers and a cable modem. All suffer from the 
>  : same symptoms.
>
> You are neglecting one important consideration, and that is the 
> download traffic.  You may well be shaping the upstream traffic, but 
> the queues transmitting downstream are also full.
>
> So, once again--I''d suggest that you consider what it takes for
your
> "ls" keystrokes and then the output to make it back to you.
>
>   * you press a key ("l"), your ssh client transmits a packet
>   * this packet goes across your congested link (probably 
>     significantly delayed, because there''s congestion here)
>   * the packet arrives on the ultimate destination
>   * the sshd on the remote side receives the packet, passes it to
>     the application layer (blah blah, bash, fork, exec, ls)
>   * the sshd receives data from the application layer and transmits 
>     a packet
>   * your traffic control structures take this inbound ssh packet and 
>     give it its dedicated slot (however you have built your queues)
>   * packet returns quickly to your system
>
> So, your shaping is great, but you aren''t shaping all of the 
> paths through which your application flow must pass.
>   
Ok, so answer me this: Does tc use the same general queue for egress and 
ingress traffic?
Or is it a matter of tc just using a default ingress filter that is 
filling for some reason?

Either way, it still doesn''t explain why everything starts to lag as 
soon as one applications outgoing
bandwidth climbs anything above 20.5k (I tested to see when the break 
was). If the ingress queue
works just fine at 20k, why does it not work just fine at >21k?
That''s
my central issue. If I can figure
that out, I can most likely filter differently to insure proper 
performance is maintained.
>  : For the record, my router PC is built as such:
>  : CentOS 5 64bit
>  : AMD Sempron 2800+ (64bit, 1.6Ghz)
>  : 2GB of DDR
>  : 2GB of swap
>  : 
>  : As you can see, this PC is more than capable of acting as my router.
>
> And then some.  Very reasonable looking box.
>
> Good luck,
>
> - -Martin
>
> - -- 
> Martin A. Brown
> http://linux-ip.net/
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.2 (GNU/Linux)
> Comment: pgf-0.72 (http://linux-ip.net/sw/pine-gpg-filter/)
>
> iD8DBQFG32pfHEoZD1iZ+YcRAp8DAKDDb7eO6/cNZp+lLg8tyO07QffzuQCfcTA3
> DnHkDHC/Ea09qNsuEBhi+vs> =VBjg
> -----END PGP SIGNATURE-----
>
>   
Thanks for the input Martin. I really do appreciate the time you have 
taken with trying to help me.

In an effort to make this work better, I am currently working on setting 
up both egress and ingress qdiscs.
(You''re e-mail prompted me to take a break. :P) For the sake of 
simplicity, I am going to setup my ingress
qdiscs (practically) the same way as I do my egress filters (even though 
they may not get used, I will still
set them up for the time being just in case I need them). Then, using fw 
marking, I''m going to filter using
iptables on both egress and ingress (though to be honest I really have 
no idea how I''m going to do the ingress
filtering and get it back to the right place, though I assume it will 
take care of it self, just like egress).

Maybe once I do this I will start to see improvements. You can expect 
that in a few days I will post again when
I have exhausted all my (misguided or otherwise) ideas about how to make 
this work.

Thanks again,


Vadtec

Ok, I''m almost to tearing my hair out now...

I am trying to do more proper ingress policing as Martin has said I do. 
But every tutorial or example I have seen on traffic shaping with tc 
says very little (if anything at all) about ingress policing. I have 
tried a few methods to get this to work including: adding qdiscs to it 
(failed), adding filters that would give it a FWMARK to classify with 
(failed), adding HTBs to it (failed). I finally gave up and just routed 
all the ingress to the same root as I have no idea what the heck I''m
doing.

Can someone please provide me with an example of how to split ingress 
policing so that I can achieve what Martin is saying I need to achieve?

Vadtec

Ok, I messed around with 6 different setups over 10 hours yesterday. The 
only one I can get to work properly is my original one.

So, now I''m to the theory stage of trying to figure this out. I got a 
reply from a mailing list user saying I need to do egress filtering in 
two places.

While I could not understand what they were saying very well, it did 
leave me to ponder this theory. It seems to me the whole problem has 
been how I am handling ingress traffic on eth0 (WAN interface). As it 
stands, I do rate limit it and will drop if its coming in to fast. But 
is there anything thats stopping me from routing ingress traffic through 
the egress queues on its way to the LAN? Or will that seriously break 
traffic shaping?

Is what I''m thinking is, the ingress qdisc doesn''t really
control
anything. So, if I were to route it (say with an iptables rule) to an 
egress qdisc on eth1, I could truly control ingress traffic.

I really don''t think this will work as it seems like I am quashing all 
the traffic down one side of what should be a two sided link. While I 
cannot think of a way to visualize this with ASCII art, I can summarize 
the ingress and egress pathways in linear format, as such:

   Egress (LAN to Internet)
--------> LAN traffic ---> eth1 (egress) ---> eth0 (egress) ---> WAN
------------------------------------------
|                                                                       
                                                                        
                           |
|                                                                       
                                                                        
                           |
|                                                                       
                                                                        
                           |
|  Ingress (Internet to LAN)                                            
                                                                        
             |
--------LAN <--- eth1 (ingress) <--- eth0 (egress to eth1 ingress) <---
eth0 (ingress) <--- WAN traffic <--------

or

   Egress (LAN to Internet)
--------> LAN traffic ---> eth1 (egress) ---> eth0 (egress) ---> WAN
------------------------------------------
|                                                                       
                                                                        
                           |
|                                                                       
                                                                        
                           |
|                                                                       
                                                                        
                           |
|  Ingress (Internet to LAN)                                            
                                                                        
             |
--------LAN <--- eth1 (egress) <--- eth0 (ingress to eth1 ingress) <---
eth0 (ingress) <--- WAN traffic <--------

I hate to be so pessimistic. But so far all I''ve gotten is everyone 
saying "You need to filter ingress traffic" with no real or concrete 
examples of how to do such a thing. And the LARTC How To doesn''t 
describe it very well either. It''s like ingress filtering is just not 
done, and those that do it are using such complicated methods that it''s
not worth sharing them.

So, unless someone can provide me with a concrete example of true 
ingress filtering, or how to filter ingress on the LAN side or WAN side 
or whichever side I need to filter it on, I am completely stuck.

Vadtec

The advice you received is pretty good.
Avoid ingress shaping at all costs, and
you don''t need it anyway for your situation.

Use egress shaping on both your internal and
external interfaces.
Traffic coming IN to your network gets shaped
as egress traffic on the LAN interface.
Traffic going OUT from your network gets
shaped as egress traffic on the WAN interface.
So all shaping is egress, but you''re able to
shape in both directions by always delaying
packets as they are SENT by your router.

Think of it this way : all you can really do is
delay sending packets (or ultimately drop them
which is the same as infinitely delaying).
Packets arrive when they arrive, you have no
control over that. This is why shaping has to be
done on egress traffic -- it''s the only lever you have
to pull on.

David Boreham wrote:
> The advice you received is pretty good.
> Avoid ingress shaping at all costs, and
> you don''t need it anyway for your situation.
>
> Use egress shaping on both your internal and
> external interfaces.
> Traffic coming IN to your network gets shaped
> as egress traffic on the LAN interface.
> Traffic going OUT from your network gets
> shaped as egress traffic on the WAN interface.
> So all shaping is egress, but you''re able to
> shape in both directions by always delaying
> packets as they are SENT by your router.
>
> Think of it this way : all you can really do is
> delay sending packets (or ultimately drop them
> which is the same as infinitely delaying).
> Packets arrive when they arrive, you have no
> control over that. This is why shaping has to be
> done on egress traffic -- it''s the only lever you have
> to pull on.
>
>
>
Thank you! This is the first explanation that has actually made sense to 
me. Before I had a vague idea of what was being said.

I do have one question though. On the egress shaping on eth1 (LAN 
interface), when using iptables I should do everything in the 
POSTROUTING chain correct? That way it gets routed to the proper LAN 
node and still gets shaped, correct? If thats the case, I can have a 
setup working in no time (I hope).

Many thanks,

Vadtec

>
>
> I do have one question though. On the egress shaping on eth1 (LAN 
> interface), when using iptables I should do everything in the 
> POSTROUTING chain correct? That way it gets routed to the proper LAN 
> node and still gets shaped, correct? If thats the case, I can have a 
> setup working in no time (I hope).
Well now it''s my turn to be confused ! What''s the connection
between
iptables and your traffic shaping setup ? Are you marking packets for
shaping, something like that ? If so I have no idea. My routing and
shaping are completely separate and unrelated in any way (I use
tc filter to classify packets).

David Boreham wrote:
>
>>
>>
>> I do have one question though. On the egress shaping on eth1 (LAN 
>> interface), when using iptables I should do everything in the 
>> POSTROUTING chain correct? That way it gets routed to the proper LAN 
>> node and still gets shaped, correct? If thats the case, I can have a 
>> setup working in no time (I hope).
> Well now it''s my turn to be confused ! What''s the
connection between
> iptables and your traffic shaping setup ? Are you marking packets for
> shaping, something like that ? If so I have no idea. My routing and
> shaping are completely separate and unrelated in any way (I use
> tc filter to classify packets).
>
>
>
iptables (I forget what version onward supports this) has the ability to 
classify packets that it routes via -j CLASSIFY --set-class X:Y. For all 
intents and purposes, this is the same as -j MARK --set-mark X. 
Basically, it allows a user to use iptables to do the actual 
classification of the packet. For an example, take a look at: 
http://lartc.org/howto/lartc.cookbook.fullnat.intro.html, 
http://www.stanford.edu/~fenn/linux/ (this is the tutorial I base my 
iptables and tc rules on)

I am not familiar enough with tc filter syntax to brave it yet. Hence my 
use of iptables to classify packets, which I think is much easier anyways.

I was simply asking whether I would classify the packets in PREROUTING, 
OUTPUT, or POSTROUTING. I assume OUTPUT or POSTROUTING will let me 
achieve my goals with respect to egress on eth1. Guess it''s time to 
enter my dark age of tc+iptables. :P

Sorry if I confused you with my question. It''s the only way I know how 
to traffic shape (so far).

Vadtec

Vadtec wrote:
> MAX_RATE=''360kbit''
> MAX_RATEA=''360''
> INGRESS_RATE=''1400kbit''
If you actuakky sync at 1.5/384kbit these are far too close as DSL sync 
rates are at atm level. It is possible to patch kernel/tc to make things 
work properly, but for testing try 1000/300kbit.

>    $TC qdisc add dev $IFext root handle 1: htb default 20
using default on eth is not ideal as arp will go there - though sfq will 
help you a bit. Don''t set perturb too low as it causes packet
reordering.

>    $TC filter add dev $IFext parent 1: protocol ip prio 10 u32 match ip 
> protocol 6 0xff match u8 0x05 0x0f at 0 match u16 0x0000 0xffc0 at 2 
> match u8 0x10 0xff at 3 flowid 1:10
This won''t work - the last "at 3" should be "at 33"


General other points from some things raised in the rest of the thread -

Bittorrent will use udp if the client does trackerless/dht, but it 
shouldn''t amount to much bandwidth. The port settings are for listening
and don''t mean you won''t use others aswell.

I''ve noticed that if bandwidth is restriced by the bt client and your 
mtu  is <1500 then you can get 1 large and one small packet for every 
chunk sent by the client - if you do your own QOS then turn off the 
client limiter, or you could have some nasty reordering as the small 
packets overtake the large ones (depending on what rules you end up using)

Ingress shaping is different from egress - you have to sacrifice 
bandwidth to get it to work, though. I don''t know why in your test 
latency for ssh lagged when you reached 20KB/sec upload - maybe you need 
to look at counters/tcpdump to be sure your client sets TOS OK. As there 
is  a rule for ICMP you can ping somewhere to see how much lag you are 
getting.

You are right that you can shape your ingress as egress on the LAN 
facing interface, but you need to make allowances for the fact you are 
shaping from the wrong end of the bottleneck. Until you''ve got egress 
sorted I would stick to policing ingress eth0 downto 1mbit while testing.

One more possibility - if you have a multicore CPU then kernel should be 
set to use get time of day or jiffies as a time source for shaping not 
cpu/tsc.

If you have gig nics then there may be other things needed - but only 
for traffic generated on the shaping box its self - I assume your 
torrent box is on a machine on the LAN?

Andy.